1 Introduction↩︎

A core area for Artificial Intelligence (AI) research is to understand the central components of modern learning systems, e.g. large language models (LLMs), assuming certain regularity conditions. Extensive research efforts have been mobilized to tackle the explainability of AI question ([1]), leveraging ideas from computer science ([2]–[4]), information theory ([5], [6]), and theoretical physics ([7]–[9]), to mathematical insights from high-dimensional probability theory ([10], [11]), and algebraic and differential topology ([12]–[14]) either to prove mathematical guarantees or to elucidate model behavior.

In many—if not most—AI research papers, a manifold hypothesis is tacitly assumed, that the data are concentrated near a low curvature manifold without boundary. This assumption drives the research just as an assumption of Gaussianity drives classical approaches to data modeling. For instance, the Uniform Manifold Approximation and Projection (UMAP) algorithm ([15]) explicitly assumes that data are sampled near a low dimensional manifold and cautions the reader that the algorithm may not be reliable if this assumption is violated. Yet many researchers make inferences from UMAP, rarely checking that this assumption holds, and even arguing it is unnecessary. This argument has been routinely rejected in the literature ([16], [17]), and so it is important to test this kind of assumption before data modeling. One common testing strategy is simply “plotting the data” to ascertain the empirical distribution. Plotting high-dimensional data, however, is non-trivial, but there are statistical tests to help us understand the underlying distribution (Mann-Whitney, Kolomogorov-Smirnoff, etc.). In this paper, we provide an analogous test for the regularity of token embeddings.

The first layer of most LLMs is usually comprised of the token embeddings, which are vector representations of tokens (fragments of texts). Thus, a good first step to explain the behavior of an LLM is to understand its initial token subspace where these embedding vectors lie. For the remainder of this paper, the term “token”’ used in the context of the token subspace will refer to this embedding. Numerous papers (for instance, see [18]–[25]) have pointed to unexpected behaviors exhibited by LLMs that hinge on subtle changes in wording and text layout between apparently similar prompts, suggesting that certain semantically similar tokens may have dramatically different topological neighborhoods in the token subspace. Linguistically, these irregularities may correspond to polysemy or homonyms ([26]).

The presence of irregularities imply small changes in language can drive large variation in the token subspace. Moreover, if the token subspace of an LLM is, in fact, singular, hence not a manifold, irregularities can persist into the output of the LLM, perhaps unavoidably and potentially even because of its architecture. This is because the input and output of LLMs are a fixed finite set of tokens yet the internal architectures of the model assume a structure resembling a Riemannian manifold: a metrizable space that locally preserves inner products. Not accounting for singularities in the token subspace may thereby confound our understanding of the LLM’s behavior.

1.1 Contributions↩︎

Our contributions are: (1) two novel statistical tests (for manifolds and for fiber bundles; see below), (2) evidence to reject both hypotheses for several LLMs (Section 5.2), and therefore (3) the identification of otherwise unreported structure in the embeddings for these LLMs. We emphasize that while this paper demonstrates the manifold and fiber bundle tests on token embeddings, our main contribution is primarily a methodology for hypothesis testing on embedded points.

Because connected manifolds have a unique dimension, and because the dimension of fiber bundles are characterized by Theorem 1, our two tests address the following hypotheses given in Table 1.

Both tests are implemented as Algorithm 3. The embedded reach \(\tau\) defines an assumption on the curvature of the underlying space. Low reach corresponds to high curvature and vice versa ([27]). Classical Lagrange interpolation asserts that any finite sample of points can be fit exactly by a manifold. The resulting manifold may have extremely high curvature, which is undesirable because it results in numerical instability ([28]).

To demonstrate our method, Figure 4 (a)–(c) shows the application of the manifold test to several synthetic examples. In Section 5.2, we then provide empirical evidence that the token embedding function obtained from each of four different open source LLMs of moderate size—GPT2 ([29]), Llemma7B ([30]), Mistral7B ([31]), and Pythia6.9B ([32])—cannot be a manifold due to non-constant local dimension as well as tokens (points) without an intrinsic dimension. When these tests are rejected at token \(\psi\), this implies that \(\psi\) is less stable under mathematical transformations than other tokens. In Section 3.2, we show the endemic nature of these instabilities by proving that context cannot resolve singularities persistently. And finally, we provide conditions (Theorem 2) for when singularities can propagate into the output of an LLM for generic transformers.

1.2 Background and related work↩︎

Manifold learning takes its roots in nonlinear dimension reduction of high dimensional data into low dimensional representations of smooth manifolds. While these algorithms have played an important role in modern data science especially in domains such as cosmology ([33]), astrophysics ([34], [35]), dynamical systems ([36], [37]), and quantum chemistry ([38]), the main utility of these methods have been for data visualization. For instance, even though UMAP provided an optimal algorithm for fitting sampled data to a manifold, the authors themselves concede that “if the global structure is of primary interest then UMAP may not be the best choice for dimensionality reduction”. A manifold learning algorithm will find some smooth structure to fit the data rather than “error out” when the data is not sampled from a manifold as it is not a hypothesis test.  [39] showed the dimension of ReLU-Nets are inhomogenous, and so when algorithms fit a manifold to data optimally, for instance, the found manifold may have extremely high curvature or dimension. Both of these are undesirable properties in a model, and both implicitly suggest that a manifold would be a poor choice for a representation ([28]).

To address this concern, [40] presented a test for whether data lie on a manifold with a specified embedded reach, but did not present the values of constants that are critical for practical implementation. More recently, [41], [42] present tests for the manifold hypothesis, with promising results on low dimensional datasets. Our paper remedies both issues: we construct a practical implementation of hypothesis tests for manifolds and fiber bundles with a specified embedded reach.

The token input embedding matrix defines the salient subspace of the latent space for an LLM, by specifying where the tokens are located. There is no a priori reason to suspect it is a manifold. It has already been shown that local neighborhoods of each token have salient topological structure ([13]), and one of the most basic parameters is the dimension near any given token in this space. Higher dimension at a token \(\psi\) means that \(\psi\) has greater density while lower dimensional tokens have less density ([26]). Intuitively, a densely populated neighborhood \(B(\psi)\) will generically occupy a higher dimension than otherwise.

Assuming that word embeddings yield manifolds, some researchers have used global dimension estimators on token input embeddings and word embeddings ([44]–[46]). By using a local (not global) dimension estimator, [43] presented the first (to our knowledge) direct test of whether the token subspace is a manifold for the token input embeddings for several LLMs, although no rigorous hypothesis test was presented. The subspace of tokens has a multimodal distribution of dimension estimates (Figure 1), so unless the space is highly disconnected, it cannot be a manifold.

2 Mathematical background↩︎

A manifold is a space in which every point has a Euclidean neighborhood, and the space can be covered by taking finitely many of these neighborhoods at each point. It follows that if a manifold is connected, then the dimension of each of these Euclidean neighborhoods is the same; we call this the dimension of the manifold. Figure 4 (a) shows the hollow sphere, which is an example of a \(2\)-dimensional manifold as a subset of \(\mathbb{R}^3\).

Given an embedding dimension \(\ell\), embedded tokens lie on a subspace \(X\) of Euclidean space \(\mathbb{R}^\ell\), with the topology and geometry inherited from \(\mathbb{R}^\ell\). Equivalently, one can consider an embedding function \(e:X \to \mathbb{R}^\ell\), which is one-to-one and has a full rank matrix of partial derivatives (Jacobian matrix). Embedding functions therefore preserve topological structure. The inherited information includes a metric (the Euclidean distance) and a notion of volume. The volume form on a subspace is induced by the Euclidean volume. If the subspace is a manifold that is compatible with the inherited metric and volume form, we call the subspace a Riemannian manifold. For a \(2\)-dimensional Riemannian manifold, the volume form is the familiar notion of area. In general, if a space is a Riemannian manifold of dimension \(d\), then the volume \(v\) of a ball of radius \(r\) centered at any point is given by an equation of the form [47], \[\label{eq:volume95approx} v = K r^d + \text{(correction terms)}r^{d+1},\tag{1}\] where \(K>0\) is a constant depending on \(d\). If the correction terms are large, the manifold is said to have high curvature.

The definition of a manifold is global: if even one point fails to have a Euclidean neighborhood, then the space cannot be a manifold, and such a point is called a singularity. One commonly encountered singularity is a boundary, in which a point has a neighborhood that is topologically equivalent to a half space. Spaces whose only singularities are boundaries are called manifolds with boundary. For instance, the solid ball in \(\mathbb{R}^3\) is a manifold with boundary. Its set of singularities is the hollow \(2\)-dimensional sphere.

Classifying the types of singularities up to topological equivalence of their neighborhoods is in general an open problem, though some types are common. Figure 4 (b) shows a space with cusp and boundary singularities. Figure 4 (c) shows a singularity associated with a change in dimension.

In order to handle singularities, we also consider a generalization of a manifold called a fibered manifold with boundary (fiber bundle for short). This is a space in which every point has a neighborhood that can be written as a Cartesian product \(B\times F\), where \(B\) is a manifold and \(F\) is a manifold with boundary. We call \(B\) the base space and \(F\) the fiber space.

Curvature (the correction terms in Equation 1 ) does not generalize cleanly to spaces with singularities. Instead, the embedded reach plays a similar role in what follows. The embedded reach (simply reach) for a space \(X \subseteq \mathbb{R}^\ell\) is the largest distance \(\tau \ge 0\) that a point \(y \in \mathbb{R}^\ell\) can be from \(X\), yet still have a unique closest point on \(X\). For convex subspaces, \(\tau = \infty\). Conversely, spaces with a cusp (like Figure 4 (b)), have \(\tau=0\). Manifolds with small reach have high curvature, and vice versa ([27]).

3 Theoretical insights↩︎

Theorem 1 asserts that for a fiber bundle, the number of tokens within a ball of a given radius \(r\) centered at a fixed token \(\psi\) is a piecewise log-log-linear function of radius, in which the slopes correspond to dimension, and the slopes cannot increase as \(r\) increases.

3.1 Volume versus radius for fiber bundles↩︎

The learned token embedding vectors in an LLM each have the same number of elements, which we refer to as \(\ell\). Thus, the explicit representation of the token subspace \(T\) used by an LLM arises by an embedding function \(e: T \to \mathbb{R}^{\ell}\) into the latent space \(\mathbb{R}^{\ell}\). Since embeddings are one-to-one, each token \(\psi \in T\) corresponds to a unique element \(e(\psi) \in \mathbb{R}^{\ell}\) in the latent space. With a slight abuse of notation, we can write \(\psi\) for both.

The probability distribution of tokens in \(T\) can be modeled by the volume form \(v\). If a fiber bundle is the correct representation of the space of tokens, Theorem 1 characterizes the resulting probability distribution in \(\mathbb{R}^\ell\) using parameters (the exponents in Equation 2 ) that can be estimated from the token input embedding. These parameters are bounded by the dimensions of \(S\) and \(T\). In Algorithm 3, Monte Carlo estimates of volume are used, simply by counting tokens within a given radius. Although the convergence of Monte Carlo estimates is a strong assumption, it is also extremely well-established in the literature.

In short, if we take the logarithm of both the volume and radius in Equation 2 , we obtain a linear curve for a manifold. Abrupt slope changes imply that the space is not a manifold. For a fiber bundle, we obtain a piecewise linear curve, in which the slopes are given by \(\dim S < \dim T\), and the slopes must decrease through a discontinuity as the radius increases, as shown in Figure 2.

3.2 Context does not persistently resolve singularities↩︎

The remaining question is whether singularities can persist into the output of the LLM. Can the user elicit the instabilities caused by singularities? We provide sufficient conditions for unstable outputs:

As a consequence, there always exists such an \(m\) satisfying the hypothesis if \(2d < \ell\), i.e. the bounding manifold has dimension less than half the latent space dimension. If our hypothesis tests show that the token subspace contains singularities and the hypotheses of Theorem 2 are satisfied, singularities will persist into the output of the LLM for generic transformers even for large context window sizes.

4 Methods↩︎

The tests we propose are implemented via Algorithm 3. The coordinates of the points (token embedding vectors, in Section 5.2) are fixed when we analyze the data, and they do not change from run to run. Algorithm 3 is fully deterministic, so there is no requirement of multiple runs. Centered at a given point, the sorted list of distances to all other points contains the radii at which the Monte Carlo estimate of the volume of a ball increases. The corresponding volumes are simply the indices of this list. Both tests estimate dimension as slope in the log-log progression of the entries versus the indices (\(3\)-point centered difference estimates via numpy.gradient()), and detect slope changes.

5 Results↩︎

To demonstrate the validity of our tests in practice, we demonstrate them on three synthetic datasets in Section 5.1 shown in Figure 4 (a)–(c). We then turn to the main thrust of the paper, testing LLM token embeddings in Section 5.2. Source code is available at [48].

5.1 Synthetic datasets↩︎

Figure 4 (a) shows the result of applying our manifold hypothesis test to the unit sphere in \(\mathbb{R}^3\). Since the sphere is a manifold, the \(p\) value for every point tested is close to \(1\), and no rejections of either hypothesis occur.

Figure 4 (b) shows the result of applying the manifold test to a portion of the surface given by \[z^8 + x^2 + y^2 = 0, \text{ where } x^2+y^2 \le 1.\] This surface is a fiber bundle in which the base space is simply the origin, and the fiber spaces are the radial line segments tracing out from the origin to the edge of the disk. It is a manifold with boundary (not a manifold), and the origin \(x=y=z=0\) is a cusp singularity. The number of points participating in the cusp and boundary is small (\(41\) points) compared with the total number of points in the dataset (\(1200\) points).

The manifold hypothesis test strongly detects the cusp singularity in Figure 4 (b), with \(p < 5\times 10^{-8}\). The manifold test fails to detect the boundary (\(p > 0.001\)). While using a larger \(v_{max}\) in Algorithm 3 improves the test’s sensitivity to the point where the boundary is detected, it also reduces the specificity of the test for identifying which points are singularities. On the other hand, the fiber bundle test is not rejected at all, which is consistent with the space being a fiber bundle.

Finally, Figure 4 (c) shows a sample of \(3000\) points drawn from a space shaped like a candy “lollipop,” constructed by attaching a line segment (the “stick”) to a point on the surface of a hollow sphere (the “candy”). The lollipop is neither a manifold nor a fiber bundle. On the interior of the stick, the space is a \(1\)-dimensional manifold. The candy is a \(2\)-dimensional manifold, except where the stick attaches. The entire space has exactly two singular points. At the very end of the stick is a singular point (a boundary point). Where the stick attaches to the candy is another singularity; it is a transition from the \(1\)-dimensional stick to the \(2\)-dimensional candy.

Running the manifold hypothesis test on the lollipop reveals low \(p\)-values on points near the boundary of the stick (\(75\) points) and at the transition (\(58\) points) (both have \(p< 10^{-5}\)). Due to random sampling, the exact points of singularity are not actually present in the dataset; these detections are near the two singularities. None of the points near the boundary point of the stick cause the fiber bundle hypothesis to be rejected with \(p < 10^{-5}\), while \(2\) points near the transition cause the fiber bundle hypothesis to be rejected with \(p < 10^{-5}\). Since even one rejection is sufficient, the tests correctly identify that the lollipop is neither a manifold nor a fiber bundle.

5.2 LLM token embeddings↩︎

We applied our Algorithm 3 to the token subspaces of the four open source LLMs introduced in Sec. 1.1: GPT2, Llemma7B, Mistral7B, and Pythia6.9B. In each LLM we tested, Table [tab:summary] exhibits evidence that the token subspace is not globally a manifold because the local dimension is highly variable (“dim.” columns). Moreover, we can infer that they are either not fiber bundles or have high curvature locally; in either case—violating the hypotheses or having high curvature—model behavior near certain tokens will be highly unstable (low \(p\)-values in the “rejects” columns). Tables 4–9 in the Supplementary material list every violation we found. Additionally, Figure 4 (d) shows the application of the manifold test to the tokens of Mistral7B² in the vicinity of the token ember. The manifold hypothesis is rejected at various tokens (low \(p\)-value, shown in darker color).

For each LLM we studied, we chose two pairs of \(v_{min}\) and \(v_{max}\) parameters in Algorithm 3: one yielding a small radius neighborhood and one yielding a larger radius neighborhood. These parameters were chosen by inspecting a small simple random sample of tokens, with the aim of identifying where changes in dimension were likely to occur. See Section 8.6 for details. This configuration results in three tests: a single test for the manifold hypothesis, the fiber bundle test for small radius, and the fiber bundle test for large radius.

Table [tab:summary] shows the results of our hypothesis tests for the four models we analyzed. The models consequently have token input embeddings with vastly different topology, and all of them exhibit highly significant rejections of the manifold hypothesis. GPT2, Llemma7B and Mistral7B also reject the fiber bundle hypothesis. The rejections of the fiber bundle hypothesis are more frequent in the larger radius region than the smaller radius region, which is consistent with the polysemy interpretation of [26].

The “rejects” columns of Table [tab:summary] give the number of tokens that fail our test (in the top rows of each cell). The listed \(p\) is the overall \(p\)-value after Bonferroni correction. The “dim.” column shows the quartiles for the estimates of dimensions, at least for the tokens for which a dimension can reliably be ascribed. Note for each of the tokens listed in Tables 4–9, no dimension estimate is possible.

Since there are many rejections of the manifold hypothesis, we summarize some general trends, obtained by manual inspection of Tables 4–9 in the Supplementary material, to which we direct the reader. The GPT2 tokens at irregularities are tokens that can only appear at the beginning of words. The Pythia6.9B tokens at irregularities are nearly all word fragments or short sequences of text that are quite meaningless on their own. The Llemma7B and Mistral7B tokens at irregularities are a combination of the previous two: either they can only appear at the beginning of words or they are word fragments.

While most of the tokens are not shared between the LLMs, Llemma7B and Mistral7B have identical token sets. The fact that Table [tab:summary] shows significant differences between these two models indicates that the structure of the irregularities for these two models is quite different. This alone implies that their response to the same prompt is expected to be markedly different, even without considering the differences in their respective transformer stages, corroborating the results shown by [20].

Estimates of the bounding manifold dimension used in Theorem 2 for the four LLMs we studied are in the “dimension” columns of Table [tab:summary], with the most stringent being the dimension for smaller radius. Table 2 shows the results of testing the hypotheses of Theorem 2 using the medians (Q2). Mistral7B and Pythia6.9B satisfy the hypotheses of Theorem 2 using the small radius dimension estimates, but GPT2 and Llemma7B do not. All four LLMs satisfy the hypotheses of Theorem 2 if we instead use the larger radius dimension estimates. This means that it is likely that the transformers of Mistral7B and Pythia6.9B are unable to resolve the singularities that we show exist in the token subspace, so singularities will persist in the output of the model. If the true bounding manifold for GPT2 or Llemma7B is smaller than the small radius estimate suggests, these too will likely produce singularities in their output.

6 Discussion↩︎

The large distribution of local dimension (Figure 1 and Table [tab:summary]) rejects that the token embedding is a globally smooth manifold for each of the LLMs we studied. Beyond non-constant dimension, we also uncovered token neighborhoods without a well-defined intrinsic dimension, a structure not previously reported in these embeddings. These irregular structures suggest that the topology of the embeddings are more complicated than generally assumed. Consequently, all four token subspaces cannot locally be manifolds with large reach, and three cannot be fiber bundles with large reach. Given the aggregation of training data across domains (and even across natural languages), the potential for internal contradictions (contronyms) and confounders (natural language, tweets, source code), it is not surprising to see that our hypotheses are rejected. The semantic role of tokens likely contributes to LLM sensitivity of prompts that has been observed in the literature ([24], [25], [49]–[52]) and may underscore why elucidating LLM behavior remains challenging and open. The manifold hypothesis gave both an intuitive and mathematically grounded explanation for the behaviors of learning machines ([53]–[55]) yet the instability of LLM outputs strongly suggested that these models were different. Our work gives conclusive empirical evidence against the manifold hypothesis for token embeddings and may offer novel ways to interpret LLM behavior as we are not in a regular setting.

Such irregularities are noticeable in a principal components analysis (PCA) projection of the token subspace (Figures 4 (d), 8 and 9) but are extremely hard to interpret a priori. Mathematicians are still classifying singularities in \(\mathbb{R}^3\)([56]) implying that the classification problem is far from complete (even in \(\mathbb{R}^3\)). Intuitively, this means that the use of inner products to perform token de-embeddings by orthogonal projections may not be accomplishing the intended objectives. Even small perturbations within the latent space may result in instabilities in these projections, providing potential explanations for the instabilities in LLM outputs. In addition to token instability, the fact that token subspaces are not manifolds and may have small reach means that the geodesic distance between tokens can also be very unstable. While the distance along geodesics can be defined, it may not correlate with any sense of semantic distance between tokens. Furthermore, as [43] found, in most of the models, there are tokens with dimension \(0\) neighborhoods. These tokens are theoretically isolated, implying that the token subspace is disconnected and hence intrinsically infinitely far from other tokens.

Singularities may arise either as artifacts of the training process or from features of the languages being represented. Consistent with the idea that polysemy may yield singularities ([26]), several of the tokens where the fiber bundle hypothesis is rejected are clear homonyms. For instance, both “affect” and “monitor” can be used as either nouns or verbs, and their meanings are different in these two roles. A token may also correspond to a pair of homonyms after the addition of a prefix or suffix. A token like “aunder” can be prefixed to yield the word “launder”, a word with multiple meanings of opposite sense. Specifically, one can “launder” clothing (positive connotation) or “launder” money (negative connotation). Several other tokens where the fiber bundle hypothesis is violated form words with substantially different meanings or grammatical roles upon adding a prefix or suffix. For instance, “wins” can function as a noun, as a verb, or form the adjective “winsome”.

The differences in how the manifold and fiber bundle hypotheses are rejected across different LLMs suggest that the training methodology for each model leaves an indelible fingerprint (or gDNA coined by [20]). Making general assertions about LLMs without consideration of their training and evolution is likely fraught. Even between Llemma7B and Mistral7B, which have identical token sets, prompts likely cannot be “ported” from one LLM to another without significant modifications if they contain tokens near irregularities, likely a consequence of the invariance of domain theorem. This may have serious ramifications for interacting LLMs such as those powering multi-agentic frameworks being deployed today ([57]–[59]).

Tokens that begin a word or are a word fragment are often located at an irregularity. Additionally, in Llemma7B (but not Mistral7B) and Pythia6.9B, the tokens with unusually low fiber dimension often contain non-printing or whitespace characters. This suggests that these models are quite sensitive to text layout, perhaps to the exclusion of more semantically salient features in the text. Given our findings, future experiments can be run to explore the impact of irregular tokens on the variability of responses produced by different LLMs.

A natural question is if we can find a token embedding that is everywhere regular. One way to approach an answer is to view the number of tokens in a vocabulary as a hyperparameter. There might be a certain vocabulary size that minimizes the number of singularities given the corpus, architecture, etc. Even the best vocabulary might lead to a space that could still be singular, for reasons inspired by [26]. Consider the symbol, which happens to be singular for Mistral7B (See Table 7, number 29). In natural language, it has a very distinctive meaning in terms of calling out footnotes, while in other contexts it can be used for multiplication or marking text as bold (in Markdown). Therefore, this character connects these different semantics. Suppose that each language were in fact well-described by a manifold. Then these manifolds would be joined together in the entire latent space at the (and likely other points). As a result, this token with high likelihood would be a singularity because the dimensions for different languages are almost surely different ([46]). In reality, tokens are neither in 1:1 correspondence with words nor with grammatical symbols. Not only do parts of words map onto a single token, but tokenization is a many to one map across the board. How well context / attention / transformers can unfurl these mappings is a big part of the mystery of how these models work. Identifying and analyzing the irregularities may help us understand the approximate limits of attention/transformers architectures.

As we attempt to model natural phenomena particularly in our physical world, we know singularities exist (e.g. phase transitions), and so assuming regularity limits our understanding of the natural world. Our work extends this notion to the linguistics of natural language where its richness and complexity do not appear to manifest in a smooth representation.

7 Limitations↩︎

While topological theory motivates our statistical test, only the slope changes that occur for radii less than the embedded reach correspond to rejections of the manifold or fiber bundle hypotheses. If the reach of a manifold is small, then it likely has a high local curvature, and measuring distances on that surface will be numerically unstable. And lastly, we emphasize that these tests allow us to reject the structure assumed in our null hypothesis (smooth and regular), but do not permit us to infer an alternative. While rejecting regularity does imply singularity, it does not afford us any evidence as to the type of singularity. That must be determined post hoc if possible.

Acknowledgments and Funding Disclosure↩︎

The authors would like to thank Mohammed Abouzaid, Anand Sarwate, Andrew Lauziere, and Andrew Engel for helpful suggestions on a draft of this manuscript.

This material is based upon work partially supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001124C0319. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA). Distribution Statement “A” (Approved for Public Release, Distribution Unlimited) applies to the portion of the work funded by DARPA.

The authors have no competing interests to declare.

8 Supplementary Material↩︎

8.1 Additional discussion of the literature↩︎

Although there a number of mathematically distinct definitions of dimension, they coincide for manifolds. Intrinsic dimension of a point cloud has been widely studied in the literature [60]. All of the papers that address practical estimation of intrinsic dimension from data do so under the assumption that the data lie on a manifold. For instance, [61] provides maximum likelihood estimates, operating under the assumption that data “can be efficiently summarized in a space of a much lower dimension, such as a nonlinear manifold.” Moreover, [62] provides a surprisingly effective approach using extremely small neighborhoods, a fact which underlies our use of our test on a “small radius neighborhood” in Section 5.2.

Even though we ultimately reject the manifold hypothesis, the token subspace has been studied under the assumption that it is a manifold. For instance [63] observe that,“Language models often construct similar local representations as constructed from local linear embeddings.” We note that most of the tokens in a typical vocabulary fail to reject the manifold hypothesis, which means that without the deeper analysis we pursue, it is easy to overlook important differences. Furthermore, our Table [tab:summary] shows significant topological differences in the local representations between models.

As introduced in Section 1.2, we mentioned that [63] show a few things that are consistent with our findings. They note that “Token embeddings exhibit low intrinsic dimensions,” which agrees with our in Table [tab:summary], at least in the large radius neighborhood. Their estimates of dimension for GPT2 is within the confidence interval we computed. They do not recognize that there might be a fiber bundle structure in play, so [63] does not estimate dimensions from a smaller radius neighborhood.

Relevant to Theorem 2, there is a possibility that the implicit space of possible activations can be inferred from the behavior of the model [64]. Since Theorem 2 represents the context window as a product of copies of the latent space, a projection of this implicit structure is the token subspace. In this vein, [64] is not inconsistent with other findings within the literature, for instance [43]. This latter paper shows that the token subspace can be inferred (up to topological equivalence) from the responses of the LLM to single-token prompts. Taken together, [43], [64] and the present one have pretty significant implications. Even though the token subspace may have about 0.1% of tokens that are singular, we suspect that a user will be encountering singular tokens on a regular basis when using LLMs (which seems empirically plausible). Estimating the rate of occurrence of a singular token remains an open question.

8.2 LLM Background↩︎

At an abstract but precise level, an LLM consists of several interacting processes, as outlined in Figure 5. An LLM implements a transformation of a sequence of tokens (the query) into a new sequence of tokens (the response). Formally, if each input token is an element of a metric space \(T\), then the LLM is a transformation \(T^w \to T^m\), where \(w\) is the number of tokens in the query and \(m\) is the number of tokens in the response. This transformation is typically not a function because it is stochastic—it involves random draws.

To operate upon tokens using numerical models, such as could be implemented using neural networks, we must transform the finite set of tokens \(T\) into numerical data. This is typically done by way of a pair of latent spaces \(X = \mathbb{R}^{\ell}\) and \(Y=\mathbb{R}^q\). The dimension \(q\) of \(Y\) is chosen to be equal to the number of elements in \(T\), so that elements of \(Y\) have the interpretation of being (unnormalized) probability distributions over \(T\).

The transformation \(T^w \to T^m\) is constructed in several stages.

Input tokenization

: Each token is embedded individually via the token input embedding function \(e: T \to X\). As a whole, \(X^w\) is called a latent window.

Transformer blocks

: The probability distribution for the next token is constructed by a continuous function \(f : X^w \to Y\). This is usually implemented by one or more transformer blocks.

Output tokenization

: Given the output of one of the transformer blocks \(f\), one can obtain an output token in \(T\) by a random draw. Specifically, if \((\psi_1, \psi_2, \dotsc, \psi_w)\) is the current window in \(X^w\), then the next token \(t\) is drawn from the distribution given by \(f(\psi_1, \psi_2, \dotsc, \psi_w)\).

Next window prediction

: Given that token \(t\) was drawn from the distribution, the next latent window itself is constructed by a transformation \(F : X^w \to X^w\), which advances the window as follows: \[F(\psi_1, \psi_2, \dotsc, \psi_w) := (\psi_2, \dotsc, \psi_w, t).\]

Note well: the function \(e^w : T^w \to X^w\) which transforms a sequence of individual tokens into coordinates in the latent space is merely the first stage in the process. As such, \(e^w\) is distinct from the embedding of sequences provided by many LLM software interfaces. That sequence embedding is properly an intermediate stage within the function \(f\).

The focus of this paper is specifically upon the structure of the token input embedding \(e: T \to X = \mathbb{R}^{\ell}\). Since the token set \(T\) is finite, \(e\) can be stored as a matrix. In this matrix, each column corresponds to an element of \(T\), and thereby ascribes a vector of numerical coordinates to each token. By replacing the last layer of the deep neural network \(f: X^w \to Y\), a vector of probabilities for the next token is obtained from the activations of the last layer. One can therefore interpret the probabilities as specifying a token output embedding. Both the tokenization and the transformer stages are learned during training, and many strategies for this learning process are discussed extensively in the literature. These two stages interact when they produce the LLM output, so it is important to understand the lineage of a given tokenization as being from a particular LLM.

8.3 Computational resource requirements↩︎

Applying our entire method (starting with the token embedding matrix and ending with the \(p\)-values for each of the three tests at every token) to each model took approximately 12 hours of wall clock time (per model) on an Intel Core i7-3820 with 32 GB of CPU RAM and no GPU running at 3.60GHz. Almost all of the runtime was spent computing the token pairwise distance matrix \(D\) in Algorithm 3. The distance matrix calculation was performed using scipy.spatial.distance_matrix(), which produces a dense matrix, by computing the distance between all pairs of tokens in the latent space. Because of the relatively low runtime requirements, we did not feel the need to use sparsity-enhancing techniques to accelerate the distance matrix calculation. The subsequent stages, including the three hypothesis tests run in only a few minutes on the same machine.

8.4 Interpretation of fiber bundles in terms of spatial variability↩︎

It is usual to describe measurements as exhibiting the combined effect of multiple spatial scales. One way this might arise is if we could express each measurement as being an ordered pair (signal, noise), with the short spatial scales correlating to noise and the longer spatial scales correlating to the signal. With this interpretation, if the space of all possible signals is \(S\) and the space of all noise values is \(V\), we could represent the space of all possible measurements as the cartesian product \(E=S \times V\). This cartesian product representation can arise for noiseless data as well, as it merely captures multiple spatial scales. In what follows, we will call \(S\) the base space and \(V\) the fiber space.

Figure 6 shows an example of this situation. It consists of a \(1\)-dimensional base space (the large radius neighborhood) and \(1\)-dimensional fibers (the small radius neighborhood), which in this case forms a narrow strip in the plane. Volumes (areas, in this case) of balls of small radius scale quadratically (slope \(2\) in a log-log plot), but scale asymptotically linearly (slope \(1\) in a log-log plot) for large radii. The transition between these two behaviors is detectable by way of a corner in the plot. This situation is easily and robustly estimated from the data; the gray points in Figure 6 (right) are derived from a random sampling of points drawn from the strip.

To test for multiple spatial scales, we propose a signal model that is mathematically represented by a fiber bundle. In a fiber bundle, the long spatial scales are modeled by a space \(S\), but all possible measurements are modeled by a function \(p : E \to S\). The idea is that fibers \(p^{-1}(b)\) are still cartesian products: pairs of both spatial scales, and these are all identical up to smooth changes of coordinates. Our method relies upon a particular geometric property of fiber bundles: we can identify if the fibers are not all identical according to when the conclusion of Theorem 1 is violated.

One possible way that a fiber bundle might arise is if the underlying noise model is homoscedastic, which is to say that the dimension of the noise near a token does not depend on that token. In this case, the fibers (small radii) mostly correspond to noise, and the base (large radii) correspond mostly to signal. Nevertheless, a fiber bundle may arise with noiseless data as well; the presence of multiple spatial scales is what matters.

On the other hand, if the underlying noise model is heteroscedastic—it depends on the token in question—then the fiber bundle model should be rejected. Figure 2 shows a situation that is not a fiber bundle, since there is a change in the dimension of the fiber. In the left portion of the figure, the fiber dimension is \(0\) while in the right portion the fiber dimension is \(1\). This is detectable by looking at the volume versus radius plots for two samples. While both samples show corners in their volume versus radius plots, Theorem 1 establishes that the slopes always decrease with increasing radius for a fiber bundle. This is violated for the sample marked \(\psi_1\), so we conclude that the space is not a fiber bundle. On the other hand, because the sample marked \(\psi_2\) does not exhibit this violation, it is important to note that if a sample yields data consistent with Theorem 1, we cannot conclude that the space is a fiber bundle.

8.5 Mathematical proof of Theorems 1 and 2↩︎

This section contains mathematical justification for the fiber bundle hypothesis proposed earlier in the paper and the proof of Theorems 1 and 2. The central idea is the use of a special kind of fiber bundle, namely a fibered manifold with boundary.

By the submersion theorem ([65]), if the Jacobian matrix of a fibered manifold \(p: T \to S\) at every point has rank equal to the dimension of \(S\), then the preimages \(p^{-1}(\psi) \subseteq T\) of each point \(\psi \in T\) are all diffeomorphic to each other. These preimages form the fibers discussed in the earlier sections of the paper.

As a consequence, each point \(y\) in the base space \(S\) has an open neighborhood \(U\) where the preimage \(p^{-1}(U)\) is diffeomorphic to the product \(U \times p^{-1}(y)\), which is precisely the base-fiber split discussed in Section 8.4. Specifically, the base dimension is simply the dimension of \(S\), whereas the fiber dimension is the \((\dim T - \dim S)\).

The notion of a fibered manifold \(p : T \to S\) forms the intrinsic model of the data, which is only implicit in an LLM. The tokens present in a given LLM can be thought of as a sample from a probability distribution \(v\) on \(T\), which can be taken to be the Riemannian volume form on \(T\) normalized so that \(v(T)=1\).

It is a standard fact that if \(e\) is a fibered manifold or an embedding, then \(e_* v\) is also a volume form.

Notice that the second case in Equation 2 may be precluded since while it holds for small \(r\), it may be that \(\rho(\psi)\) may not be sufficiently small. As a consequence, the second case only occurs when both \(r\) and \(\rho(\psi)\) are sufficiently small. In the results shown in Section 5.2, both cases appear to hold frequently.

We now address Theorem 2. Informally, we cannot get rid of all stochasticity during training (due to finite precision, stochastic gradient descent, etc.). This ensures that the transformers in practice are always in general position, hence generic.

Given a context window of \(w-1\) tokens, an LLM uses its learned parameters to predict the next (\(w\)-th) token. This gets added to the context window and the process continues. (See Section 8.2 for details.) In this section, we provide a mathematical argument showing that the use of context is insufficient to resolve singularities present in the token subspace for a generic transformer in a persistent manner. Again, denote the token subspace by \(T\), noting that Section 5.2 shows that \(T\) is unlikely to be a manifold for typical LLMs. Let us begin with a lemma:

Consequently, the contrapositive of Lemma 1 states that if \(T\) is not a manifold but \(T^{w-1}\) is a manifold, it follows that \(T^w\), the space of sequences of tokens, cannot be a manifold. At best, the latent space for every other context window size can be a manifold. For every proposed context window length, a longer window will exhibit singularities. In short, lengthening the context window does not persistently resolve singularities in the token embedding.

For instance, if we use the embedding interface provided by the LLM to embed a sequence of \(w\) tokens, we will obtain a vector in the latent space \(\mathbb{R}^{\ell}\). On the other hand, if we embed each of the tokens individually yet sequentially, we obtain a vector in \(\mathbb{R}^{\ell \times w}\). The transformer uses the vector in \(\mathbb{R}^{\ell \times w}\) to determine the vector in \(\mathbb{R}^{\ell}\). Theorem 2 states that the latter (\(\mathbb{R}^{\ell \times w}\)) is crucial in understanding the output of the LLM. (See Section 8.2 for further details.)

8.6 Case study of some tokens in GPT2↩︎

Figure 7 demonstrates our method on three tokens used by GPT2. The tokens $ and # play an important role in many programming languages, whereas does not. (Note that these tokens were chosen for illustrative purposes. The \(p\)-value for rejecting the hypotheses is larger than \(\alpha = 10^{-3}\) used in Section 5.2, so these particular tokens do not appear in any of the tables in the following sections.) The token # does not show any rejections of our hypotheses. While this does not allow one to conclude that the vicinity of # is a manifold or fiber bundle, we can use Theorem 1 to estimate the dimension from the slope of the curve (approximately \(53\), once logarithms of the values on both axes are taken). The token exhibits a rejection of the manifold hypothesis, but not the fiber bundle hypothesis. We can conclude that there is a irregularity at a distance of roughly \(3.9\) units of . The curve for $ exhibits two slope changes at radii \(3.8\) and \(4.0\). The smaller radius slope change represents a violation of the manifold hypothesis, while the large radius slope change represents a violation of the fiber bundle hypothesis.

Given the presence of multiple slope changes, it is necessary to select the parameters \(v_{min}\) and \(v_{max}\) in Algorithm 3 carefully. Ideally, one aims to capture at most one slope change between \(v_{min}\) and \(v_{max}\). This is not specifically required by Algorithm 3, since it uses a sliding window, multiple rejections may be detected.

8.7 Case study of a neighborhood of ember in Mistral7B↩︎

Figures 8 and 9 show the application of this paper’s manifold hypothesis test to a neighborhood of the token ember in Mistral7B. Since the latent space of Mistral7B is \(4096\), we have projected the coordinates of each token shown to \(3\) using PCA. The inset shows all of the tokens in Mistral7B (rendered with t-SNE), with a crosshairs showing the location of the zoomed-in region. Each point shown in Figure 8 is a token used by Mistral7B, colored based upon the \(p\)-value for the manifold hypothesis test. Darker colors correspond to lower \(p\)-values, which are less consistent with the hypothesis that the token’s neighborhood is a manifold with low curvature.

Given that the we have established that manifold rejections occur in Section 5.2, we can attempt to understand possible causes. Intuitively, several non-manifold features are visibly present⁴ in Figure 8. The tokens “ember” and “P” appear to be both putative examples of cusp points, places where the space exhibits a sharp exposed point. Manifolds cannot have cusp points, so attempting to approximate a cusp with a manifold—for instance, with UMAP—necessitates high curvature. The tokens “:” and “ly” appear to be pinch points—like what are present in a chain of sausages—which again cannot be approximated by a manifold unless it has high curvature.

8.8 Table of fiber bundle rejections in all four LLMs↩︎

See Table 3.

8.9 Tables of manifold rejections in all four LLMs↩︎

See Tables 4–9.

8.10 Limitations of theory↩︎

Because our approach requires the distances between all pairs of tokens, to use the hypothesis tests presented in this paper, one needs to have access to the token embedding vectors. This is why our paper considers four open source models of moderate size – GPT2 ([29]), Llemma7B ([30]), Mistral7B ([31]), and Pythia6.9B ([32]). While [66] does provide an approach to extract the token embeddings using systematic prompting, which would allow the application of our tests to proprietary models, their method requires exquisite control of the prompting and is computationally expensive.

The reach itself is difficult to estimate accurately from sampled data ([27], [67], [68]). Our implementation tests the radius below an estimated reach, determined by an inspection of the radius-vs-volume plots of a random sample of tokens. If the true reach is smaller than our estimates, the curvature of the manifolds that fit the data well, found by any method, will be high.

References↩︎