1 Introduction↩︎

Transformers are neural networks built around self-attention mechanisms [1]. They have played a central role not only in natural language processing but also across machine learning more broadly. Also, recent work on looped or depth-recurrent Transformers, which performs inference by repeatedly iterating through layers that share the same weights, has attracted attention due to its high performance [2]–[6]. Motivated by their empirical success, a growing theoretical literature seeks to clarify the mathematical principles behind the evolution of token representations [7]–[11]. In particular, a shared-block regime is investigated from aspects of length generalization, expressive power, recursive latent reasoning, and compositional generalization [2], [4], [5].

A fruitful viewpoint treats repeated self-attention updates as interacting-particle dynamics and studies their mean-field limits [12]–[15]. In particular, [13], [14] formulate simplified self-attention models as interacting particle systems on the sphere and analyze long-time clustering and limiting configurations. Moreover, [16] shows that these dynamics can exhibit metastability: although they ultimately converge to a single cluster, they may remain near multi-cluster states for very long time scales. This line of work provides a mathematically precise route from iterative self-attention updates to gradient-driven dynamics governed by an energy functional.

A recurring conclusion of this framework is degeneracy or mode-collapse of token distributions. In the variational analysis above in regard to the dynamics of distributions of tokens, long-time limits and maximizing configurations often collapse to Dirac measures or single clusters [13], [14], [16]. In other words, according to these analyses, long inferences performed by (loop-based) Transformers ultimately lead to a collapse to a single point. While there are differing views on this issue based on experience, oversmoothing is not inevitable [17], and masking together with layer normalization can prevent simple rank-one collapse [18], it has yet to be settled. Based on this situation, we pose the following research questions: What are effective methods for preventing mode collapse in mean-field-based transformer models? Can we provide a theoretical basis for these methods?

1.1 Contributions↩︎

This paper endeavors to fill the aforementioned gap by extending the mean-field theoretical framework such as [13] to include the variables that are used in actual practice but are often omitted from the analysis. In particular, in addition to the dynamics of token variables that has been considered in depth, we include Positional encoding and prefix tokens inserted at inference time [19], [20] as tangible probabilistic auxiliary variables, and consider the joint law of the token and auxiliary variables with a fixed auxiliary marginal. More generally, we develop the framework of what we call Unnormalized Self-Attention model with Auxiliary Variables (USA-AV) and show that, with sufficient regularity, the auxiliary variables prevent the token distribution from collapsing. Figure 1 outlines the result shown by our theory.

The main contributions of this paper are as follows.

• A fixed-auxiliary variational framework for anti-collapse. We formulate self-attention with auxiliary variables (e.g., positional encodings and prompt prefixes) in the USA-AV model, which describes the dynamics of the token variable under a fixed auxiliary marginal through joint laws.

• Conditional Dirac structure with marginalized noncollapse. Under a fixed auxiliary marginal, we show that maximizers may still be Dirac in the conditional content variable at each auxiliary value, while the marginalized content distribution remains non-Dirac. This identifies the precise sense in which auxiliary variables prevent collapse: not by destroying conditional Dirac structure, but by allowing the Dirac location to move with the auxiliary variable.

• Positional encoding yields orbit-wise exact realization of a target law. For positional variables, we show that RoPE-type kernels already generate spread maximizers, and that arbitrary measurable phase fields go further: once a rotation orbit is fixed, every probability law on that orbit can arise exactly as a maximizing marginalized output law.

• Prefix tokens upgrade this to gauge-level exact realization of a target law. For prefix tokens, we prove that any \(\eta\)-pushforward law generated by a measurable map into the sphere can be realized exactly as a maximizing marginalized output law through an orthogonal gauge; when the prompt space is standard Borel and the prompt law is nonatomic, this yields exact realization of arbitrary target laws on \(\mathbb{S}^{d-1}\).

The results above suggest that positional encodings and prefix tokens function not merely as side information, but as genuine anti mode-collapse mechanisms inside the idealized shared-weight self-attention model analyzed here. Most importantly, these anti mode-collapse mechanisms can be designed with appropriate kernels to produce a target distribution through the marginalization of the auxiliary variables. Throughout, the phrase looped transformer-type self-attention dynamics is meant to emphasize this scope restriction: the theorems concern a shared-weight surrogate, not an arbitrary full Transformer architecture. In Sections 3 and 4 we refer informally to the last two principles as orbit-wise and gauge-level universality, respectively, but the actual mathematical statements are exact realization results for maximizing marginals rather than universal approximation statements for sequence-to-sequence maps.

1.2 Notation↩︎

We use \(\|\cdot\|\) for norms, \(\langle\cdot,\cdot\rangle\) for Euclidean inner products, \(\delta\) for a Dirac measure, \(C^1\) for continuously differentiable functions, \(\mathcal{P}(\Omega)\) for probability measures on \(\Omega\), and \(W_1\) for the \(1\)-Wasserstein distance. For a measurable map \(G:\Omega\to\Omega'\) and a measure \(\nu\in\mathcal{P}(\Omega)\), we denote by \(G_{\#}\nu\) the pushforward of \(\nu\) by \(G\), defined by \((G_{\#}\nu)(B)=\nu(G^{-1}(B))\) for every Borel set \(B\subset\Omega'\). Let \(\mathbb{S}^{d-1}:=\{x\in\mathbb{R}^d:\|x\|=1\}\) be the unit sphere. We write \(\mathop{\mathrm{div}}_x\) for the surface divergence in the spherical variable, \(\sigma_{\mathbb{S}^{d-1}}\) for the uniform probability measure on \(\mathbb{S}^{d-1}\), and \(\mathbb{T}:=\mathbb{R}/\mathbb{Z}\) for the one-dimensional torus. We write \(\pi_x,\pi_{\mathcal{A}},\pi_{\mathcal{Z}}\) for the coordinate projections. \(\beta > 0\) denotes the inverse temperature.

2 Fixed-Auxiliary Variational Framework↩︎

2.1 Preparation: Mean-Field Framework for Analyzing Transformers↩︎

We first introduce the framework of mean-field transformers, which arises from a standard self-attention block [1]. For a single-head attention layer with \(n\) token inputs, one may write an update of the \(i\)-th token \(x_i^\ell\), \(i=1,\dots,n\), at an \(\ell\)-th layer as \[x_i^{\ell+1}=\mathrm{LN}\left(\sum_{j=1}^n \alpha_{ij}^{\ell} W_Vx_j^{\ell}\right),\] where \(\alpha_{ij}^{\ell}\propto\exp(\langle Qx_i^{\ell},Kx_j^{\ell}\rangle)\), \(Q,K,W_V\) are weight matrices, and \(\mathrm{LN}(\cdot)\) denotes the (RMS) layer normalization without a scale parameter.

The interacting-particle and mean-field viewpoints developed for Transformer self-attention [14], [21] justify the following idealization: take tied layers and a small residual step, use layer normalization to keep \(x_i^\ell\) on the sphere, impose a symmetric query–key structure and absorb the value map into coordinates, i.e., \(Q=K=W_V=I\), and replace the softmax denominator by a fixed normalization or time rescaling. In the resulting continuous-depth limit, the token states follow an interacting system of the form named the unnormalized self-attention (USA) model for the \(i\)-th token \(x_i\): \[\dot{x}_i=P_{x_i}^{\perp} \left(\frac{1}{n}\sum_{j=1}^n \exp(x_j^\top x_i) x_j\right), \qquad x_i \in \mathbb{S}^{d-1}.\] Here, \(P_{x_i}^{\perp}\) denotes the projection onto a tangent space of the sphere. This is exactly the unnormalized self-attention flow used below.

2.2 USA-AV Model and its Dynamics↩︎

Let \((\mathcal{A},d_\mathcal{A})\) be a compact metric space, and let \(\rho\) be a Borel probability measure on \(\mathcal{A}\). We set \(X:=\mathbb{S}^{d-1}\times \mathcal{A}\), where the content variable is \(x\in\mathbb{S}^{d-1}\) and the auxiliary variable is \(\xi\in\mathcal{A}\). Throughout, we consider admissible joint laws preserving the auxiliary marginal \(\rho\). In Section 3, dedicated to the role of positional encoding, we specialize to \(I:=(0,1]\) and set \(\mathcal{A}=I\). For Section 4, dedicated to the role of prefixes, we continue with the idea of fixed marginals and consider a standard Borel prompt space \(\mathcal{Z}\) with Borel probability law \(\eta\).

The point of this section is to record only the minimal dynamical and variational structure needed for the main results. In particular, the interaction kernel is reused along depth/time, so the model should be read as a shared-weight or looped-block surrogate [2]–[5] rather than as a full untied-depth transformer. When a metric on \(X\) is needed, we use \(d_X\bigl((x,\xi),(y,\zeta)\bigr):=\|x-y\|+d_\mathcal{A}(\xi,\zeta)\).

We first specify the class of kernels used throughout the fixed-auxiliary framework.

Condition [H3] is the main dynamical assumption: it guarantees that, after averaging over the other particles, the force field remains uniformly Lipschitz in the content variable. Throughout the finite-particle system the auxiliary labels are fixed and only the content variables evolve.

The auxiliary-blind USA model is recovered by choosing a kernel that ignores the auxiliary labels altogether.

Two positional specializations, obtained by taking \(I:=(0,1]\) and \((\mathcal{A},\rho)=(I, \mathrm{d}s)\), will be used repeatedly later.

Basic ODE well-posedness and the empirical-law weak formulation are standard under Definition 1; for completeness they are recorded as Propositions 2 and 3 in Appendix 11.1.

The empirical law of the particle system is \(\mu^{(n)}_{\tau}:= \frac{1}{n}\sum_{i=1}^n \delta_{(x_i(\tau),\xi_i)}\in\mathcal{P}(X)\). When the empirical auxiliary law \(\rho_n:=\frac{1}{n}\sum_{i=1}^n \delta_{\xi_i}\) converges to \(\rho\), the natural fixed-marginal mean-field class is \(\mathcal{P}_{\rho}:=\{\mu\in\mathcal{P}(X): \mu( \mathrm{d}x, \mathrm{d}\xi)=\mu^\xi( \mathrm{d}x)\rho( \mathrm{d}\xi)\}\). Since \(X\) is a standard Borel space, every \(\mu\in\mathcal{P}_{\rho}\) admits the disintegration \(\mu( \mathrm{d}x, \mathrm{d}\xi)=\mu^\xi( \mathrm{d}x)\rho( \mathrm{d}\xi)\). This fixed-marginal decomposition is the basic structural constraint throughout the paper.

Because the vector field acts only on the content coordinate, the auxiliary label is frozen, while the conditional content law at each label is transported on the sphere by the mean attention force generated by all other conditional laws. In particular, any solution starting in \(\mathcal{P}_\rho\) remains in \(\mathcal{P}_\rho\) for all times.

Under strengthened Lipschitz hypotheses on \(\nabla_x h\) in both arguments, stated in Appendix 11.1, the mean-field equation is globally well posed and satisfies a standard Dobrushin-type stability estimate. In particular, empirical laws converge to the mean-field solution when the initial data converge in \(W_1\); see Proposition 4.

2.3 Energies under a fixed auxiliary marginal↩︎

To extend the variational arguments as in [13] to our framework with \(\mathcal{P}_\rho\), we need only to establish the corresponding mean-field energy and Lyapunov monotonicity. We provide the explicit form of the corresponding energy and monotonicity claims for the discretely supported \(\mu_\tau^{(n)}\) in the Appendix 12.1. In this section we provide two general results to be used later.

When \(\mu( \mathrm{d}x, \mathrm{d}\xi)=\mu^\xi( \mathrm{d}x)\rho( \mathrm{d}\xi)\), this functional simply averages pairwise interactions over the auxiliary labels and the conditional content laws. The point is that the mean-field equation in Definition 3 is an ascent flow for this functional.

3 Anti-Collapse via Positional Encoding↩︎

We now specialize the general USA-AV framework to positional variables. Thus throughout this section \(I:=(0,1]\), \(X:=\mathbb{S}^{d-1}\times I\), and \(\mathcal{P}_I:=\{\mu\in\mathcal{P}(X): \mu( \mathrm{d}s, \mathrm{d}x)= \mathrm{d}s \mu^s( \mathrm{d}x)\}\). For periodic positional kernels we identify \(I\) with the torus \(\mathbb{T}\) by gluing the endpoints. In this section, we discuss two points: (i) RoPE kernels admit exact maximizing laws whose conditional slices remain Dirac while the marginalized content law is genuinely non-Dirac, and (ii) once a rotation orbit is fixed, the marginalized maximizing law represents measures along that orbit.

3.1 RoPE constructions and orbit-wise exact realization↩︎

We consider a special case of the RoPE-type embedding in Example 3: fix \(\beta>0\) and \(\omega\in\mathbb{R}\), and consider \(h_R\bigl((x,s),(y,t)\bigr):=\frac{1}{\beta}\exp \bigl(\beta \langle x, R_{\omega(t-s)} y\rangle\bigr)\); write \(\mathcal{E}_R:=\mathcal{E}_{h_R}\). For a measurable path \(x:I\to\mathbb{S}^{d-1}\), we write \(\Gamma[x]:=(s\mapsto(s,x(s)))_{\#}ds\) for its graph law.

Thus RoPE already gives an exact maximizing configuration whose output law is spread along an orbit rather than collapsing to a single point. The choice of \(u\) is arbitrary, and changing the component of \(u\) orthogonal to the active rotation plane changes the orbit without changing the maximal value. The next theorem shows that the orbit law need not be uniform: every probability law on the chosen orbit can be realized exactly.

More generally, fix a two-dimensional subspace \(E\subset\mathbb{R}^d\), and for each \(\theta\in\mathbb{R}\) let \(R_\theta\in O(d)\) denote the orthogonal transformation that rotates by angle \(\theta\) on \(E\) and acts as the identity on \(E^\perp\). Given a measurable antisymmetric phase kernel \(g:I\times I\to\mathbb{R}\), set \(h_g((x,s),(y,t)):= \frac{1}{\beta}\exp (\beta \langle x, R_{g(s,t)} y\rangle)\) with \(g(t,s)=-g(s,t)\), and write \(\mathcal{E}_g:=\mathcal{E}_{h_g}\). Our goal is to realize prescribed orbit-type maximizers \(x^\psi(s):=R_{\psi(s)}u\) for a fixed \(u\in\mathbb{S}^{d-1}\) and arbitrary measurable phase fields \(\psi:I\to\mathbb{R}\).

Part (ii) precisely describes the phrase orbit-wise universality: once the orbit \(\mathcal{O}_u\) is fixed, every probability law on that orbit can occur exactly as a maximizing marginalized output law. Positional encoding therefore does not merely exhibit one noncollapsed maximizer.

3.2 Energy maximization with conditional Dirac measures↩︎

We present the principle that explains the results obtained above, namely the result of energy maximization using the conditional Dirac distribution. In the auxiliary-variable setting of Section 2, once the auxiliary marginal \(\rho\) is fixed and nonatomic, the diagonal in \(\mathcal{A}\times\mathcal{A}\) is negligible and the energy is affine in each conditional slice \(\mu^\xi\). Maximizers can therefore be Diracized auxiliary value by auxiliary value, and the problem reduces to maximizing over graph laws.

This theorem does not say that the marginalized content law must be a Dirac mass. It only says that, at each fixed auxiliary value \(\xi\), one may choose an optimizing conditional law that is Dirac. If the maximizing point \(x^\ast(\xi)\) varies with \(\xi\), then the marginalized law \(\bar\mu( \mathrm{d}x):=\int_{\mathcal{A}} \delta_{x^\ast(\xi)}( \mathrm{d}x) \rho( \mathrm{d}\xi)\) can be continuous or otherwise nondegenerate.

Specializing Theorem 3 to \((\mathcal{A},\rho)=(I, \mathrm{d}s)\), the optimization reduces to the pathwise problem \[\label{eq:J-path} \max_{x:I \to \mathbb{S}^{d-1}}\widetilde{\mathcal{E}}_h[x(\cdot)], \qquad \widetilde{\mathcal{E}}_h[x(\cdot)] :=\frac{1}{2}\int_0^1\int_0^1 h\bigl((x(s),s),(x(t),t)\bigr) ds dt.\tag{2}\]

This gives a short proof outline for Theorems 1 and 2. In both cases, \(h\bigl((x,s),(y,t)\bigr)\le e^\beta/\beta\) pointwise, so \(\mathcal{E}_h[\mu]\le e^\beta/(2\beta)\) for every admissible law. The candidate paths are then chosen so that this upper bound is attained for every pair \((s,t)\): for RoPE, \(R_{\omega(t-s)}x^\ast(t)=x^\ast(s)\); for the phase-field construction, \(R_{g(s,t)}x^\psi(t)=x^\psi(s)\). Hence every interaction term is exactly \(e^\beta/\beta\), so the reduced energy in 2 attains the maximal value \(e^\beta/(2\beta)\), and therefore the original mean-field energy is maximized as well.

The same conditional-Dirac reduction is also the starting point for the separated-kernel analysis below. If \(h\bigl((x,s),(y,t)\bigr)=b(s,t) k(x,y)\) with \(k\) positive definite and \(\Phi(x):=k(x,\cdot)\) the feature map into an RKHS \(\mathcal{H}\), then by Theorem 3 any maximizer in the positional setting may be written as \(\mu^\ast=\Gamma[x^\ast]\), and with \(m(s):=\Phi(x^\ast(s))\) one has \(\mathcal{E}_h(\mu^\ast)=\frac{1}{2}\int_I\int_I b(s,t)\langle m(s),m(t)\rangle_{\mathcal{H}} ds dt\) and \(\|m(s)\|\equiv R\). Thus the maximization reduces to arranging a unit-vector field against the spectrum and sign pattern of \(b\), which is precisely the viewpoint used in the concrete examples of Section 5 and the supplementary separated-kernel statements in Appendix 14.

4 Fixed Prompts as Auxiliary Variables↩︎

This section records prefix tokens as a second concrete realization of the same fixed-auxiliary mechanism. The auxiliary variable is now an externally chosen prompt \(z\in\mathcal{Z}\) with fixed law \(\eta\), while only the content variable is optimized. The main point is that the prompt law itself can be transported directly into a maximizing marginalized output law through an orthogonal gauge.

4.1 Main fixed-prompt realization theorem↩︎

Given a fixed prompt distribution \(\eta\in \mathcal{P}(\mathcal{Z})\), set \(X_{\mathcal{Z}}:=\mathbb{S}^{d-1}\times\mathcal{Z}\) and define the admissible class \(\mathcal{P}_{\eta}:=\{\mu\in \mathcal{P}(X_{\mathcal{Z}}): (\pi_{\mathcal{Z}})_{\#}\mu=\eta\}\). Thus any \(\mu\in\mathcal{P}_{\eta}\) admits a disintegration \(\mu( \mathrm{d}x, \mathrm{d}z)=\mu^z( \mathrm{d}x)\eta( \mathrm{d}z)\), and the prompt marginal is frozen exactly as the positional marginal was in Section 2. For a measurable map \(G:\mathcal{Z}\to\mathbb{S}^{d-1}\), write \(\Gamma_\eta[G]:=(z\mapsto(G(z),z))_{\#}\eta\). In this prompt setting we use the distinguished prompt energy notation on \(X_{\mathcal{Z}}\), namely \(\mathcal{E}_h^{\rm pr}[\mu]:=\frac{1}{2}\iint_{\mathcal{Z}\times \mathcal{Z}}\iint_{\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}}h\bigl((x,z),(y,w)\bigr)\mu^z( \mathrm{d}x)\mu^w( \mathrm{d}y)\eta( \mathrm{d}z)\eta( \mathrm{d}w)\).

Given a measurable map \(\Psi:\mathcal{Z}\to O(d)\) and \(\beta>0\), consider the kernel \[\label{eq:def-h-prompt} h_{\Psi}\bigl((x,z),(y,w)\bigr) := \frac{1}{\beta} \exp\Bigl( \beta \bigl\langle x,\Psi(z)\Psi(w)^\mathrm{T}y\bigr\rangle \Bigr).\tag{3}\] We also write \(\mathcal{E}_\Psi:=\mathcal{E}_{h_\Psi}^{\rm pr}\).

The next theorem is the main result of the section. It is the prompt analogue of the phase-field construction from Section 3, but also a genuine strengthening: the maximizing law is no longer confined to a single one-parameter orbit.

Theorem 4 can be viewed as replacing \(s\mapsto R_{-\omega s}\) in the RoPE case by the more general prompt-dependent orthogonal transformation \(z\mapsto \Psi(z)\). Therefore the fixed reference measure \(\eta( \mathrm{d}z)\) plays the same role as the reference measure \(\rho( \mathrm{d}\xi)\) in USA-AV, specialized to \(\mathrm{d}s\) in the positional sections. The difference is that positional encoding realizes laws on a one-dimensional orbit, whereas prefix tokens allow those on general spaces and hence an arbitrary \(\eta\)-pushforward law on the sphere.

At the variational level, prefix tokens again reduce to a fixed-marginal maximization problem. The prompt-side Diracization principle is recorded in Appendix 12.5, and the finite-prompt discrete analogue is recorded in Appendix 13.

5 Examples↩︎

We give several explicit positional kernels as examples and the maximizing marginalized laws they produce. Throughout this section, we keep the positional setting \(I=(0,1]\) with uniform reference law \(\mathrm{d}s\), and for concreteness we write the orbit formulas in \(d=3\). The first and third examples are consequences of the separated-kernel analysis recorded in Appendix 14, while the second is the canonical RoPE construction from Theorem 1.

These three examples summarize the positional message in concrete terms. Auxiliary variables do not automatically prevent collapse; what matters is the geometry of the coupling. A pointwise nonnegative distance-bias kernel still gives a Dirac maximizer, single-frequency RoPE gives a uniform latitude-circle law, and a cosine Toeplitz kernel gives a uniform great-circle law.

6 Experiments↩︎

6.1 Measuring collapse through many layers↩︎

We experimentally validate whether auxiliary variables prevent mode-collapse in the attractive dynamics. In particular, we compare the original USA model as a baseline with two gauge-structured auxiliary models, RoPE and prefix tokens.

We updated finite \(n\) particles \(\{x_i(\tau)\}_{i=1}^n\) using three different methods: (i) the standard USA model, (ii) the RoPE-type USA-AV model, and (iii) the USA-AV model with a fixed prompt, and measured the degree of distribution collapse using the metric \(\mathsf G_x(\tau) :=1-\frac{1}{n(n-1)}\sum_{i\neq j}\bigl\langle x_i(\tau),x_j(\tau)\bigr\rangle\). The closer \(\mathsf G_x\) is to zero, the closer the distribution is to collapse to a single point. The updates were performed up to time \(\tau_f=20\). Detailed settings are described in Section 9.2.

Figure 2 shows a clear separation between the baseline USA model without auxiliary variables and the USA-AV model. At \(\tau=20\), the baseline has numerically collapsed in the original frame for all three system sizes. In contrast, the RoPE and fixed-prompt runs remain noncollapsed and the uncertainty denotes one standard error over seeds.

6.2 Representation of distributions↩︎

This experiment examines the representation of the distribution of \(n=256\) particles generated by the USA-AV model with various kernels on \(\mathbb{S}^{d-1}\) with \(d=3\). In particular, we study six scenarios: (i) a baseline USA model without auxiliary variables, (ii) a nonnegative distance-bias kernel, (iii) a Toeplitz spectral kernel, (iv) the standard RoPE kernel, (v) a generalized RoPE kernel, and (vi) a USA-AV model with prefix tokens. Details are provided in Section 9.3.

Figure 3 shows the final distributions of the particles on a common sphere for all six cases. The baseline and distance-bias kernels collapse to a Dirac mass at the north pole, as expected. The Toeplitz spectral kernel produces a great-circle orbit, consistent with the dominance of the first Fourier mode. The standard RoPE kernel yields a uniform latitude circle. The generalized RoPE kernel generates a more complicated wavy closed curve, illustrating exact realization of nontrivial orbits. The fixed-prompt scenario produces a multi-cluster distribution corresponding to metastable clusters. These outcomes confirm the spectral dichotomy and show how different auxiliary geometries either enforce collapse or yield nontrivial marginal laws.

7 Conclusion and Future Directions↩︎

We developed an auxiliary variational framework for the mean-field transformer model. By optimizing joint laws for tokens and auxiliary variables, we describe its update through conditional Diracization as a structural anti-collapse mechanism. In particular, we showed that RoPE and prefix tokens prevent mode collapse and yield exact realization of maximizing laws. These results suggest that collapse in simplified mean-field transformer models partly reflects an auxiliary-free model class.

A limitation of this study is its limited treatment of dynamics. While we provide a brief dynamical metastability analysis in Section 10, we need a more concrete connection between the anti-mode collapse effect and the inference dynamics of transformers. Future work should connect this variational picture to training dynamics and characterize which noncollapsed output laws are useful for representation and downstream computation.

8 Related Work↩︎

8.1 Shared-weight and looped transformer architectures↩︎

A growing body of work studies transformers that reuse the same block across several computation steps. Looped Transformers improve length generalization on algorithmic tasks [2], their expressive power and the role of timestep encoding are analyzed in [3], recursive latent-space reasoning is studied in [4], and depth-recurrent transformers have recently been proposed for compositional generalization [5]. These papers differ in goals and training setups, but they share the architectural motif most relevant to the present work: repeated application of a shared-weight attention block. Our model should be read as a continuous-depth mean-field surrogate of that regime, not as a theorem about arbitrary untied-depth transformer stacks.

8.2 Dynamical and mean-field analyses of self-attention↩︎

Another line of work interprets repeated self-attention updates as interacting-particle or mean-field dynamics [12]–[14], [25]. Within this framework, clustering and dynamic metastability have been studied in [16]. Our paper belongs primarily to this dynamical stream. The point of departure is that the standard attractive position-free baseline is collapse-prone, and we ask how fixed auxiliary variables alter both the maximizing laws and the long-time picture.

8.3 Collapse, oversmoothing, and attention sinks↩︎

Recent work studies several degeneration phenomena in transformer models. Oversmoothing is not inevitable [17], masking and layer normalization can qualitatively change the long-depth behavior [18], and localization-type collapse has also been analyzed in simplified settings [9]. Attention sinks arise in standard softmax language models as disproportionate attention to initial or dedicated sink tokens [26]. That phenomenon is conceptually adjacent but mathematically different from the present paper: our collapse statements concern shared-weight attractive mean-field dynamics and Dirac concentration of marginalized content laws, rather than first-token sink behavior in untied-depth softmax language models.

8.4 Positional encodings, prefix tokens, and kernel viewpoints↩︎

Practical transformers use a wide variety of positional mechanisms. After the original absolute encodings of [1], later work developed relative position representations and segment-level recurrence [22], [27], relative-bias formulations such as T5 and DeBERTa [28], [29], linear-bias schemes such as ALiBi [23], and rotational encodings such as RoPE [24]. For long contexts these are often combined with sparse or local attention patterns, such as Longformer, BigBird, and Reformer [30]–[32]. On the prompting side, in-context conditioning and prefix/prompt tuning show that prepended tokens can steer model behavior [19], [20], [33], [34]. Our contribution is not a new practical PE or prompting recipe, but a variational theory explaining how fixed auxiliary marginals can prevent marginalized collapse. The energy viewpoint is also related to kernel theory: kernel mean embeddings give a language for functionals of probability measures [35], indefinite kernels can support richer geometries than positive-definite ones [36], and Toeplitz positional kernels naturally invite Fourier analysis [37], [38].

9 Details of experiments↩︎

9.1 Common experimental setup↩︎

We solve the finite-particle system ?? with fixed auxiliary labels \(\xi_i\). Unless otherwise specified, the numerical solver is a projected second-order Runge–Kutta method with step size \(\Delta\tau\in\{10^{-2},5\cdot10^{-3}\}\), followed by the normalization \(x_i\leftarrow x_i/\|x_i\|\) after each step. The terminal time is specified in each individual experiment, or the run is stopped earlier if the relative energy increment over a window of length \(1\) is below \(10^{-8}\). Unless otherwise specified in an individual experiment, each reported statistic is averaged over \(100\) random seeds, and the main runs are repeated for \(n\in\{64,128,256,512\}\) to check finite-particle stability.

9.2 Details of Section 6.1↩︎

9.2.0.1 Data-generating model.

We work on \(\mathbb{S}^{d-1}\) with \(d=3\) and fix the inverse-temperature parameter to \(\beta=1\). We use three system sizes, \(n\in\{64,128,256\}\). In each case there are \(m=4\) particles at each auxiliary value and hence \(L=n/4\) auxiliary values. For the baseline and RoPE runs, the auxiliary variable is position, \[s_\ell=\frac{\ell-1}{L},\qquad \ell=1,\dots,L.\] For the fixed-prompt run, the auxiliary variable is a prompt label \(z_\ell\) with prompt phase \(\theta_\ell=2\pi(\ell-1)/L\) and gauge \(\Psi(z_\ell)=R_{\theta_\ell}\), where \(R_\theta\) rotates the first two coordinates and leaves the third coordinate fixed. The initial gauge-frame states \(q_{\ell,a}(0)\) are sampled independently and uniformly from \(\mathbb{S}^{d-1}\) and then mapped back to the original frame. Thus the RoPE initialization is \[x_{\ell,a}(0)=R_{-\omega s_\ell}q_{\ell,a}(0),\qquad \omega=2\pi,\] and the fixed-prompt initialization is \(x_{\ell,a}(0)=\Psi(z_\ell)q_{\ell,a}(0)\). For each random seed, the same gauge-frame cloud \(\{q_{\ell,a}(0)\}\) is used in all three conditions.

9.2.0.2 Models compared.

The three kernels are \[\begin{align} h_0((x,\xi),(y,\zeta)) &=\frac{1}{\beta} e^{\beta\bigl\langle x,y\bigr\rangle},\tag{5}\\ h_R((x,s),(y,t)) &=\frac{1}{\beta}\exp\{\beta\bigl\langle x,R_{\omega(t-s)}y\bigr\rangle\},\tag{6}\\ h_\Psi((x,z),(y,w)) &=\frac{1}{\beta}\exp\{\beta\bigl\langle x,\Psi(z)\Psi(w)^\mathrm{T}y\bigr\rangle\}. \tag{7} \end{align}\] The first kernel ignores the auxiliary labels. The second is the RoPE kernel of Theorem 1. The third is the fixed-prompt gauge kernel of Theorem 4. In gauge variables, \(q_i=R_{\omega s_i}x_i\) for RoPE and \(q_i=\Psi(z_i)^\mathrm{T}x_i\) for prefix tokens, the latter two dynamics reduce to the same attractive dynamics as the baseline. This makes the comparison a direct test of whether collapse occurs in the gauge frame or in the original content frame.

9.2.0.3 Execution and evaluation.

We integrate the finite-particle ODE with step size \(\Delta\tau=10^{-2}\) up to \(\tau_f=20\) and record the trajectory every \(0.5\) time units. The experiment is repeated over \(100\) independent random seeds. We report the collapse gap \[\label{eq:exp-Gx-new} \mathsf G_x(\tau) :=1-\frac{1}{n(n-1)}\sum_{i\neq j}\bigl\langle x_i(\tau),x_j(\tau)\bigr\rangle.\tag{8}\] as the main observable, together with the gauge-frame gap \(\mathsf G_q(\tau)\), the conditional diameter \(\mathsf D_{\rm cond}(\tau)\), and the relative energy gap \(\Delta_E(\tau)\) as diagnostic checks. For the RoPE and fixed-prompt models, conditional collapse predicts \[\mathsf G_q(\tau_f)\approx0, \qquad \mathsf D_{\rm cond}(\tau_f)\approx0, \qquad \Delta_E(\tau_f)\approx0,\] while the original-frame gap \(\mathsf G_x(\tau_f)\) remains positive because different auxiliary values rotate the common gauge-frame Dirac center to different content locations.

9.3 Details of Section 6.2↩︎

9.3.0.1 Set-up and models.

All scenarios use the mean-field USA-AV dynamics described in Section 2 with the same projected solver and a common inverse temperature \(\beta=1\) for ease of comparison. We fix the particle count at \(n=256\) and sample initial tokens uniformly on \(\mathbb{S}^{d-1}\). Positions \(s\in\mathbb{T}\) are drawn uniformly unless stated otherwise. The individual models are as follows:

1. Baseline (no auxiliary variables). This is the purely attractive USA model from Section 2, which is known to collapse to a Dirac mass.

2. Nonnegative distance-bias. We use the separated kernel \[h_b((x,s),(y,t)) = b(s,t)\frac{1}{\beta}\exp\bigl(\beta\langle x,y\rangle\bigr),\qquad b(s,t)=\epsilon+\exp \bigl(-{\rm dist}_{\mathbb{T}}(s,t)^2/(2\ell^2)\bigr),\] with \(\epsilon=0.02\) and length scale \(\ell=0.1\). This kernel should collapse to a Dirac law despite the positional labels.

3. Toeplitz spectral. We take \(h_T((x,s),(y,t))=c(t-s) \langle x,y\rangle\) with \(c(\Delta)=\cos(2\pi m\Delta)\) and \(m=1\), so that the first Fourier mode dominates. The theory predicts an \(m\)-fold circle-type orbit on \(\mathbb{S}^{d-1}\).

4. Standard RoPE. The rotary positional encoding corresponds to the exponentiated kernel \(h_{\rm RoPE}((x,s),(y,t))=\beta^{-1}\exp\{\beta\langle R_{-\omega s}x, R_{-\omega t}y\rangle\}\) with \(\omega=2\pi\). In the maximizer, the tokens lie on a latitude circle determined by the effective gauge.

5. Generalized RoPE. To illustrate that RoPE can realize more general orbits, we allow the rotary angle to depend on position via a sinusoidal phase field \(\alpha(s)\) and use \(h_\alpha((x,s),(y,t)):=\beta^{-1}\exp\{\beta\langle x,R_{\alpha(t)-\alpha(s)}y\rangle\}\). This produces a wavy closed curve on the sphere.

6. Prefix tokens. We insert \(K_{\rm pr}=3\) fixed prompt tokens with prescribed positions and embeddings, choose gauges \(\Psi(z_i)\), and use the prompt kernel 7 . This induces three well-separated clusters and tests the metastability mechanism of Theorem 5.

10 Dynamic complement: Metastability under positional kernels↩︎

We consider the distance-bias kernel \(h\bigl((x,s),(y,t)\bigr)= b(s,t)\frac{1}{\beta} e^{\beta\langle x,y\rangle}\) with \(b\ge0\), and assume that the initial configuration is organized into \(K\ge2\) well-separated clusters \(C_1,\dots,C_K\). More precisely, cluster \(C_p\) starts inside a spherical cap of radius \(r_0\) around a unit vector \(\bar u_p\), and the initial inter-center angles are bounded below by \(2\sigma_0>0\). Let \(\Delta_p(\tau):=\max_{i,j\in C_p}\arccos\langle x_i(\tau),x_j(\tau)\rangle\) be the diameter of cluster \(p\), let \(u_p(\tau)\) be the normalized cluster average, and let \(\Theta_{pq}(\tau):=\arccos\langle u_p(\tau),u_q(\tau)\rangle\) be the angle between cluster centers. We denote by \(B_\parallel\) and \(B_\times\) the effective intra- and inter-cluster couplings, and by \(w_p=|C_p|/n\) and \(W_{pq}\) the corresponding coarse-grained weights; the exact definitions are given in Appendix 15.

This theorem makes precise the picture that the static maximizer alone misses. Even when the energy landscape eventually favors one-direction aggregation, the dynamics first contracts within each cluster and then remains for an exponentially long time near a reduced multi-cluster configuration. The detailed assumptions, constants, and proof are given in Appendix 15.

11 Technical material for Section 2↩︎

11.1 Statements moved from Section 2↩︎

For the stability estimate used in the mean-field limit, we assume the strengthened Lipschitz bounds on \(\nabla_x h\) with respect to its first and second arguments: \[\begin{align} \bigl\|\nabla_x h((x,\xi),(y,\zeta))-\nabla_x h((x',\xi'),(y,\zeta))\bigr\| &\le L_x d_X\bigl((x,\xi),(x',\xi')\bigr), \tag{9}\\ \bigl\|\nabla_x h((x,\xi),(y,\zeta))-\nabla_x h((x,\xi),(y',\zeta'))\bigr\| &\le L_\mu d_X\bigl((y,\zeta),(y',\zeta')\bigr). \tag{10} \end{align}\]

11.2 Finite-particle and empirical-law statements↩︎

This simple invariance statement plays two roles in the rest of the paper: it justifies the sphere-constrained dynamics as a genuine ODE on a compact manifold, and it lets us estimate all later quantities without tracking radial errors.

The only auxiliary estimate needed in the proof above is the Lipschitz dependence of the tangent projection on the base point.

11.3 Auxiliary well-posedness material↩︎

This subsection collects the precise weak-solution formulation behind Definition 3. The strategy is standard but worth making explicit: we first define weak solutions, then study frozen linearized flows, and finally close the nonlinear problem by a contraction argument on curves of measures. This discussion is summarized in Proposition 5.

Let the state space be \[M:=X=\mathbb{S}^{d-1}\times\mathcal{A},\] where \(\mathbb{S}^{d-1}\subset\mathbb{R}^d\) (\(d\ge 2\)) is the sphere and \((\mathcal{A},d_\mathcal{A})\) is the compact metric auxiliary space introduced in Section 2. Then \(M\) is compact. We equip it with the distance \[d_X\bigl((x,\xi),(y,\zeta)\bigr):=\|x-y\|+d_\mathcal{A}(\xi,\zeta).\]

Assume that a kernel \(h:M\times M\to\mathbb{R}\) is given and is differentiable with respect to the first argument \(x\). We write its derivative as \[\nabla_x h\bigl((x,\xi),(y,\zeta)\bigr)\in\mathbb{R}^d.\] Using the orthogonal projection \(P_x^\perp\) onto the tangent space of the sphere and the force \[F(x,\xi;\mu):=\int_M \nabla_x h\bigl((x,\xi),(y,\zeta)\bigr) \mu( \mathrm{d}y, \mathrm{d}\zeta),\] we define the velocity field \[\label{eq:Vmu} V[\mu](x,\xi):=P_x^\perp F(x,\xi;\mu)\in T_x\mathbb{S}^{d-1}\subset\mathbb{R}^d.\tag{11}\] Since the auxiliary component is frozen, the characteristic flow acts only on the sphere coordinate.

11.3.1 Mean-field (measure) equation and weak solutions↩︎

For a curve of probability measures \((\mu_\tau)_{\tau\ge 0}\subset\mathcal{P}(M)\), we write the USA-AV mean-field equation (continuity equation) as \[\label{eq:CE} \partial_\tau\mu_\tau + \mathop{\mathrm{div}}_{x} \bigl(\mu_\tau V[\mu_\tau]\bigr)=0 \quad\text{on }M, \qquad \mu_{\tau=0}=\mu_0\in\mathcal{P}(M).\tag{12}\] Here \(\mathop{\mathrm{div}}_x\) denotes the surface divergence with respect to the spherical variable \(x\). Since \(V[\mu_\tau](\cdot,\xi)\in T_x\mathbb{S}^{d-1}\), this equation is intrinsically well defined.

11.3.2 Assumptions (strong regularity: Lipschitz type)↩︎

11.3.3 Preparatory lemmas: Lipschitz continuity of tangent projections and velocity fields↩︎

11.3.4 Flow maps for frozen curves (linearized problem)↩︎

11.3.5 Pushforward representation of solutions to the linear continuity equation↩︎

11.3.6 Construction of solutions to the nonlinear problem: the contraction mapping (flow-map method)↩︎

11.4 Proof of Proposition 4↩︎

12 Proofs for the variational reductions in Sections 2–4↩︎

12.1 Statements moved from Section 2↩︎

12.2 Energy identities and disintegration formulas↩︎

12.3 Proof of Theorem 3↩︎

12.4 Euler-Lagrange condition for maximizers↩︎

12.5 Supplementary fixed-prompt statements↩︎

13 Proofs for the explicit constructions in Sections 3 and 4↩︎

13.1 Auxiliary rigidity lemma↩︎

This subsection collects a short rigidity fact that is used in several proofs but would interrupt the main narrative if stated earlier. It is elementary, but it clarifies exactly where the rigidity used later comes from.

13.2 RoPE and phase-field constructions↩︎

13.3 Proof of Theorem 2↩︎

13.4 Fixed-prompt gauge constructions↩︎

14 Supplementary separated-kernel statements and proofs↩︎

14.1 Deferred sign-changing Toeplitz example↩︎

14.2 Proofs↩︎

15 Supplementary metastability setup and proof↩︎

15.1 Deferred setup↩︎

The content space is the sphere \(\mathbb{S}^{d-1}\subset\mathbb{R}^d\) with \(d\ge2\), and the position variable lies in \(I=[0,1)\), identified with \(\mathbb{T}\) for periodic kernels. We consider \(n\) particles \((x_i,s_i)\) for \(i=1,\dots,n\), where \(x_i\in\mathbb{S}^{d-1}\) is time dependent and \(s_i\in[0,1)\) is fixed. The positional kernel \(b:[0,1)\times[0,1)\to[0,\infty)\) is assumed symmetric, bounded, and measurable, and the content kernel is the exponentiated inner product \[k_\beta(x,y)=\frac{1}{\beta} e^{\beta\langle x,y\rangle}\qquad(\beta\ge1).\] The full kernel is separated: \[\label{eq:h} h\big((x,s),(y,t)\big):=b(s,t) k_\beta(x,y).\tag{15}\] Then the finite-particle energy and projected gradient-ascent flow are \[\begin{align} \mathcal{E}(\boldsymbol{x}) &:= \frac{1}{2n^2}\sum_{i,j=1}^{n} h\big((x_i,s_i),(x_j,s_j)\big) = \frac{1}{2\beta n^2}\sum_{i,j=1}^{n} b_{ij} e^{\beta\langle x_i,x_j\rangle},\tag{16}\\ \dot{x}_i &= P_{x_i}^\perp \Bigl[\frac{1}{n}\sum_{j=1}^{n} b_{ij} e^{\beta\langle x_i,x_j\rangle} x_j\Bigr],\qquad b_{ij}:=b(s_i,s_j),\tag{17} \end{align}\] and standard calculations yield \[\label{eq:Energy-production} \frac{ \mathrm{d}}{ \mathrm{d}\tau} \mathcal{E}(\boldsymbol{x}(\tau)) = \frac{1}{n}\sum_{i=1}^{n}\Bigl\|P_{x_i}^\perp \frac{1}{n}\sum_{j=1}^{n} b_{ij} e^{\beta\langle x_i,x_j\rangle} x_j\Bigr\|^2 \ge 0 .\tag{18}\]

15.1.0.1 Clustered initialization and effective couplings

Fix an integer \(K\in\{2,\dots,n\}\), and partition the index set into \(K\) clusters, \(\{1,\dots,n\}=C_1\sqcup\cdots\sqcup C_K\). For each cluster \(p\), choose an initial center \(\bar u_p\in\mathbb{S}^{d-1}\) and a radius \(r_0\in(0,\tfrac14)\) such that \[\label{eq:init} \max_{i\in C_p}\arccos\langle x_i(0),\bar u_p\rangle\le r_0, \qquad \min_{p\ne q}\arccos\langle \bar u_p,\bar u_q\rangle \ge 2\sigma_0\tag{19}\] with \(\sigma_0\in(0,\tfrac\pi4)\) half of the initial separation angle. Let \(n_p:=|C_p|\) and \(w_p:=n_p/n\). Define the coarse-grained positional matrix \[\label{eq:Wpq} W_{pq}:=\frac{1}{n_p n_q}\sum_{i\in C_p}\sum_{j\in C_q} b_{ij} \qquad(p,q=1,\dots,K),\tag{20}\] and assume \(W_{pq}\ge0\) and \(W_{pp}>0\). We also define \[\label{eq:intra-cross} B_\parallel:=\min_{p}\min_{i\in C_p}\sum_{j\in C_p}\frac{b_{ij}}{n} , \qquad B_\times:=\max_{p\ne q}\max_{i\in C_p}\sum_{j\in C_q}\frac{b_{ij}}{n} .\tag{21}\]

The following observables separate the two geometric effects that matter dynamically: fast contraction inside each cluster and slow drift between different clusters.

In the sequel we assume \(b\ge0\), i.e. the matrix \(\mathbf{B}=(b_{ij})\) is entrywise nonnegative. This corresponds to the natural situation where the self-attention weights are not suppressive (negative).

15.1.0.2 Detailed form of Theorem 5.

Under 15 21 and the initial condition 19 , the following hold.

1. Rapid aggregation (fast time scale). There exist constants \(c_1,c_2>0\) such that \[\Delta_p(\tau) \le c_1 r_0 e^{-c_2 \beta B_\parallel \tau} + C_1 e^{-\beta(1-\cos\sigma_0)}\qquad(p=1,\dots,K),\] so each cluster contracts exponentially and becomes essentially point-like by time \(T_f:=O \big((\beta B_\parallel)^{-1}\log r_0^{-1}\big)\).

2. Propagation of smallness and trapping (metastable well). Choose \(r\in(0,\tfrac{\sigma_0}{4})\) sufficiently small, and assume \[\label{eq:beta-gap} \beta \ge \beta_\star :=\frac{1}{1-\cos\sigma_0} \log\Bigl(\frac{8 B_\times}{B_\parallel r}\Bigr).\tag{22}\] Define \[T_m:=\inf\{\tau\ge T_f:\max_p\Delta_p(\tau)>2r\;\text{ or }\;\min_{p\ne q}\Theta_{pq}(\tau)<\sigma_0\}.\] Then for all \(\tau\in[T_f, T_m]\) one has \[\Delta_p(\tau)\le 2r,\qquad \Theta_{pq}(\tau)\ge \sigma_0\qquad(\forall p\ne q).\] Moreover, \[T_m \ge c_3 \frac{B_\parallel}{B_\times} e^{ \beta(1-\cos\sigma_0)} .\]

3. Asymptotic reduction on the slow manifold (coarse-grained \(K\)-point system). For \(\tau\in[T_f,T_m]\), the center directions satisfy \[\label{eq:reduced-ODE} \dot{u}_p = P_{u_p}^\perp\Bigl[\sum_{q=1}^{K} w_q W_{pq} e^{\beta\langle u_p,u_q\rangle} u_q\Bigr] + R_p(\tau),\qquad \|R_p(\tau)\| \le C_2\bigl(r+e^{-\beta(1-\cos\sigma_0)}\bigr).\tag{23}\] Hence \(u=(u_1,\dots,u_K)\) follows the projected gradient-ascent flow of a \(K\)-point system up to an error of order \(O(r+e^{-\beta(1-\cos\sigma_0)})\).

4. Stepwise collapse (time scale to the first merger). For the effective energy of the reduced system, \[\mathcal{E}^{(K)}(u) := \frac{1}{2\beta}\sum_{p,q=1}^{K} w_p w_q W_{pq} e^{\beta\langle u_p,u_q\rangle},\] let \(\delta_{\rm gap}:=\min_{p\ne q}\bigl(1-\cos\Theta_{pq}(T_f)\bigr)\). Then there exist constants \(c_4,c_5>0\) such that \[c_4 \frac{1}{\Lambda} e^{ \beta \delta_{\rm gap}} \le \tau_{\rm merge}-T_f \le c_5 \frac{1}{\lambda_{\rm cross}} e^{ \beta \delta_{\rm gap}},\] where \(\tau_{\rm merge}\) is the first merger time, \(\Lambda:=\max_{p}\sum_q w_q W_{pq}\), and \(\lambda_{\rm cross}:=\min_{p\ne q}\bigl(w_qW_{pq}+w_pW_{qp}\bigr)\). Thus the time until the first merger is of order \(e^{\beta\delta_{\rm gap}}\), and is bounded from below proportionally to \(T_m\).

15.2 Proof of Theorem 5↩︎

15.3 Supplement to the metastability analysis↩︎

This subsection fills in the differential inequalities used in Section 6. The emphasis is on transparency of the estimates rather than on introducing new conceptual ingredients. In this appendix, under the USA-AV ODE \[\dot{x}_i = P_{x_i}^{\perp} \Bigl[\frac{1}{n}\sum_{j=1}^{n} b_{ij} e^{\beta\langle x_i,x_j\rangle} x_j\Bigr], \qquad P_{x}^{\perp}=I-xx^\top,\quad x_i\in \mathbb{S}^{d-1},\] we derive the exact time derivative of the two-point inner product \(\langle x_i,x_j\rangle\) and fully justify the inequality used in Lemma 9. Below we assume that \(b_{ij}=b(s_i,s_j)\ge0\) is symmetric and write \(\theta_{ij}:=\arccos\langle x_i,x_j\rangle\in[0,\pi]\) and \(f_{ij}:=1-\langle x_i,x_j\rangle=1-\cos\theta_{ij}\).

15.3.1 Exact expansion of the time derivative of inner products↩︎

15.3.2 Signs of the projection terms and basic estimates↩︎

We use the following two estimates.

15.3.3 Derivation of the inequality in Lemma 9↩︎

Applying Lemma 12 to the self-terms \(\ell=j\) and \(\ell=i\) in ?? , and Lemma 13 to all remaining terms, gives \[\begin{align} \frac{ \mathrm{d}}{ \mathrm{d}\tau} f_{ij} &\le -\frac{2}{n} b_{ij} e^{\beta\langle x_i,x_j\rangle} f_{ij} +\frac{1}{n}\sum_{\ell\neq j} b_{i\ell}e^{\beta\langle x_i,x_\ell\rangle} (1-\langle x_j,x_\ell\rangle) +\frac{1}{n}\sum_{\ell\neq i} b_{j\ell}e^{\beta\langle x_j,x_\ell\rangle} (1-\langle x_i,x_\ell\rangle). \label{eq:fprime-ineq-raw} \end{align}\tag{24}\] Now decompose the sums according to a cluster partition \(\{1,\dots,n\}=C_p\sqcup\cdots\). Assuming \(i,j\in C_p\), one bounds the terms inside \(C_p\) using \(1-\langle x_j,x_\ell\rangle\le 1-\cos\Delta_p(\tau)\) and \(e^{\beta\langle x_i,x_\ell\rangle}\le e^{\beta}\), while for \(\ell\notin C_p\) one uses \(1-\langle x_j,x_\ell\rangle\le 2\) and \(\langle x_i,x_\ell\rangle\le \cos\sigma_0\), so that \(e^{\beta\langle x_i,x_\ell\rangle}\le e^{\beta\cos\sigma_0}\). Combining these bounds with \(e^{\beta\langle x_i,x_j\rangle}\ge e^{\beta\cos\Delta_p(\tau)}\ge e^{\beta(1-\Delta_p(\tau)^2/2)}\) yields \[\label{eq:fprime-final} {\quad \frac{ \mathrm{d}}{ \mathrm{d}\tau} f_{ij} \le - 2 B_\parallel e^{\beta(1-\Delta_p^2/2)} f_{ij} + C e^{\beta} \Delta_p^2 + 4 B_\times e^{\beta\cos\sigma_0}. \quad}\tag{25}\] This is the rigorous differential inequality used in the main text.

Remark. The concise display in the main text is equivalent to the cluster-wise upper bound of 24 , once the estimates involving \(\Delta_p\) and \(\sigma_0\) are inserted.

15.4 Full expansion of the angular-velocity estimate in Step 4↩︎

The purpose of this section is to make the slow-merger estimate completely explicit. Starting from the reduced \(K\)-point dynamics, we isolate the geometric bounds that ultimately determine the exponential merger time scale. In this appendix we derive the upper and lower bounds for the time derivative of the inter-center angle \(\Theta_{pq}\) used in Step 4 of the main text, starting from the exact coarse-grained \(K\)-point system.

15.4.1 Exact formula for the coarse-grained \(K\)-point system↩︎

By part (iii) of the main text, the centers \(u_p\in \mathbb{S}^{d-1}\) satisfy \[\label{eq:reduced} \dot{u}_p = P_{u_p}^{\perp}\Bigl[\sum_{r=1}^{K} w_r W_{pr} e^{\beta\langle u_p,u_r\rangle}u_r\Bigr] + R_p, \qquad\|R_p\|\le \varepsilon_{\rm red},\tag{26}\] where \(\varepsilon_{\rm red}:=C(r+e^{-\beta(1-\cos\sigma_0)})\) is the error bound. For the pairwise angle \(\Theta_{pq}:=\arccos\langle u_p,u_q\rangle\), \[\frac{ \mathrm{d}}{ \mathrm{d}\tau} \cos\Theta_{pq} = \langle \dot{u}_p,u_q\rangle + \langle u_p,\dot{u}_q\rangle.\] Substituting 26 and using self-adjointness of \(P_{u_p}^{\perp}\) gives \[\label{eq:cosdot-exact} \frac{ \mathrm{d}}{ \mathrm{d}\tau} \cos\Theta_{pq} = \sum_{r=1}^{K} w_r W_{pr} e^{\beta\langle u_p,u_r\rangle} \big\langle u_r, P_{u_p}^{\perp}u_q\big\rangle +\sum_{r=1}^{K} w_r W_{qr} e^{\beta\langle u_q,u_r\rangle} \big\langle u_r, P_{u_q}^{\perp}u_p\big\rangle + \langle R_p,u_q\rangle+\langle u_p,R_q\rangle.\tag{27}\] Hence \[\label{eq:onediff} \frac{ \mathrm{d}}{ \mathrm{d}\tau}\bigl(1-\cos\Theta_{pq}\bigr) = -\sum_{r} w_r W_{pr}e^{\beta\langle u_p,u_r\rangle} \big\langle u_r,P_{u_p}^{\perp}u_q\big\rangle -\sum_{r} w_r W_{qr}e^{\beta\langle u_q,u_r\rangle} \big\langle u_r,P_{u_q}^{\perp}u_p\big\rangle + O(\varepsilon_{\rm red}).\tag{28}\]

15.4.2 Application of spherical trigonometry (law of cosines)↩︎

For three points \(u_p,u_q,u_r\in \mathbb{S}^{d-1}\), the spherical law of cosines gives \[\langle u_r,u_q\rangle = \langle u_r,u_p\rangle \langle u_p,u_q\rangle + \sin\Theta_{rp} \sin\Theta_{pq} \cos\varphi_{r|p,q},\] where \(\varphi_{r|p,q}\) is the angle at the vertex \(p\) of the geodesic triangle \(pqr\). Therefore \[\label{eq:proj-id} {\quad \big\langle u_r, P_{u_p}^{\perp}u_q\big\rangle = \langle u_r,u_q\rangle - \langle u_r,u_p\rangle \langle u_p,u_q\rangle = \sin\Theta_{rp} \sin\Theta_{pq} \cos\varphi_{r|p,q}. \quad}\tag{29}\] Similarly, \(\langle u_r, P_{u_q}^{\perp}u_p\rangle =\sin\Theta_{rq} \sin\Theta_{pq} \cos\varphi_{r|q,p}\). Hence \[\label{eq:proj-bounds} - \sin\Theta_{rp} \sin\Theta_{pq} \le \big\langle u_r, P_{u_p}^{\perp}u_q\big\rangle \le \sin\Theta_{rp} \sin\Theta_{pq},\tag{30}\] since \(\cos\varphi\in[-1,1]\).

15.4.3 Upper and lower bounds for the angular velocity↩︎

15.4.3.1 Upper bound.

From 28 and 30 , \[\begin{align} \frac{ \mathrm{d}}{ \mathrm{d}\tau}\bigl(1-\cos\Theta_{pq}\bigr) &\le - w_q W_{pq} e^{\beta\cos\Theta_{pq}} (1-\cos^2\Theta_{pq}) - w_p W_{qp} e^{\beta\cos\Theta_{pq}} (1-\cos^2\Theta_{pq})\notag\\ &\qquad + \sin\Theta_{pq}\sum_{r\notin\{p,q\}} w_r\Bigl(W_{pr} e^{\beta\cos\Theta_{pr}} \sin\Theta_{rp} +W_{qr} e^{\beta\cos\Theta_{rq}} \sin\Theta_{rq}\Bigr) + O(\varepsilon_{\rm red}). \label{eq:upper} \end{align}\tag{31}\] On time intervals where the initial separation \(\Theta_{pr},\Theta_{qr}\ge \sigma_0\) is maintained, one has \(\sin\Theta_{rp},\sin\Theta_{rq}\le 1\), \(e^{\beta\cos\Theta_{pr}},e^{\beta\cos\Theta_{rq}}\le e^{\beta\cos\sigma_0}\), and \(\sum_r w_r W_{pr}\le \Lambda\). Thus \[\label{eq:upper2} \frac{ \mathrm{d}}{ \mathrm{d}\tau}\bigl(1-\cos\Theta_{pq}\bigr) \le -\bigl(w_q W_{pq}+w_p W_{qp}\bigr) e^{\beta\cos\Theta_{pq}} (1-\cos^2\Theta_{pq}) + C \Lambda e^{\beta\cos\sigma_0} \sin\Theta_{pq} + O(\varepsilon_{\rm red}).\tag{32}\]

15.4.3.2 Lower bound.

Using the lower side of 30 in the same way gives \[\label{eq:lower} \frac{ \mathrm{d}}{ \mathrm{d}\tau}\bigl(1-\cos\Theta_{pq}\bigr) \ge -\bigl(w_q W_{pq}+w_p W_{qp}\bigr) e^{\beta\cos\Theta_{pq}} (1-\cos^2\Theta_{pq}) - C \Lambda e^{\beta\cos\sigma_0} \sin\Theta_{pq} + O(\varepsilon_{\rm red}).\tag{33}\]

15.4.4 Two-sided estimates for the merging time by time integration↩︎

For convenience, write \(y(\tau):=\sin\Theta_{pq}(\tau)\in(0,1)\). Rewriting 32 and 33 using \(1-\cos\Theta=\tfrac12 y^2 + O(y^4)\) and \(1-\cos^2\Theta=y^2(1-\tfrac14 y^2)\) yields differential inequalities of the form \[\dot{y} \le -\alpha e^{\beta} y + \gamma e^{\beta\cos\sigma_0} + O(\varepsilon_{\rm red}),\qquad \dot{y} \ge -\bar\alpha e^{\beta} y - \bar\gamma e^{\beta\cos\sigma_0} + O(\varepsilon_{\rm red}),\] where \(\alpha\simeq w_q W_{pq}+w_p W_{qp}\) and \(\gamma\simeq C\Lambda\). By comparison for linear differential inequalities, the time needed to reach the threshold \(y_c\approx c e^{-\beta(1-\cos\sigma_0)}\) from an initial angle \(y(T_f)=y_0\) satisfies \[\frac{1}{\bar\alpha} e^{-\beta} \log\frac{y_0}{y_c} \lesssim \tau_{\rm merge}-T_f \lesssim \frac{1}{\alpha} e^{-\beta} \log\frac{y_0}{y_c} + \frac{\gamma}{\alpha} e^{-\beta(1-\cos\sigma_0)}.\] Substituting \(y_c \asymp e^{-\beta(1-\cos\sigma_0)}\) gives the estimate stated in the main text, \[c_4 \frac{1}{\Lambda} e^{ \beta(1-\cos\sigma_0)} \le \tau_{\rm merge}-T_f \le c_5 \frac{1}{\lambda_{\rm cross}} e^{ \beta(1-\cos\sigma_0)},\] using the rough comparison \(\lambda_{\rm cross}\lesssim w_q W_{pq}+w_p W_{qp}\lesssim \Lambda\).

Summary. In this appendix we derived (i) the exact expansion ?? for the time derivative of pairwise inner products together with projection estimates (Lemmas 12 and 13), leading to the cluster-based inequality 25 ; (ii) upper and lower bounds 32 , 33 for the derivative of the pairwise angle in the coarse-grained \(K\)-point system using the spherical law of cosines; and (iii) the exponential-scale estimate of the merging time by integrating these bounds in time. This fills in the details omitted in Step 1 and Step 4 of the main text.

16 Notes on Section 6↩︎

Section 6 contains numerical experiments only and has no deferred proofs.

17 Notes on Section 7↩︎

Section 7 contains no deferred proofs beyond the appendix material already organized above.

References↩︎