1 Introduction↩︎

In deep nonlinear networks with all or some layers initialized at small scale, training exhibits saddle-structured dynamics characterized by long plateaus and sharp transitions corresponding to successive feature activation [1], [2]. Recent work has formalized this picture in several ways, including saddle-to-saddle descriptions of training dynamics [3], staircase-style complexity acquisition [4], neural-race reductions for gated networks [5], alternating feature-learning dynamics in two-layer networks [6], and frequency-ordered learning phenomena [7]. These perspectives are largely descriptive or confined to shallow settings, and do not give a sharp dynamical mechanism for the transitions between stages in deep networks with smooth nonlinear activations. General saddle-escape results [8] bound worst-case escape times under generic perturbations but do not use the symmetries that govern stagewise learning in overparameterized deep networks.

Exact theories of learning dynamics fall into three lines of work: the deep-linear line, giving closed-form mode-wise solutions with depth-dependent plateaus [9]–[11]; the saddle-to-saddle line, ordering low-rank saddles in deep linear and diagonal-linear networks via balancing and small initialization, with ReLU extensions [12]–[15]; and the teacher-student line, reducing high-dimensional dynamics to low-dimensional order parameters under SGD and in mean-field or lazy limits [16]–[25]. Much of this work relies on linear or \(1\)-homogeneous structure, shallow architectures, or infinite-width limits.

1.0.0.1 This work.

We identify a single scalar quantity — the number \(r\) of layers at the small initialization scale, not the total depth \(L\) — that sets the plateau escape time \(\tau_\star = \Theta(\varepsilon^{-(r-2)})\) in deep networks with smooth activations for a single-mode teacher at initialization scale \(\varepsilon\). We derive an exact identity for the drift of the layer imbalance, valid for any loss function and activation. A single functional \(\varphi_\sigma(z) = z \sigma'(z) - \sigma(z)\) (the pointwise failure of Euler’s identity for \(\sigma\)) groups activations by the order \(q\) of the first nonlinear term in their Taylor expansion (recovering the linear case of [9], [26]).

Two theoretical arguments predict the same \(r{-}2\) exponent. On the permutation-symmetric submanifold, the normalized-metric identity of 7.2 combines with an approximate balance law to reduce the matrix flow to a one-dimensional integral. Off the manifold at He-normal initialization [27], we work directly with the coordinate-free signal energy \(\gamma(W) = \mathbb{E}[f\,g]\); an AM-GM bound on the layerwise gradient tensors gives the same exponent. The agreement is evidence that \(r\) is a property of the flow, not the reduction.

1.0.0.2 Contributions.

• We derive a nonlinear analog of the layer-imbalance conservation law of deep-linear networks, valid for any smooth activation and any differentiable loss, and show that a single functional of the activation classifies deep networks into four dynamical regimes.

• We reduce gradient flow on the permutation-symmetric submanifold to a one-dimensional integral, and derive a critical-depth escape-time law \(\tau_\star = \Theta(\varepsilon^{-(r-2)})\) set by the number of bottleneck layers rather than the total depth.

• We lift the \(r{-}2\) exponent off the manifold at He-normal initialization with \(r\) bottleneck layers rescaled by \(\varepsilon\) via a signal-energy argument on the scalar \(\gamma(W) = \mathbb{E}[f g]\), which is independent of any ansatz; this gives the scaling in a regime where the manifold is not attracting.

1.0.0.3 Setup.

We study \(L\)-layer fully connected feedforward networks \(f(x) = W_L \sigma(W_{L-1} \sigma(\cdots \sigma(W_1 x)))\) with a smooth activation \(\sigma: \mathbb{R}\to \mathbb{R}\), weight matrices \(W_l \in \mathbb{R}^{n_l \times n_{l-1}}\) of width \(n_0 = d\), \(n_1 = \cdots = n_{L-1} = N\), and scalar output \(n_L = 1\). We write \(z_l \doteq W_l h_{l-1}\) for the pre-activations and \(h_l \doteq \sigma(z_l)\) for the post-activations with \(h_0 = x\). Inputs are Gaussian \(x \sim \mathcal{N}(0, I_d)\) and the loss is \(\mathcal{L}(W) = \tfrac{1}{2} \mathbb{E}_{x,y}[(f(x) - y)^2]\). [thm:identity,thm:offmanifold] use uniform Euclidean flow \(\dot{W}_l=-\nabla_{W_l}\mathcal{L}\); the scalar-manifold results and corresponding experiments use the normalized flow \(\dot{W}_1=-\nabla_{W_1}\mathcal{L}\), \(\dot{W}_l=-N^{-1}\nabla_{W_l}\mathcal{L}\) (\(l\ge2\)). The target is a single-hidden-layer teacher with \(r^*\) orthonormal modes \(y = \sum_{k=1}^{r^*} \beta_k \sigma(v_k^\top x)\), \(\beta_1 > \beta_2 > \cdots > \beta_{r^*} > 0\), producing the characteristic sequence of \(r^*\) saddles separating the stagewise modes. [sec:scalar,sec:escape] specialize to the single-mode case \(r^* = 1\); the multi-mode teacher returns in 5.

2 Imbalance Identity and Activation Classes↩︎

Before proceeding to the scalar reduction of the gradient flow ODE, we present an exact identity that holds for the full network, for any smooth activation, and for any differentiable loss. It controls the evolution of the layer imbalance \(\Delta_l \doteq \|W_{l+1}\|_F^2 - \|W_l\|_F^2\) and identifies a functional of the activation that dictates whether layer norms are conserved, drift at cubic order, or drift at quadratic order.

A full proof of 1 is deferred to 7.1. The identity also lifts to a matrix-valued statement for \(W_{l+1}^\top W_{l+1} - W_l W_l^\top\) (7.3), recovering the deep linear conservation \(W_{l+1}^\top W_{l+1} - W_l W_l^\top = \mathrm{const}\) [10], [28], [29] as the special case \(\varphi_\sigma\equiv 0\); a Noether-style view of how such symmetry-induced conservation is broken by discretization and weight decay appears in [30], [31].

Using the identity, we may now classify smooth activations \(\sigma\) by two indices: whether \(\sigma(0) = 0\), and the order \(q \ge 2\) of the first nonlinear term in the Taylor expansion \(\sigma(z) = a_0 + \alpha z + a_q z^q + O(z^{q+1})\) with \(a_q \ne 0\). The parameter \(q\) controls both the leading drift order (1) and the size of the first universality correction (1).

Representative activations include (A) linear (and a.e.ReLU); (B\(_3\)) \(\tanh\), \(\mathrm{erf}\), \(\sin\); (C) GELU, Swish; (D) sigmoid, softplus. We state all downstream results in terms of \(q\) for generality, but most activations of interest are \(q=3\). Proofs follow by Taylor expansion (7.6). We note that 1 couples \(\varphi_\sigma(z_l)\) to a backprop factor, so global control of \(\varphi_\sigma\) alone does not bound the drift; in the small-initialization regime this factor stays \(\Theta(1)\), making \(q\) the main structural prediction.

For the remainder of this work we specialize to Class B activations under a symmetric balanced ansatz where the restriction of 2 to the ansatz manifold is sharper: the drift scales as \(\varepsilon^{L+2}\) in the layer initialization scale \(\varepsilon\), higher order than the layer scales themselves.

3 Scalar Reduction on the Symmetric Manifold↩︎

1 is sharpest when evaluated on a flow-invariant manifold: the same type of reduction performed in deep-linear theory via \(W_{l+1}^\top W_{l+1} - W_l W_l^\top \equiv 0\). Our Class B analog is the following.

The ansatz is the \(S_N\)-fixed-point manifold of hidden-neuron permutation symmetry [32], [33]: for a single-mode teacher with Gaussian inputs and population squared loss, the data, loss, and teacher are all invariant under permutation of the \(N\) neurons in any hidden layer, so the gradient flow preserves the submanifold on which every neuron in a given layer shares one row. Flow-invariance (8.1) and the scalar readout structure it forces make this the nonlinear counterpart of exact balance in deep-linear theory. The normalization absorbs the layer-\(1\) width factor (\(\|W_1\|_F^2 = N X_1^2\), \(\|W_l\|_F^2 = X_l^2\) for \(l \ge 2\)), so the \(X_\ell\) live on a common \(\Theta(1)\) scale. On the manifold, the normalized scalar imbalance \(\widetilde{\Delta}_\ell \doteq \|W_\ell\|_F^2/c_\ell - \|W_1\|_F^2/c_1\) with \(c_1 = N\), \(c_\ell = 1\) (\(\ell \ge 2\)) reduces exactly to \(X_\ell^2 - X_1^2\); this is the ansatz analog of the matrix imbalance of 2 .

Throughout this section we fix a single-mode teacher \(y = \beta_1 \sigma(v_1^\top x)\) with \(\hat{w} = v_1\) and an activation \(\sigma(u) = \alpha u + a_q u^q + O(u^{q+1})\), with \(q \ge 2\) the order of the first nonlinear correction (\(q = 3\) for Class B, \(q = 2\) for Class C). The escape statements assume \(K^{(\sigma)} = \beta_1 h_\sigma\alpha^{L-1}/\sqrt N>0\). Proofs of \(S_N\)-fixed-point and flow-invariance, and the descent to a scalar flow, are given in 8.

3.1 The Scalar Chain and the Exact Reduced ODE↩︎

Under the ansatz, induction on \(l\) shows that the forward pass collapses to a scalar composition (8): \[\label{eq:chain} f(x) = \sqrt{N} X_L \sigma \bigl(X_{L-1} \sigma(\cdots \sigma(X_1 g))\bigr), \qquad g \doteq \hat{w}^\top x.\tag{3}\]

Because the ansatz is flow-invariant (8.1), the normalized population flow on \(W\) descends to the \(L\) scalars \((X_1, \ldots, X_L)\). On the ansatz, \(dX_\ell/ds=-N^{-1}\partial_{X_\ell}\mathcal{L}\) for every \(\ell\); the reduced time is \(t=s/N\). Thus the Euclidean scalar flow below is not induced by uniform-rate matrix flow.

Writing \(c_0 \doteq g\), \(c_1 \doteq X_1 g\), and \(c_\ell \doteq X_\ell \sigma(c_{\ell-1})\) for the scalar pre-activations of the chain, the chain rule along the composition gives the explicit closed form \[\label{eq:dxf-explicit} \partial_{X_\ell} f = \sqrt{N} \Bigl(\prod_{m > \ell} \sigma'(c_{m-1}) X_m\Bigr) \sigma(c_{\ell-1}), \qquad 1 \le \ell \le L,\tag{5}\] with the empty-product convention \(\partial_{X_L} f = \sqrt{N} \sigma(c_{L-1})\), so 4 is fully explicit in the scalar parameters \(X\) alone. The full derivation is given in 8.

3.2 Leading-Order Origin Expansion↩︎

The scalar chain 3 is a composition of multiplications by \(X_\ell\) and activations \(\alpha u + O(u^q)\). Expanding around the origin, every monomial in \(f(X, g)\) arises from choosing, at each of the \(L - 1\) nonlinear activations, either the linear branch \(\alpha u\) or a nonlinear insertion of order \(\ge q\). To assist coordinate-wise expansion we shall use later, we record a useful fact: assigning weight \(1\) to each \(X_\ell\) and weight \(0\) to \(g\), a monomial whose first nonlinear insertion occurs at the \(j\)-th activation carries weighted \(X\)-degree at least \(L + (q - 1) j\), minimized at \(j = 1\). The cheapest nonlinear configuration is therefore a single insertion at the earliest activation, with all other activations taken linear.

The first nonlinear insertion contributes at \(X\)-degree \[\label{eq:si-f} f(X, g) = \sqrt{N} \alpha^{L-1} \Bigl(\prod_{m=1}^{L} X_m\Bigr) g + O(\|X\|^{L + q - 1}),\tag{6}\] and \(\partial_{X_\ell} f = \sqrt{N} \alpha^{L-1} (\prod_{m \ne \ell} X_m) g + O(\|X\|^{L + q - 2})\). Substituting into \(\dot{X}_\ell = \beta_1 \mathbb{E}[\sigma(g) \partial_{X_\ell} f] - \mathbb{E}[f \partial_{X_\ell} f]\), the teacher term contributes at order \(L - 1\) via Stein’s identity \(h_\sigma \doteq \mathbb{E}_g[g \sigma(g)] = \mathbb{E}_g[\sigma'(g)]\), while the self-interaction is subleading at \(2L - 1\). The result is the leading-order expansion about the origin \[\label{eq:origin-ode} \dv{X_\ell}{t} = K^{(\sigma)} \prod_{m \ne \ell} X_m + R_\ell(X), \qquad K^{(\sigma)} \doteq \frac{\beta_1 h_\sigma \alpha^{L-1}}{\sqrt{N}},\tag{7}\] with \(|R_\ell(X)| = O(\|X\|^{\min(L + q - 2, 2L - 1)})\), simplifying to \(O(\|X\|^{L+1})\) for Class B activations. The leading drive \(K^{(\sigma)} \prod_{m \ne \ell} X_m\) is identical across layers; the content of the reduction lies in the differences \(X_\ell^2 - X_1^2\), where the \(\prod_{m \ne \ell}\) structure survives into the first nonlinear correction and drives a further cancellation.

3.2.0.1 Approximate imbalance invariant.

The matrix imbalance of 2 restricts to the scalar differences \(D_\ell \doteq X_\ell^2 - X_1^2\), which along 4 drift at rate \(O(\|X\|^{L+q-1})\) — one order higher than the coordinatewise remainder of 7 : the degree-\((L{+}q{-}2)\) piece of \(R_\ell\) inherits the \(\prod_{m\ne\ell}\) structure of \(\partial_{X_\ell} f\) and the \(\ell\)-independent scalar in front cancels in \(2 X_\ell R_\ell - 2 X_1 R_1\) (for Class B the drift is thus \(O(\|X\|^{L+2})\)). This near-balance is what reduces the escape analysis to one dimension: the imbalance stays small over the escape window, so \(Y \doteq X_1^2\) together with the fixed gaps \(D_\ell\) closes a separable scalar ODE. The formal statement with proof and the normal-form extension are deferred to [app:approx-balance,app:normal-form].

4 Escape Laws and the Critical-Depth Exponent↩︎

The near-balance above allows us to reduce the escape-time asymptotics of the \(L\)-dimensional reduced flow of 4 to a one-dimensional quadrature. Two distinct facts contribute. First, one-dimensional quadrature is generic: wherever the analytic reduced field is nonvanishing, the flow-box theorem [34] yields coordinates \((U, I_1, \ldots, I_{L-1})\) in which the flow transverses only \(U\) while the \(I_i\) are invariants, so the escape time is a one-dimensional integral. Second, near the small balanced ray the change of variables takes the specific near-identity shape \(U_\ell = X_\ell + O(s^q)\), giving the balanced-product flow \(\dot{U}_\ell = \Phi(U) \prod_{m \ne \ell} U_m\) whose invariants preserve the deep-linear balance law at leading order. Along any orbit the squared differences \(U_\ell^2 - U_1^2\) are exactly constant in the normal-form coordinates, so with \(U \doteq U_1\) and gaps \(D_\ell \doteq U_\ell(0)^2 - U_1(0)^2\) the dynamics collapse to \[\label{eq:1d-ode} \frac{d U}{d t} = \Phi(U, D) \sqrt{\prod_{\ell \ge 2}(U^2 + D_\ell)}, \qquad \Phi(U,D) = K^{(\sigma)}\bigl(1 + \lambda U^{q-1} + \rho(U, D)\bigr),\tag{8}\] where \(\Phi\) is the one-dimensional drive, \(\rho(U, D) = O(U^q)\), and \(\lambda\) is the analytic obstruction to full \(1\)-homogeneity of the reduced flow; the same coefficient that, against the \(U^{-(L-1)}\) weight in the escape integrand, produces a resonance at \(L = q+1\) where the first correction integrates to \(\log(1/\varepsilon)\). The formal normal form, its construction, and the resonance analysis are developed in 9. Every closed form below is an evaluation of 8 .

We refer to \(t_\mathrm{esc}^\mathrm{lead}\) as the closed-form or asymptotic escape time. Setting \(X_\ell^0 = \varepsilon\) for all \(\ell\) gives \(D_\ell = 0\) and reduces 8 to \(\dot{Y} = 2 K^{(\sigma)} Y^{L/2}\). Direct integration from \(Y_0 = \varepsilon^2\) to \(Y = 1\) gives \[\label{eq:phase-transition} t_\mathrm{esc}^\mathrm{lead, bal}(L, \varepsilon) = \begin{cases} [K^{(\sigma)}]^{-1} \log(1/\varepsilon), & L = 2,\\[3pt] \dfrac{1 - \varepsilon^{L-2}}{(L-2) K^{(\sigma)} \varepsilon^{L-2}}, & L \ge 3. \end{cases}\tag{11}\] The small-\(\varepsilon\) escape time is therefore logarithmic in two-layer networks and polynomial \(\varepsilon^{-(L-2)}\) in networks of depth three or more (9 and 2 (a)). The jump at \(L = 3\) separates the shallow (logarithmic) and deep (polynomial) scaling regimes.

4.0.0.1 Critical-depth law.

The polynomial exponent \(L - 2\) is not a property of the nominal depth but of the critical depth, defined as the number of layers at the bottleneck scale. Sort initial scales \(0 < s_1 \le \cdots \le s_L\) and suppose \(s_1 = \cdots = s_r = \varepsilon\) while \(s_{r+1}, \ldots, s_L = \Theta(1)\), with a strict hierarchy \(s_{j+1}/s_j \to \infty\) as \(\varepsilon \to 0\) across the boundary \(j = r\). A shell-by-shell evaluation of 9 on intervals \([s_j^2, s_{j+1}^2]\) where the integrand behaves as if the network had effective depth \(j\) yields the following law.

Each layer removed from the bottleneck reduces the polynomial exponent by one: the dominant shell shifts from \(r\) to \(r-1\), changing the antiderivative exponent from \(-(r-2)\) to \(-(r-3)\). In the extreme case \(r = 1\), only one layer lies at the bottleneck scale and the escape integral is \(\Theta(1)\).

4.0.0.2 Universality across activations.

3 has a single activation-dependent scalar: the prefactor \(K^{(\sigma)} = \beta_1 h_\sigma \alpha^{L-1} / \sqrt N\). The shape of the escape integral 9 is activation-agnostic, and the first correction is controlled by the single parameter \(q\), the order of the first nonlinear Taylor term of \(\sigma\) defined in 1.

The parameter \(q\) organizes the two nonlinear classes: Class B (\(q = 3\): tanh, erf, sin) has the smallest correction \(O(\varepsilon^2)\) away from the resonance \(L = 4\); Class C (\(q = 2\): GELU, Swish) has a next-to-leading factor \(1 + \gamma_C\, U\) with \(\gamma_C \doteq 3 \sigma''(0)^2 / [2 \alpha h_\sigma]\), producing an \(O(\gamma_C\, \varepsilon)\) deviation from the master curve (9). In both cases the correction is the \(\lambda U^{q-1}\) term in the renormalized quadrature of 8 . 3 plots \(K^{(\sigma)} t_\mathrm{esc}\) for five activations: after rescaling, Class B curves collapse onto the master curve with residuals consistent with the \(O(\varepsilon^{q-1})\) correction. We also note that Class D (\(\sigma(0) \ne 0\)) reduces to the centered case via \(\tilde{\sigma} = \sigma - \sigma(0)\) (treated in 9.4).

5 Multi-Mode Dynamics and Off-Manifold Corrections↩︎

The single-mode theory of 4 extends to a multi-mode teacher \(y = \sum_{k = 1}^{r^*} \beta_k \sigma(v_k^\top x)\) with \(\beta_1 > \cdots > \beta_{r^*} > 0\): each mode \(k\) breaks out along a stage-\(k\) saddle after modes \(1, \ldots, k-1\) have escaped. At generic Gaussian init, off-manifold corrections are not dependent on the on-manifold dynamics, raising two questions: (i) Does the critical-depth law of 4 depend on the ansatz, or is it intrinsic to the single-mode observable? (ii) Is the block-aligned ansatz itself an attractor, and what captures the escape-time shift when it is not? Below we resolve (i) affirmatively via a teacher-signal scalar that does not depend on any reduction (5); answering (ii) is more involved and is the subject of 11.

Fix an aligned single-mode teacher \(y = \beta_1 \sigma(v_1^\top x)\) and set \(g \doteq v_1^\top x \sim \mathcal{N}(0, 1)\), \(h_\sigma \doteq \mathbb{E}_g[g \sigma(g)]\) (the first Hermite coefficient; the escape result assumes \(h_\sigma>0\)). With \(\delta_\ell \doteq \nabla_{z_\ell} f\) the layer-\(\ell\) backprop and \(h_{\ell - 1}\) the forward activation, define the mode-\(1\) signal energy \(\gamma\) and teacher-signal gradient tensor \(G_\ell\): \[\label{eq:gamma-Gell} \gamma(W) \doteq \mathbb{E}[f(x) g], \qquad G_\ell(W) \doteq \mathbb{E}\bigl[g \delta_\ell h_{\ell - 1}^\top\bigr] \in \mathbb{R}^{n_\ell \times n_{\ell - 1}},\tag{13}\] together with \(T(W) \doteq \sum_\ell \|G_\ell\|_F^2\) (aggregate layerwise gradient mass) and \(M \doteq \max_\ell \|W_\ell\|_\mathrm{op}\) (bottleneck operator scale). Since \(\nabla_{W_\ell} \gamma = G_\ell\) by the chain rule, \(\gamma\) is an observable whose dynamics do not depend on any reduction.

The proof of 2 is chain rule plus Stein’s identity applied to the Hermite decomposition \(\sigma(g) = h_\sigma g + \sigma_\perp(g)\); the proof of 1 is Cauchy–Schwarz on the Frobenius pairing together with the filtered-composition estimates \(\|f\|_{L^2} \lesssim \prod_{m = 1}^r y_m \asymp |\gamma|\) (the lower bound \(|\gamma| \gtrsim \prod_m y_m\) using the initial-nondegeneracy hypothesis of 5 and its propagation on the bootstrap interval) and \(\|\delta_\ell h_{\ell - 1}^\top\|_{L^2} \lesssim \prod_{m \ne \ell} y_m \asymp \|G_\ell\|_F\), where \(y_m\) is the bottleneck linear-path gain in layer \(m\). Both are given in 10. Substituting 1 into ?? and using \(\gamma \le m_0^r \ll 1\) on the bootstrap interval gives the clean differential inequality \[\label{eq:gamma-dot-lower} \dot{\gamma} \ge c T(W), \qquad c = \tfrac12 \beta_1 h_\sigma,\tag{15}\] valid on \([0, \tau_{m_0}]\) for \(m_0\) small enough.

5 makes the single-mode exponent intrinsic to \(M\) and \(\gamma\) rather than to any ansatz. The product structure \(\|G_\ell\|_F \asymp \|B_\ell\|_2 \|A_{\ell-1}\|_2\) is the rank-one Frobenius identity applied to the filtered-composition expansion of 8.3, and the bootstrap interval \([0, \tau_{m_0}]\) is self-certifying since \(\tau_{m_0} \ge C_-\varepsilon^{-(r-2)}\) covers the escape window (10). Prefactors depend on off-manifold geometry; 4 confirms the exponent empirically.

5.0.0.1 Multi-mode geometry.

With \(r^*\) teacher modes, the natural extension of 1 partitions hidden neurons into \(r^*\) blocks with the single-mode ansatz imposed inside each. This block-aligned ansatz is preserved by the flow but not attracting at generic Gaussian init: linearizing around the stage-\(1\) saddle gives a positive off-block eigenvalue whenever a mixed on-block/off-block loop gain exceeds \(1\). This is treated in more detail in 11.

6 Discussion↩︎

We have shown that the homogeneity deficit \(\varphi_\sigma(z) = z\sigma'(z) - \sigma(z)\) classifies activations into four regimes and controls escape from the zero saddle. On the symmetric submanifold the matrix flow reduces to a one-dimensional integral; off the manifold at He-normal init, a signal-energy argument on \(\gamma(W) = \mathbb{E}[fg]\) yields the same exponent \(\tau_\star = \Theta(\varepsilon^{-(r - 2)})\), identifying \(r\) as the quantity that sets the plateau. Prior work on deep linear networks exploits the exact conservation law \(W_{\ell + 1}^\top W_{\ell + 1} - W_\ell W_\ell^\top = \mathrm{const}\); our identity generalizes it beyond only linear or ReLU activations, setting the escape-time depth-dependence of \(\tau_\star\).

Our analysis identifies the first mode escape time and provides approximations to successive escape times — an exact theory of modewise escape times in multi-mode teacher-student networks remains open and is an exciting area of future research. We also note that our framework is asymptotic as \(\varepsilon \to 0^+\); understanding training dynamics in arbitrarily large initialization networks is another important avenue for future work. More generally, an ab initio theory of the learning dynamics in deep neural networks remains an important goal for the field.

Acknowledgments↩︎

The authors would like to thank David Kim for many useful discussions and Arjun Banerjee for comments on the manuscript. This work was supported in part by the U.S. Army Research Laboratory and the U.S. Army Research Office under Contract No. W911NF-201-0151. DR acknowledges support from VESSL AI.

7 Proof and Extension of 1↩︎

We give the derivation of 1, state the matrix-valued refinement, and record the almost-everywhere version for ReLU.

7.1 Full Derivation of 1↩︎

Fix a layer index \(l \in \{1, \ldots, L-1\}\). Write \(\ell\) for the per-example loss and \(\mathcal{L} = \mathbb{E}[\ell]\) for the population loss, with expectation taken over the input distribution (and any label noise). The pre- and post-activations are \(z_l = W_l h_{l-1}\) and \(h_l = \sigma(z_l)\), with \(h_0 = x\) the input. We use \(\langle A, B \rangle_F \doteq \tr(A^\top B)\) for the Frobenius inner product and \(\odot\) for the Hadamard (entrywise) product.

Under gradient flow \(\dot{W}_l = -\nabla_{W_l} \mathcal{L}\), \[\begin{align} \dv{t}\|W_l\|_F^2 &= \dv{t}\tr(W_l^\top W_l)\\ &= \tr(\dot{W}_l^\top W_l) + \tr(W_l^\top \dot{W}_l)\\ &= 2 \tr(W_l^\top \dot{W}_l)\\ &= 2 \langle W_l, \dot{W}_l \rangle_F\\ &= -2 \langle W_l, \nabla_{W_l} \mathcal{L} \rangle_F. \end{align}\] To prove our claim, all that remains is to express \(\langle W_l, \nabla_{W_l} \mathcal{L} \rangle_F\) as an expectation in pre-activation space.

The per-example chain rule gives \[\begin{align} \nabla_{W_l} \ell &= \nabla_{z_l} \ell \cdot h_{l-1}^\top, \end{align}\] a rank-one outer product between the upstream gradient \(\nabla_{z_l}\ell \in \mathbb{R}^N\) and the downstream activation \(h_{l-1}\in \mathbb{R}^N\). Taking the expectation over the data distribution and substituting, \[\begin{align} \langle W_l, \nabla_{W_l} \mathcal{L} \rangle_F &= \tr\bigl(W_l^\top \mathbb{E}[\nabla_{z_l} \ell \cdot h_{l-1}^\top]\bigr)\\ &= \mathbb{E}\bigl[\tr(W_l^\top \nabla_{z_l} \ell \cdot h_{l-1}^\top)\bigr]\\ &= \mathbb{E}\bigl[\tr(h_{l-1}^\top W_l^\top \nabla_{z_l} \ell)\bigr]\\ &= \mathbb{E}\bigl[\tr((W_l h_{l-1})^\top \nabla_{z_l} \ell)\bigr]\\ &= \mathbb{E}\bigl[\tr(z_l^\top \nabla_{z_l} \ell)\bigr]\\ &= \mathbb{E}\bigl[\langle \nabla_{z_l} \ell, z_l \rangle\bigr], \end{align}\] where between the second and third lines we have used linearity of \(\tr\) and \(\mathbb{E}\), between the third and fourth lines we use cyclicity of the trace, and between the fourth and fifth lines we substitute \(z_l = W_l h_{l-1}\). Combining, \[\begin{align} \label{eq:dnorm-final} \dv{t}\|W_l\|_F^2 &= -2 \mathbb{E}\bigl[\langle \nabla_{z_l} \ell, z_l \rangle\bigr]. \end{align}\tag{17}\]

Backpropagation relates the upstream gradients at adjacent layers. Differentiating \(h_l = \sigma(z_l)\) entrywise and \(z_{l+1} = W_{l+1} h_l\), \[\begin{align} \nabla_{z_l} \ell &= \left(\frac{\partial h_l}{\partial z_l}\right)^{ \top} \nabla_{h_l} \ell = \mathop{\mathrm{diag}}(\sigma'(z_l)) \cdot W_{l+1}^\top \nabla_{z_{l+1}} \ell = \sigma'(z_l) \odot \bigl(W_{l+1}^\top \nabla_{z_{l+1}} \ell\bigr), \end{align}\] where the first equality is the Jacobian–vector chain rule, the second uses the Jacobian of \(\sigma\) applied entrywise (a diagonal matrix with entries \(\sigma'(z_l)_i\)) along with the affine Jacobian of \(z_{l+1}\) in \(h_l\), and the third rewrites the diagonal multiplication as a Hadamard product.

Take the inner product of the backprop identity with \(z_l\). Using the Hadamard inner product identity \(\langle a \odot b, c \rangle = \langle b, a \odot c \rangle\) for vectors \(a, b, c \in \mathbb{R}^N\), \[\begin{align} \langle \nabla_{z_l} \ell, z_l \rangle &= \langle \sigma'(z_l) \odot (W_{l+1}^\top \nabla_{z_{l+1}} \ell), z_l \rangle\\ &= \langle W_{l+1}^\top \nabla_{z_{l+1}} \ell, \sigma'(z_l) \odot z_l \rangle\\ &= \langle W_{l+1}^\top \nabla_{z_{l+1}} \ell, z_l \odot \sigma'(z_l) \rangle. \end{align}\] Separately, the forward pass \(z_{l+1} = W_{l+1} \sigma(z_l) = W_{l+1} h_l\) gives \[\begin{align} \langle \nabla_{z_{l+1}} \ell, z_{l+1} \rangle &= \langle \nabla_{z_{l+1}} \ell, W_{l+1} \sigma(z_l) \rangle = \langle W_{l+1}^\top \nabla_{z_{l+1}} \ell, \sigma(z_l) \rangle. \end{align}\] Subtracting the two lines inside the expectation, the common factor \(W_{l+1}^\top \nabla_{z_{l+1}} \ell\) pairs against \(z_l \odot \sigma'(z_l) - \sigma(z_l) = \varphi_\sigma(z_l)\): \[\begin{align} \mathbb{E}\bigl[\langle \nabla_{z_l} \ell, z_l \rangle\bigr] - \mathbb{E}\bigl[\langle \nabla_{z_{l+1}} \ell, z_{l+1} \rangle\bigr] &= \mathbb{E}\bigl[\langle W_{l+1}^\top \nabla_{z_{l+1}} \ell, \varphi_\sigma(z_l) \rangle\bigr]. \end{align}\] Applying 17 at layers \(l\) and \(l+1\) and subtracting, \[\begin{align} \dv{t}\bigl(\|W_{l+1}\|_F^2 - \|W_l\|_F^2\bigr) &= -2 \mathbb{E}\bigl[\langle \nabla_{z_{l+1}} \ell, z_{l+1} \rangle\bigr] + 2 \mathbb{E}\bigl[\langle \nabla_{z_l} \ell, z_l \rangle\bigr]\\ &= 2 \mathbb{E}\bigl[\langle W_{l+1}^\top \nabla_{z_{l+1}} \ell, \varphi_\sigma(z_l) \rangle\bigr], \end{align}\] which is 2 . 0◻

7.2 Layerwise-Preconditioned Identity↩︎

For fixed positive layerwise rates \(\rho_1,\ldots,\rho_L\), consider \(dW_l/ds=-\rho_l\nabla_{W_l}\mathcal{L}\) and define \[\Delta_l^{(\rho)} \doteq \rho_{l+1}^{-1}\|W_{l+1}\|_F^2-\rho_l^{-1}\|W_l\|_F^2.\] Then \[\frac{d}{ds}\Delta_l^{(\rho)}=2\mathbb{E}\!\left[\left\langle W_{l+1}^\top\nabla_{z_{l+1}}\ell,\varphi_\sigma(z_l)\right\rangle\right].\] Indeed, the calculation in 7.1 becomes \(d[\rho_l^{-1}\|W_l\|_F^2]/ds=-2\mathbb{E}\langle\nabla_{z_l}\ell,z_l\rangle\); subtracting adjacent layers gives the displayed identity. 1 is the case \(\rho_l=1\).

7.3 Matrix-Valued Refinement of 1↩︎

The rank-one chain-rule step used above lifts without a trace, giving a matrix-valued analog of 1 whose scalar trace recovers the identity.

The matrix refinement is strictly stronger than 1: it records the directional structure of the imbalance drift in the \(N \times N\) layer space, whereas the scalar identity retains only its trace. Tracking the full matrix is what allows the off-manifold analysis of 5 to decompose the drift into distinct channels (on-block versus off-block) rather than a single number.

7.4 Approximate Imbalance Invariant on the Ansatz↩︎

The matrix imbalance of 2 restricts to the symmetric balanced ansatz as the scalar differences \(D_\ell \doteq X_\ell^2 - X_1^2\). Along the flow of 4 , these differences move slowly.

Remark on why the drift is one order smaller than naive bounds predict for deep-interior pairs. The proof of 3 turns on which monomial dominates \(R_\ell\), not just its size. The degree count of 3 gives the coordinatewise remainder \(R_\ell(X) = O(\|X\|^{\min(L + q - 2, 2L - 1)})\) in 7 (simplifying to \(O(\|X\|^{L + 1})\) for Class B at \(L \ge 2\)), so termwise multiplication yields the naive bound \(\tfrac{d}{dt} D_\ell = 2 X_\ell R_\ell - 2 X_1 R_1 = O(\|X\|^{L + q - 1})\). The per-insertion analysis of 8.3 (23 24 ) sharpens the pair-drift story by identifying inside the degree-\((L + q - 2)\) piece of \(R_\ell\) a single dominant monomial: the one in which the very first activation fires the nonlinear slice \(\rho(u) = a_q u^q + \cdots\) while activations \(2, \ldots, L\) stay linear. That monomial carries the \(q\)-vs-\(1\) derivative asymmetry between \(\ell = 1\) (differentiating inside \(\rho\), factor \(q\)) and \(\ell \ge 2\) (differentiating outside \(\rho\), factor \(1\)). For pairs \((i, j)\) with \(i, j \ge 2\) the factor is the same on both sides, producing exact cancellation and the improved drift \(O(\|X\|^{L + q})\) — one full order below the naive rate — as recorded in 26 . For the \(\ell \leftrightarrow 1\) pair the factor mismatch gives a candidate term \((1 - q)c(X)\prod_mX_m\) of order \(\|X\|^{L+q-1}\); it is present when the corresponding Hermite coefficient is nonzero.

3 is the nonlinear analog of deep-linear balance: the Class B bound holds to order \(\|X\|^{L + 2}\), the Class C bound to \(\|X\|^{L + 1}\). The Class B exponent is sharper than pointwise control of 2 predicts: on the manifold, \(W_{l+1}^\top \nabla_{z_{l+1}} \mathcal{L}\) aligns with \(\varphi_\sigma(z_l)\) to produce an extra cancellation, giving \(O(\varepsilon^{L+2})\) in place of the naive \(|\varphi_\sigma(z_l)| \cdot |W_{l+1}^\top \nabla_{z_{l+1}} \mathcal{L}| = O(\varepsilon^3) \cdot O(\varepsilon^{L-3}) = O(\varepsilon^L)\). The drift can moreover be removed to all orders by a near-identity change of coordinates into a balanced product normal form \(\dot{U}_\ell = \Psi(U) \prod_{m \ne \ell} U_m\); the formal normal form, the renormalized escape law, and the \(L = q{+}1\) resonance are developed in 9. For the escape analysis, the leading-order form of 3 suffices.

7.5 Almost-Everywhere Extension and ReLU↩︎

The entire derivation above uses only almost-everywhere differentiability of \(\sigma\): every appearance of \(\sigma'\) occurs inside a Lebesgue integral (through the data expectation), and the chain rule lifts to the distributional sense. Concretely, if \(\sigma\) is Lipschitz and piecewise \(C^1\), Rademacher’s theorem provides an a.e.-defined derivative \(\sigma'\), the rank-one chain rule \(\nabla_{W_l} \ell = \nabla_{z_l} \ell \cdot h_{l-1}^\top\) holds for almost every input \(x\), and the Hadamard step \(\nabla_{z_l} \ell = \sigma'(z_l) \odot (W_{l+1}^\top \nabla_{z_{l+1}} \ell)\) is then an a.e.equality. Since the measure of pre-activations landing on the null set where \(\sigma'\) is undefined is zero under any absolutely continuous input distribution (Gaussian in particular), the expectation is unchanged by the choice of representative.

7.6 Proof of the Activation Classification↩︎

We prove 1. Write the Taylor expansion \(\sigma(z) = \sum_{k \ge 0} a_k z^k\) with \(a_1 = \sigma'(0) \ne 0\). A direct computation from \(\varphi_\sigma(z) = z \sigma'(z) - \sigma(z)\) gives \[\label{eq:phi-taylor} \varphi_\sigma(z) = \sum_{k \ge 0} (k - 1) a_k z^k = -a_0 + \sum_{k \ge 2} (k - 1) a_k z^k,\tag{18}\] so the linear term (\(k = 1\)) drops out and \(\varphi_\sigma\) is determined by the constant term and the post-linear Taylor coefficients.

Mutual exclusivity. Classes (A), (B\(_q\)), (C), (D) are pairwise disjoint:

• (A) requires \(\sigma\) linear, so \(a_k = 0\) for all \(k \ge 2\); in particular \(\sigma(0) = 0\), \(\sigma''(0) = 0\), and \(\sigma\) has no nonlinear term, disjoint from any (B\(_q\)), (C), (D).

• (B\(_q\)) requires \(\sigma\) odd and nonlinear, so \(a_0 = 0\) and all even \(a_{2k} = 0\): disjoint from (C) and (D). Different \(q \in \{3, 5, 7, \ldots\}\) are disjoint by definition (each \(q\) is the smallest odd index with \(a_q \ne 0\)).

• (C) has \(\sigma(0) = 0\), disjoint from (D); and \(\sigma''(0) \ne 0\) rules out every (B\(_q\)) (which forces \(\sigma''(0) = 0\)).

Exhaustiveness. Let \(\sigma\) satisfy the hypothesis. If \(\sigma(0) \ne 0\), we are in (D). If \(\sigma(0) = 0\) and \(\sigma''(0) \ne 0\), we are in (C). The remaining case is that \(\sigma\) is odd. If \(\sigma\) is odd and linear, we are in (A). If \(\sigma\) is odd and nonlinear, all even \(a_{2k}\) vanish and there exists a smallest odd \(q \ge 3\) with \(a_q \ne 0\); this \(q\) places \(\sigma\) in exactly one (B\(_q\)).

Taylor forms. Substituting into 18 : (A) every \(a_k\) for \(k \ne 1\) vanishes, so \(\varphi_\sigma \equiv 0\). (D) \(k = 0\) gives \(\varphi_\sigma(0) = -a_0 = -\sigma(0) \ne 0\). (C) \(a_0 = 0\) and the \(k = 2\) term contributes \(a_2 z^2 = \tfrac{\sigma''(0)}{2} z^2\). (B\(_q\)) \(a_0 = a_2 = \cdots = a_{q - 1} = 0\) by oddness and minimality of \(q\), so the \(k = q\) term contributes \((q - 1) a_q z^q\) with \(a_q = \sigma^{(q)}(0) / q! \ne 0\). Oddness of \(\sigma\) implies \(\varphi_\sigma\) is odd, so only odd-power corrections appear, giving the \(O(z^{q + 2})\) remainder. 0◻

8 Ansatz Invariance and Scalar Reduction↩︎

This appendix collects the proofs underlying 3: we verify that the symmetric balanced ansatz of 1 is the \(S_N\)-fixed-point manifold of hidden-neuron permutation symmetry, prove it is flow-invariant under population gradient flow, descend the matrix flow to a scalar flow with unit conversion factors, record the explicit closed form of \(\partial_{X_\ell} f\) that enters 4 , and derive the filtered-composition remainder bounds that drive 6 , 7 , and 3.

8.1 The Ansatz as \(S_N\)-Fixed-Point Manifold↩︎

Fix a hidden layer index \(l \in \{1, \ldots, L-1\}\) and let \(\pi \in S_N\) act on \(W\) by simultaneous permutation of the \(N\) hidden neurons in layer \(l\): the rows of \(W_l\) are permuted by \(\pi\), and the columns of \(W_{l+1}\) are permuted by \(\pi\) (with the convention that at \(l = L - 1\), the scalar readout \(W_L \in \mathbb{R}^{1 \times N}\) has its \(N\) entries permuted). Under a single-mode teacher \(y = \beta_1 \sigma(v_1^\top x)\) with Gaussian inputs and population squared loss, the data law, the loss, and the teacher are each invariant under this action, so the population loss \(\mathcal{L}\) and its gradient satisfy \(\mathcal{L}(\pi \cdot W) = \mathcal{L}(W)\) and \(\nabla_W \mathcal{L}(\pi \cdot W) = \pi \cdot \nabla_W \mathcal{L}(W)\). The fixed-point set \(\{W : \pi \cdot W = W \text{ for all } \pi \in S_N \text{ acting on any hidden layer}\}\) is precisely the submanifold on which every neuron in each hidden layer shares the same incoming row and the same outgoing column. Aligning the shared input direction of layer \(1\) with \(\hat{w} \in \mathbb{R}^d\) (a free unit vector that is flow-invariant for a single-mode teacher with \(\hat{w} = v_1\)), we recover 1. Since the gradient is equivariant under the group action, it is tangent to the fixed-point set, and the flow preserves the manifold. The next lemma records this explicitly by showing that each \(\nabla_{W_l} \mathcal{L}\) inherits the entry-pattern of \(W_l\) itself.

8.2 Descent to the Scalar Flow↩︎

Because each entry of \(W_l\) is a linear function of the scalar \(X_\ell\) under the ansatz, the normalized-metric matrix flow descends cleanly to a flow on \((X_1, \ldots, X_L)\). In physical matrix time \(s\), its scalar velocity is \(dX_\ell/ds=-N^{-1}\partial_{X_\ell}\mathcal{L}\) for every layer; rescaling to \(t=s/N\) gives 4 . For \(\rho_1=1\), \(\rho_{\ell\ge2}=N^{-1}\), the weighted imbalance of 7.2 satisfies \(N^{-1}\Delta_l^{(\rho)}=X_{l+1}^2-X_l^2\) on the ansatz.

8.2.0.1 Layer \(1\).

\(W_1 = X_1 \mathbf{1}_N \hat{w}^\top\) has \(\partial W_1 / \partial X_1 = \mathbf{1}_N \hat{w}^\top\), and the entrywise gradient flow \(\dot{W}_1 = -\nabla_{W_1} \mathcal{L}\) gives \[\dot{X}_1 \mathbf{1}_N \hat{w}^\top = -\nabla_{W_1} \mathcal{L} = -\mathbb{E}_g\bigl[q_1(g) g\bigr] \mathbf{1}_N v_1^\top.\] Matching on both sides with \(\hat{w} = v_1\) yields \[\frac{dX_1}{ds} = -\mathbb{E}_g\bigl[q_1(g) g\bigr] = -\frac{1}{N}\partial_{X_1} \mathcal{L} = -\frac{1}{N}\mathbb{E}_g\bigl[(f - \beta_1 \sigma(g)) \partial_{X_1} f\bigr].\]

8.2.0.2 Internal layers \(2 \le l \le L - 1\).

\(W_l\) has \(N^2\) entries each equal to \(X_l / N\), so \(\partial W_l / \partial X_l = \mathbf{1}_N \mathbf{1}_N^\top / N\). Under the normalized metric, \(dW_l/ds=-(1/N)\nabla_{W_l}\mathcal{L}\), hence \(dX_l/(Nds) = -(1/N)\partial \mathcal{L} / \partial W_l[i,j]\) for every \((i,j)\). Summing over the \(N^2\) entries, \[N \frac{dX_l}{ds} = -\frac{1}{N}\sum_{i,j} \partial \mathcal{L} / \partial W_l[i,j] = -\partial_{X_l} \mathcal{L},\] where the last equality uses \(\partial_{X_l} \mathcal{L} = \sum_{i,j} (1/N) \partial \mathcal{L} / \partial W_l[i,j]\). Thus \(dX_l/ds = -N^{-1}\partial_{X_l}\mathcal{L}\).

8.2.0.3 Layer \(L\).

\(W_L\) has \(N\) entries each equal to \(X_L / \sqrt{N}\), so \(\partial W_L / \partial X_L = \mathbf{1}_N^\top / \sqrt{N}\). The analogous normalized-metric matching gives \[\sqrt{N}\, \frac{dX_L}{ds} = -\frac{1}{N}\sum_i \partial \mathcal{L} / \partial W_L[i] = -\frac{1}{\sqrt{N}}\partial_{X_L} \mathcal{L},\] so \(dX_L/ds = -N^{-1}\partial_{X_L}\mathcal{L}\). Combining the three cases and rescaling \(t=s/N\), \[\label{eq:exact-flow-app} \dv{X_\ell}{t} = -\partial_{X_\ell} \mathcal{L} = -\mathbb{E}_g\bigl[(f - \beta_1 \sigma(g)) \partial_{X_\ell} f\bigr], \qquad \ell = 1, \ldots, L,\tag{19}\] which is the exact reduced flow of 4 .

8.2.0.4 Explicit closed form of \(\partial_{X_\ell} f\).

With the scalar pre-activations \(c_0 \doteq g\), \(c_1 \doteq X_1 g\), and \(c_\ell \doteq X_\ell \sigma(c_{\ell-1})\) for \(2 \le \ell \le L - 1\) so that \(f(x) = \sqrt{N} X_L \sigma(c_{L-1})\), the chain rule along the composition \(g \mapsto c_1 \mapsto \cdots \mapsto c_{L-1} \mapsto f\), together with \(\partial_{X_\ell} c_\ell = \sigma(c_{\ell-1})\) from the recursion, yields \[\partial_{X_\ell} f = \sqrt{N} \Bigl(\prod_{m > \ell} \sigma'(c_{m-1}) X_m\Bigr) \sigma(c_{\ell-1}), \qquad 1 \le \ell \le L,\] with the empty-product convention \(\partial_{X_L} f = \sqrt{N} \sigma(c_{L-1})\). Substituting into 19 makes 4 fully explicit in the scalar parameters \(X\) alone, and we have completed the proof of 2.

8.3 Filtered Composition and Remainder Bounds↩︎

Write \[\sigma(u) = \alpha u + \rho(u), \qquad \rho(u) = O(u^q),\] with \(q = 3\) for Class B and \(q = 2\) for Class C. Assign every scalar \(X_\ell\) weighted degree \(1\) and \(g\) weighted degree \(0\). The reduced pre-activation \(\widetilde{z}_L\) is a composition of multiplications by \(X_\ell\) and activations: each monomial in its expansion therefore corresponds to a choice, at each of the \(L - 1\) internal activations, of either the linear branch \(\alpha u\) or one nonlinear insertion \(\rho\).

3 implies 6 directly.

8.3.0.1 Remainder bound for \(\dot{X}_\ell\) (7 ).

The exact scalar-chain gradient is \(\dot{X}_\ell = \beta_1 \mathbb{E}_g[\sigma(g) \partial_{X_\ell} f] - \mathbb{E}_g[f \partial_{X_\ell} f]\). For the teacher term, 21 together with Stein’s identity gives \[\beta_1 \mathbb{E}_g[\sigma(g) \partial_{X_\ell} f] = \sqrt{N} \beta_1 h_\sigma \alpha^{L-1} \Bigl(\prod_{m \ne \ell} X_m\Bigr) + O \bigl(\|X\|^{L + q - 2}\bigr),\] where the correction comes from the first nonlinear insertion in \(\partial_{X_\ell} f\) paired with \(\sigma(g)\) and collapsed by Hermite orthogonality. For the self-interaction, 20 21 give \(|f \partial_{X_\ell} f| = O(\|X\|^{L + (L-1)}) = O(\|X\|^{2L - 1})\). Subtracting, \[\dot{X}_\ell = K^{(\sigma)} \Bigl(\prod_{m \ne \ell} X_m\Bigr) + R_\ell(X), \qquad |R_\ell(X)| = O \bigl(\|X\|^{\min(L + q - 2, 2L - 1)}\bigr),\] with \(K^{(\sigma)} = \beta_1 h_\sigma \alpha^{L-1} / \sqrt{N}\). For Class B (\(q = 3\)) and \(L \ge 2\), \(L + q - 2 = L + 1 \le 2L - 1\), so \(|R_\ell(X)| = O(\|X\|^{L+1})\), which is the exponent in 7 .

Leading coefficient. We now identify the degree-\((L + q - 2)\) contribution to \(R_\ell\) explicitly as coming from the first-activation (\(j = 1\)) nonlinear insertion, with the form \[\label{eq:R-ell-leading} R_\ell(X) = q_\ell \beta_1 \alpha^{L-2} a_q h_\sigma^{(q)} X_1^{q-1} \prod_{m \ne \ell} X_m + O \bigl(\|X\|^{L + q - 1}\bigr),\quad q_1 = q,\;q_\ell = 1\;(\ell \ge 2),\tag{22}\] with \(h_\sigma^{(q)} \doteq \mathbb{E}_g[\sigma(g) g^q]\). The local Class B hypothesis ensures \(\alpha\ne0\) and \(a_q\ne0\), but does not ensure \(h_\sigma^{(q)}\ne0\). For \(q=3\), the Hermite expansion gives \[h_\sigma^{(3)} = \mathbb{E}_g[\sigma(g) g^3] = 6 s_3 + 3 s_1 = 6 s_3 + 3 h_\sigma.\] If \(h_\sigma^{(q)}\ne0\), the displayed coefficient is nonzero and the order is attained. If it vanishes, the displayed term cancels and the remainder is higher order; all balance and escape bounds use only the stated upper bound.

8.3.0.2 Remainder bound for the imbalance drift (3).

The filtered composition gives us more than the minimum degree alone. The degree-\((L + q - 2)\) correction to \(\partial_{X_\ell} f\) comes from the first nonlinear insertion at \(j = 1\); we can extract its structure by writing \[f_{[j = 1, \mathrm{lead}]}(X, g) = \sqrt N \alpha^{L - 2} \Bigl(\prod_{m = 2}^{L} X_m\Bigr) \rho(X_1 g)\] (linear composition through activations \(2, \ldots, L - 1\) downstream of the single nonlinear insertion \(\rho(X_1 g) = a_q (X_1 g)^q + O(X_1^{q + 2} g^{q + 2})\) at activation \(1\)). This expression is manifestly symmetric in \(X_2, \ldots, X_L\); the downstream composition is linear, so the layer-specific structure visible in the full forward pass collapses to a simple product. Differentiating: \[\partial_{X_\ell} f_{[j = 1, \mathrm{lead}]} = \begin{cases} \sqrt N \alpha^{L - 2} \bigl(\prod_{m = 2}^{L} X_m\bigr) \rho'(X_1 g) g & (\ell = 1),\\[3pt] \sqrt N \alpha^{L - 2} \bigl(\prod_{m = 2, m \ne \ell}^{L} X_m\bigr) \rho(X_1 g) & (\ell \ge 2). \end{cases}\] For \(\ell \ge 2\), multiplying and dividing by \(X_1\) recasts this as \(\sqrt N \alpha^{L - 2} X_1^{-1} \rho(X_1 g) \cdot \prod_{m \ne \ell} X_m\) — the same \(\prod_{m \ne \ell}\) structure as the leading linear-path term, with an \(\ell\)-independent scalar coefficient. Expanding \(\rho\) and contracting against \(\beta_1 \mathbb{E}_g[\sigma(g) \cdot]\) using Hermite orthogonality gives, for each \(\ell \ge 2\), \[R_\ell^{(\mathrm{lead})}(X) = \beta_1 \alpha^{L - 2} a_q h_\sigma^{(q)} X_1^{q - 1} \prod_{m \ne \ell} X_m + O(\|X\|^{L + q - 1}), \qquad h_\sigma^{(q)} \doteq \mathbb{E}_g[\sigma(g) g^q],\] with coefficient \(c(X) \doteq \beta_1 \alpha^{L - 2} a_q h_\sigma^{(q)} X_1^{q - 1}\) that is the same for every \(\ell \ge 2\). For \(\ell = 1\), \(\partial_{X_1}\) passes through \(\rho\), producing a factor of \(q\): \(R_1^{(\mathrm{lead})} = q c(X) \prod_{m \ne 1} X_m + O(\|X\|^{L + q - 1})\). As a result, for any pair \(i, j \in \{2, \ldots, L\}\), \[X_i R_i - X_j R_j = c(X) \bigl(X_i \prod_{m \ne i} X_m - X_j \prod_{m \ne j} X_m\bigr) + O(\|X\|^{L + q}) = O(\|X\|^{L + q}),\] because \(X_i \prod_{m \ne i} X_m = \prod_m X_m = X_j \prod_{m \ne j} X_m\) independently of \(i, j\). Hence differences among layers \(\{2, \ldots, L\}\) are conserved at the order-\((L + q - 2)\) correction. For the \(\ell \leftrightarrow 1\) comparison, the factor-\(q\) mismatch between \(R_1\) and \(R_\ell\) leaves a surviving drift \[\dv{t}(X_\ell^2 - X_1^2) = 2(X_\ell R_\ell - X_1 R_1) = 2(1 - q) c(X) \prod_m X_m + O(\|X\|^{L + q}) = O \bigl(\|X\|^{L + q - 1}\bigr)\] of degree exactly \(L + q - 1 = L + 2\) for Class B.

8.3.0.3 Higher-order corrections.

The argument above handles the first nonlinear insertion (\(j = 1\)) and establishes conservation of pairs \((X_i, X_j)\) with \(i, j \ge 2\) at order \(L + q - 2\). To track deeper orders we must enumerate all single nonlinear insertions: by the filtered-composition expansion (3), a monomial arising from a single \(a_q u^q\) insertion at activation \(k \in \{1, \ldots, L\}\), with all other activations linearized, contributes a term of the form \[\label{eq:insertion-k} f_{[j = k, \mathrm{lead}]}(X, g) = \sqrt N \alpha^{L - k - 1} \Bigl(\prod_{m = k + 1}^{L} X_m\Bigr) \rho_k\bigl(\alpha^{k - 1} X_1 X_2 \cdots X_k g\bigr),\tag{23}\] where \(\rho_k(u) = a_q u^q + O(u^{q + 2})\) and the convention is \(\prod_{m = L + 1}^{L} = 1\), \(\alpha^{-1} = 1\) in the \(k = L\) endpoint. The structure is: activations \(1, \ldots, k - 1\) contribute linear factors \(\alpha X_\ell\); activation \(k\) contributes the first nonlinear slice \(\rho_k\); activations \(k + 1, \ldots, L\) contribute linear factors \(\alpha X_\ell\) outside the nonlinear kernel. Differentiating 23 with respect to \(X_\ell\) produces, after the same \(X_\ell^{-1}\) homogenization used at \(k = 1\), a term proportional to \(F_k(X) \cdot \prod_{m \ne \ell} X_m\) with \[F_k(X) \cdot \text{(factor)} = \begin{cases} q F_k(X) & \text{if } \ell \in \{1, \ldots, k\}, \\ F_k(X) & \text{if } \ell \in \{k + 1, \ldots, L\}, \end{cases}\] because \(\rho_k\) is \(q\)-homogeneous in each of the upstream factors \(X_1, \ldots, X_k\) inside its argument, while the \(k + 1, \ldots, L\) downstream factors multiply \(\rho_k\) linearly. This \(q\)-vs-\(1\) asymmetry is the source of all conservation laws. Define the insertion class \(\mathcal{U}_k \doteq \{1, \ldots, k\}\) (upstream) and \(\mathcal{D}_k \doteq \{k + 1, \ldots, L\}\) (downstream). Then the contribution of the order-\(k\) insertion to \(X_\ell R_\ell\) (via \(\beta_1 \mathbb{E}_g[\sigma(g) \partial_{X_\ell} f]\) reductions to Gaussian moments, which depend only on the total power of \(g\) and hence on \(k, q\) and not on \(\ell\)) is \[\label{eq:per-insertion} (X_\ell R_\ell)\bigr|_{\mathrm{from }\;k} = \Phi_k(X) \cdot \bigl( q \mathbf{1}_{\ell \in \mathcal{U}_k} + \mathbf{1}_{\ell \in \mathcal{D}_k}\bigr) \prod_m X_m + (\text{higher order}),\tag{24}\] with \(\Phi_k(X)\) an \(\ell\)-independent scalar whose minimum total \(X\)-degree is \(L + (q - 1) k - 1\) (one factor of \(q\) raises the upstream-block total degree from \(k\) to \(q k\); the downstream block contributes \(L - k\); together with the \(\sqrt N\) and Gaussian-moment factors, \(\Phi_k\) has degree \(\ge L + (q - 1) k - 1\), so the full per-insertion contribution has degree \(\ge L + (q - 1) k\)).

8.3.0.4 Imbalance drift between two layers.

Fix \(1 \le i < j \le L\). Summing 24 over all insertions \(k\), \[\label{eq:pair-drift} \dv{t}(X_i^2 - X_j^2) = 2 \sum_{k = 1}^{L} \Phi_k(X) \bigl[ (q \mathbf{1}_{i \in \mathcal{U}_k} + \mathbf{1}_{i \in \mathcal{D}_k}) - (q \mathbf{1}_{j \in \mathcal{U}_k} + \mathbf{1}_{j \in \mathcal{D}_k})\bigr] \prod_m X_m + O(\|X\|^{L + (q - 1)L}).\tag{25}\] The bracket vanishes iff \(i\) and \(j\) lie in the same block \(\mathcal{U}_k\) or \(\mathcal{D}_k\), i.e., iff \(k < i\) or \(k \ge j\). It is nonzero (taking value \(1 - q\)) for \(k \in \{i, i + 1, \ldots, j - 1\}\). Hence the minimum \(k\) with a nonzero bracket is \(k = i\), giving \[\label{eq:pair-drift-sharp} \dv{t}(X_i^2 - X_j^2) = 2 (1 - q) \Phi_i(X) \prod_m X_m x+ O \bigl(\|X\|^{L + (q - 1)(i + 1)}\bigr) = O \bigl(\|X\|^{L + (q - 1) i}\bigr).\tag{26}\] Specializing to the case \(i = 1\) recovers the drift 23 -based computation at the top of this section; specializing to \(i, j \ge 2\) gives drift \(= O(\|X\|^{L + (q - 1) \cdot 2}) = O(\|X\|^{L + 2(q - 1)})\), one order tighter than \(L + q - 1\). For Class B (\(q = 3\)) this means: \(\dv{t}(X_\ell^2 - X_1^2) = O(\|X\|^{L + 2})\) (Prop 3), while \(\dv{t}(X_i^2 - X_j^2) = O(\|X\|^{L + 4})\) for \(i, j \ge 2\). The tighter bound for \(i, j \ge 2\) is not needed for the escape analysis: all bounds in 3 use only the weaker \(L + q - 1\) drift rate of 3.

9 Asymptotic Evaluations and Normal Forms↩︎

Here we provide supporting material for 4. We give three distinct statements:

1. an exact analytic scalarization of the reduced flow on any nonvanishing open set, provided by the analytic flow-box theorem; this is what makes the one-dimensional quadrature of 3 exact, and is recorded in 9.2;

2. a formal balanced normal form on the punctured cone around the balanced ray, which sharpens (i) into the specific balanced-product shape \(\dot{U}_\ell = \Psi(U) \prod_{m \ne \ell} U_m\) with \(U_\ell = X_\ell + O(s^q)\); this is 6, and its construction is term-by-term formal series in 9.1;

3. an asymptotic near-identity reduction that the escape analysis actually uses: only the first correction (the resonance-detecting term of 3) and the overall near-identity shift of the upper integration limit are consumed, so the formal series is truncated at low order.

The later subsections use these in combination: 9.3 proves the \((1 + o(1))\) accuracy of 3, the balanced-init scaling 11 , the bottleneck-shell decomposition, and the critical-depth law 4; and 9.4 handles the Class C universality-breaking coefficient \(\gamma_C\) and the Class D centering device that absorbs a nonzero \(\sigma(0)\) into a teacher shift.

9.1 Formal Balanced Normal Form↩︎

Write the ansatz-reduced vector field of 4 as \(V = V_0 + W\), where \(V_0 = K^{(\sigma)} \sum_\ell \bigl(\prod_{m \ne \ell} X_m\bigr) \partial_{X_\ell}\) is the leading-order drive and \(W = O(s^{L + q - 2})\) collects higher-order terms. Use polar-like coordinates \(X_L = s\), \(X_i = s \sqrt{1 + y_i}\) for \(i < L\) on the punctured cone \[\label{eq:punctured-cone} \mathcal{C}_{\delta, \kappa} \doteq \{s \in (0, \delta),\;|y_i| < \kappa,\;i < L\}.\tag{27}\] A direct computation gives \(V_0 = K^{(\sigma)} s^{L - 1} \Omega(y) \partial_s\) with \(\Omega(y) \doteq \prod_{i < L} \sqrt{1 + y_i}\) and \(V_0(y_i) = 0\), so leading-order transport is pure \(s\)-flow with \(y\) frozen.

The content of 6 is not existence of a flow-rectifying change of variables (which is generic away from critical points) but its specific asymptotic shape: \(U_i = X_i + O(s^q)\) preserves the deep-linear balance law at leading order near the small balanced ray. For \(1\)-homogeneous \(\sigma\) (linear, ReLU), \(W \equiv 0\) and \(I_i = s^2 y_i\) exactly, recovering the classical deep-linear balance \(X_i^2 - X_j^2 = \mathrm{const}\).

9.2 Exact Scalarization of the Reduced Flow↩︎

The formal balanced normal form is the near-balanced-ray statement; it rests on the fact that the analytic reduced vector field is scalarizable on any nonvanishing open set of the foliation, and that escape time is a one-dimensional quadrature of the resulting scalar speed.

Restricting 6 to the balanced ray \(X_1 = \cdots = X_L = U\) and keeping the first nontrivial series term yields the scalar normal form \[\label{eq:scalar-normal-form} \dot{U} = K^{(\sigma)} U^{L - 1} + C_1^{(\sigma)} U^{L + q - 2} + O(U^{L + 2 q - 3}),\tag{28}\] with \[C_1^{(\sigma)} = \frac{\beta_1 a_q \alpha^{L - 1} \mu_{q + 1}}{\sqrt N}, \qquad \mu_{q + 1} \doteq \mathbb{E}[g^{q + 1} \sigma(g)] / \alpha,\] To derive \(C_1^{(\sigma)}\): on the balanced ray with \(X_\ell = U\) for all \(\ell\), a single nonlinear insertion at the first activation replaces the linear branch \(\alpha c_0 = \alpha U g\) with \(a_q (U g)^q\), propagated through \(L - 1\) downstream linear branches each contributing \(\alpha U\). The resulting monomial is \(\sqrt{N} a_q \alpha^{L - 2} U^{L + q - 1} g^q\); pairing with the teacher drive \(\beta_1 \mathbb{E}[\sigma(g) \cdot g^q] = \beta_1 \alpha \mu_{q+1}\) and differentiating in \(U\) produces the coefficient above. The same filtered-composition template of 3 at general \(L\) ensures no lower-order insertions contribute. The corresponding renormalized escape quadrature on the balanced leaf is then the following.

9.3 Escape-Time Accuracy and the Critical-Depth Law↩︎

We prove the \((1 + o(1))\) statement of 3, the balanced-init scaling law 11 , and the shell decomposition that underlies 4.

9.3.0.1 Exact-to-leading-order accuracy.

Pass to \(Y \doteq X_1^2\) with gaps \(D_\ell \doteq X_\ell^2 - X_1^2\). At leading order \(\dot{X}_\ell = K^{(\sigma)} \prod_{m \ne \ell} X_m\), so \[\dot{Y} = 2 X_1 \dot{X}_1 = 2 K^{(\sigma)} \prod_{m = 1}^L X_m = 2 K^{(\sigma)} \sqrt{Y \prod_{\ell \ge 2}(Y + D_\ell)}.\] The leading \(1\)-D ODE 8 is thus exact at leading order. The exact ODE carries two corrections. First, the coordinatewise remainder \(R_\ell(X)\) of 7 , of size \(O(\|X\|^{L + 1})\), feeds each \(\dot{X}_\ell\); multiplied by \(X_1\) it contributes \(2 X_1 R_1 = O(\|X\|^{L + 2})\) to \(\dot{Y}\). Second, by 3 the gaps \(D_\ell\) drift at rate \(O(\|X\|^{L + 2})\). Both aggregate into \[\label{eq:exact-1d} \dot{Y} = 2 K^{(\sigma)} \sqrt{Y \prod_{\ell \ge 2}(Y + D_\ell)} + \widetilde{R}(Y, D), \qquad |\widetilde{R}| = O(\|X\|^{L + 2}).\tag{33}\] The relative-remainder estimate is local, so we do not take a supremum up to \(Y=1\). Set \(U_\varepsilon \doteq [\log(1/\varepsilon)]^{-1}\) and split at \(Y_\varepsilon=U_\varepsilon^2\). On \([Y_0,Y_\varepsilon]\), the normal-form remainder and gap drift give a relative integrand error \(O(U_\varepsilon^2)=o(1)\); a continuity bootstrap on this same interval transfers the estimate from the leading to the exact trajectory. Thus the inner exact and leading flight times agree up to \(1+o(1)\). On \([Y_\varepsilon,U_*^2]\), for fixed sufficiently small \(U_*\), both flows have speed comparable to their leading drive, so the flight time is \(O(\log(1/U_\varepsilon))=O(\log\log(1/\varepsilon))\) for \(L=2\) and \(O(U_\varepsilon^{-(L-2)})=O((\log(1/\varepsilon))^{L-2})\) for \(L\ge3\). The remaining compact segment \([U_*^2,1]\) has \(O(1)\) duration. These outer contributions are negligible relative to the total leading time, respectively \(\Theta(\log(1/\varepsilon))\) and \(\Theta(\varepsilon^{-(L-2)})\), proving \(t_\mathrm{esc}^\mathrm{exact}/t_\mathrm{esc}^\mathrm{lead}=1+o(1)\).

9.3.0.2 Bottleneck-shell asymptotics.

Both 4 and the ranking rule for non-balanced initialization rest on the same shell decomposition of the escape integrand.

Under the get-rich specialization \(s_1 = \cdots = s_r = \varepsilon\) and \(s_{r + 1}, \ldots, s_L = \Theta(1)\) of 4: shells \(j < r\) have zero width (\(s_j = s_{j + 1} = \varepsilon\)); shell \(j = r\) has integrand \(Y^{-r/2}/\prod_{i > r} s_i\) on \([\varepsilon^2, s_{r + 1}^2]\), where the upper limit \(s_{r + 1}^2 = \Theta(1)\) by the hierarchy hypothesis; shells \(j > r\) are over bounded intervals on which \(Y = \Theta(1)\) and contribute \(O(1)\). The only shell whose integral can diverge as \(\varepsilon \to 0\) is \(j = r\), because the integrand \(Y^{-r/2}\) is singular at \(Y = 0\) only there; the \(s_{r + 1} = \Theta(1)\) upper limit is critical to this argument: it localizes the divergence entirely to the \(j = r\) shell and ensures the \(r = 2\) case \(\int_{\varepsilon^2}^{s_3^2} Y^{-1} dY = 2 \log(s_3 / \varepsilon)\) produces exactly one \(\log(1/\varepsilon)\). The leading contribution is shell \(r\): \[\int_{\varepsilon^2}^{s_{r + 1}^2} \frac{d Y}{Y^{r/2} \prod_{i > r} s_i} = \frac{1}{\prod_{i > r} s_i} \int_{\varepsilon^2}^{s_{r + 1}^2} Y^{-r/2} d Y,\] and antidifferentiation splits by the exponent \(-r/2\): \[\begin{align} r = 1: \quad \int_{\varepsilon^2}^{s_2^2} Y^{-1/2} d Y &= 2(s_2 - \varepsilon) = \Theta(1),\\ r = 2: \quad \int_{\varepsilon^2}^{s_3^2} Y^{-1} d Y &= 2 \log(s_3 / \varepsilon) = 2 \log(1/\varepsilon) + O(1),\\ r \ge 3: \quad \int_{\varepsilon^2}^{s_{r + 1}^2} Y^{-r/2} d Y &= \frac{2}{r - 2} \varepsilon^{2 - r}(1 + o(1)). \end{align}\] These three cases are exactly the constant, logarithmic, and power-law escape laws of 4; shells \(j > r\) fold into the overall \((1 + o(1))\) factor. This completes the proof of 4.

9.4 Class C and Class D Specific Corrections↩︎

9.4.0.1 Class C next-to-leading correction.

Let \(\sigma(u) = \alpha u + (c/2) u^2 + (d/6) u^3 + O(u^4)\) with \(c = \sigma''(0) \ne 0\). We compute the universality-breaking ratio \(\gamma_C\) on the balanced ray \(X_\ell = U\) for all \(\ell\) (distinct from the signal energy \(\gamma(W)\) of 5; subscript \(C\) denotes the Class-C coefficient throughout).

A single-step induction on the scalar chain \(c_\ell = U \sigma(c_{\ell - 1})\) with \(c_1 = U g\) gives, for \(\ell \ge 2\), \[\label{eq:chain-expansion} c_\ell = \alpha^{\ell - 1} U^\ell g + \tfrac{c}{2} \alpha^{\ell - 2} U^{\ell + 1} g^2 + O(U^{\ell + 2});\tag{35}\] at each step, the \((c/2) c_{\ell - 1}^2\) term in \(\sigma(c_{\ell - 1})\) is of order \(U^{2(\ell - 1)} \ge U^{\ell + 1}\) for \(\ell \ge 3\), so the linear branch dominates the propagation of the leading nonlinear insertion. Differentiating the chain \(\partial_{X_1} f = \sqrt N g \bigl(\prod_{\ell \ge 2} X_\ell\bigr) \prod_{\ell = 1}^{L - 1} \sigma'(c_\ell)\) and substituting 35 into \(\sigma'(c_\ell) = \alpha + c\, c_\ell + O(c_\ell^2) = \alpha + c\, \alpha^{\ell - 1} U^\ell g + O(U^{\ell + 1})\), only the \(\ell = 1\) factor contributes at order \(U\), yielding on the balanced ray \[\partial_{X_1} f\bigr|_{\mathrm{bal}} = \sqrt N\, \alpha^{L - 1} U^{L - 1} g + \sqrt N\, c\, \alpha^{L - 2} U^L g^2 + O(U^{L + 1}).\] Pairing with the teacher drive and using \(\mathbb{E}[\sigma(g) g] = h_\sigma\) and \(\mathbb{E}[\sigma(g) g^2] = (c/2) \mathbb{E}[g^4] = 3 c / 2\) (odd Gaussian moments vanish) gives \[\beta_1 \mathbb{E}_g[\sigma(g) \partial_{X_1} f]\bigr|_{\mathrm{bal}} = \sqrt N\, \beta_1 \alpha^{L - 1} h_\sigma\, U^{L - 1} + \sqrt N\, \beta_1 \tfrac{3 c^2}{2} \alpha^{L - 2}\, U^L + O(U^{L + 1});\] the student self-interaction is \(O(U^{2L - 1})\) and hence subleading for \(L \ge 2\). The ratio of the \(U^L\) and \(U^{L - 1}\) coefficients is \[\gamma_C \doteq \frac{3 c^2}{2 \alpha h_\sigma} \qquad \text{for every } L \ge 2,\] \(L\)-independent by cancellation of \(\alpha^{L - 2}\). Symmetry in \(X_2, \ldots, X_L\) at the balanced point gives the same ratio in every \(\dot{X}_\ell\), so \[\label{eq:classC-ode} \dot{X}_\ell = K^{(\sigma)} \prod_{m \ne \ell} X_m \cdot \bigl[1 + \gamma_C\, X_\ell + O(\varepsilon^2)\bigr].\tag{36}\] For Class B, \(c = 0\) by oddness and \(\gamma_C = 0\). Substituting 36 into the \(1\)-D reduction along \(X_\ell = \sqrt Y\) yields an \(O(\gamma_C\, \varepsilon)\) relative correction to \(t_\mathrm{esc}^\mathrm{lead}\), vanishing as \(\varepsilon \to 0\) — the Class C branch of 1.

Class D reduces to the centered case. For \(\sigma(0) = c_0 \ne 0\), the substitution \(\tilde{\sigma}(u) \doteq \sigma(u) - c_0\) absorbs the constant into a teacher shift and relocates the saddle from \(X = 0\) to \(X^\ast = O(c_0)\); the centered activation \(\tilde{\sigma}\) falls into Class B or Class C by the parity of its leading post-linear Taylor term (e.g., sigmoid \(\to\) Class B, softplus \(\to\) Class C), and around \(X^\ast\) the reduced dynamics regain the Class B/C form with the same leading constant \(K^{(\sigma)} = \beta_1 \alpha^{L - 1} h_\sigma / \sqrt N\) and an additional relative \(O(c_0 \varepsilon)\) correction.

10 Off-Manifold Single-Mode Exponent Proofs↩︎

This appendix proves [prop:signal-energy,lem:S-bound,thm:offmanifold] and records the \(\varepsilon\)-degree argument that underlies the AM-GM step in the proof of the main theorem. Notation is the same as as in 5: \(\gamma(W) = \mathbb{E}[f g]\), \(g \doteq v_1^\top x \sim \mathcal{N}(0, 1)\), \(G_\ell = \mathbb{E}[g \delta_\ell h_{\ell - 1}^\top]\), \(T(W) = \sum_\ell \|G_\ell\|_F^2\), \(M = \max_\ell \|W_\ell\|_\mathrm{op}\), \(\delta_\ell = \nabla_{z_\ell} f\) and \(h_{\ell - 1} = \sigma(z_{\ell - 1})\).

10.0.0.1 First-Hermite sign hypothesis.

Stein’s identity gives \(h_\sigma \doteq \mathbb{E}_g[g\sigma(g)]=\mathbb{E}_g[\sigma'(g)]\). The escape arguments below assume \(h_\sigma>0\), the sign needed to turn \(\beta_1h_\sigma T(W)\) into a positive drive. This is not implied by Class B/C: \(\sigma(u)=u-au^3\) is Class B but has \(h_\sigma=1-3a\). Thus positivity is an explicit hypothesis (and is checked directly for each activation used in the experiments), not a consequence of the local Taylor class.

We now record the two \(L^2\) estimates that are the gradient-tensor analogu of 3 and will be invoked twice in the sequel: once in the signal-energy bound and once in the self-interaction bound.

10.1 Proof of the Signal-Energy Identity↩︎

By the chain rule, \(\dv{\gamma}{t} = \sum_\ell \langle \nabla_{W_\ell} \gamma, \dot{W}_\ell\rangle_F = -\sum_\ell \langle \nabla_{W_\ell} \gamma, \nabla_{W_\ell} \mathcal{L}\rangle_F\). Since \(\gamma = \mathbb{E}[g f(x; W)]\) and \(\partial_{W_\ell} f = \delta_\ell h_{\ell - 1}^\top\) (standard backprop), \(\nabla_{W_\ell} \gamma = \mathbb{E}[g \delta_\ell h_{\ell - 1}^\top] = G_\ell\). The population squared-loss gradient is \(\nabla_{W_\ell} \mathcal{L} = \mathbb{E}[(f - y) \delta_\ell h_{\ell - 1}^\top]\) with \(y = \beta_1 \sigma(g)\), so \[\label{eq:dgamma-split} \dot{\gamma} = \sum_\ell \bigl\langle G_\ell, \beta_1 \mathbb{E}[\sigma(g) \delta_\ell h_{\ell - 1}^\top]\bigr\rangle_F - \sum_\ell \bigl\langle G_\ell, \mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]\bigr\rangle_F.\tag{39}\] For the teacher term, Hermite-decompose \(\sigma(g) = h_\sigma g + \sigma_\perp(g)\) with \(\sigma_\perp(g) \doteq \sigma(g) - h_\sigma g\); orthogonality to \(g\) in \(L^2(\mathcal{N}(0, 1))\) is immediate from the definition of \(h_\sigma\), since \(\mathbb{E}[g \sigma_\perp(g)] = \mathbb{E}[g \sigma(g)] - h_\sigma \mathbb{E}[g^2] = h_\sigma - h_\sigma = 0\). The \(h_\sigma g\) component contributes exactly \(\beta_1 h_\sigma \sum_\ell \|G_\ell\|_F^2 = \beta_1 h_\sigma T(W)\), so it suffices to bound the \(\sigma_\perp\) contribution. We claim \[\label{eq:sigmaperp-bound} \bigl|\langle G_\ell, \mathbb{E}[\sigma_\perp(g) \delta_\ell h_{\ell - 1}^\top]\rangle_F\bigr| \le C M^2 \|G_\ell\|_F^2,\tag{40}\] uniformly in \(\ell\), which gives the claim. To see this, apply the outer-product decomposition 38 of 5: the linear component \(\alpha^{L - 1} B_\ell A_{\ell - 1}^\top g\) lies entirely in the \(k = 1\) Hermite mode, and \(\sigma_\perp\) is orthogonal to \(g\) by construction, so \(\mathbb{E}[\sigma_\perp(g) \cdot \alpha^{L - 1} B_\ell A_{\ell - 1}^\top g] = 0\) and \[\mathbb{E}[\sigma_\perp(g) \delta_\ell h_{\ell - 1}^\top] = \mathbb{E}[\sigma_\perp(g) \Psi_\ell(x)].\] Cauchy–Schwarz gives \(\|\mathbb{E}[\sigma_\perp(g) \Psi_\ell]\|_F \le \|\sigma_\perp\|_{L^2} \|\Psi_\ell\|_{L^2(F)} \le C \|\sigma_\perp\|_{L^2} M^2 \|B_\ell\|_2 \|A_{\ell - 1}\|_2\). Using \(\|G_\ell\|_F \asymp \|B_\ell\|_2 \|A_{\ell - 1}\|_2\) (the rank-\(1\) Frobenius identity applied to the leading tensor of 5, 38 ), pairing with \(G_\ell\) yields 40 . Therefore \[\sum_\ell \bigl\langle G_\ell, \beta_1 \mathbb{E}[\sigma(g) \delta_\ell h_{\ell - 1}^\top]\bigr\rangle_F = \beta_1 h_\sigma \sum_\ell \|G_\ell\|_F^2 \bigl(1 + O(M^2)\bigr) = \beta_1 h_\sigma T(W) \bigl(1 + O(M^2)\bigr),\] which together with the definition \(S(W) \doteq \sum_\ell \langle G_\ell, \mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]\rangle_F\) gives ?? . 0◻

10.2 Proof of the Self-Interaction Bound↩︎

By 5, on the bootstrap interval \(\|f\|_{L^2} \lesssim |\gamma(W)|\) and \(\|\delta_\ell h_{\ell - 1}^\top\|_{L^2(F)} \lesssim \|G_\ell\|_F\) with constants independent of \(\varepsilon\), \(L\), and \(r\). Apply Cauchy–Schwarz on the Frobenius pairing inside each expectation: \(|\mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]|_F \le \|f\|_{L^2} \|\delta_\ell h_{\ell - 1}^\top\|_{L^2(F)}\). This gives, for each \(\ell\), \[\bigl|\langle G_\ell, \mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]\rangle_F\bigr| \le \|G_\ell\|_F \cdot \bigl|\mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]\bigr|_F \lesssim \|G_\ell\|_F \cdot |\gamma(W)| \cdot \|G_\ell\|_F = |\gamma(W)| \|G_\ell\|_F^2.\] Summing over \(\ell \in \{1, \ldots, L\}\) gives \(|S(W)| \lesssim |\gamma(W)| T(W)\), which is the statement of 1. Substituting into ?? , the contribution of \(S(W)\) is bounded by \(C |\gamma| T\), while the leading term is \(\beta_1 h_\sigma T (1 + O(M^2))\); since \(|\gamma| \le m_0^r\) and \(M \le m_0\) on the bootstrap interval, the self-interaction is strictly lower-order than the positive teacher term, and for \(m_0\) small enough \[\dot{\gamma} \ge \tfrac12 \beta_1 h_\sigma T(W)\] throughout \([0, \tau_{m_0}]\). 0◻

10.3 Proof of 5↩︎

Before turning our attention to the upper and lower bounds, we verify that the non-bottleneck assumption of 2(i) is self-sustaining on the bootstrap interva, i.e.,the non-bottleneck operator-norm window is not an input hypothesis but a consequence of the small-signal regime.

6 closes the bootstrap: the non-bottleneck stack drifts by at most \(O(m_0^2)\) in Frobenius norm on the entire bootstrap interval, and the same perturbation estimate preserves the realized teacher-aligned chain bounds. Thus neither the operator-norm window nor the chain controls needed by the linear-path expansion are separate time-uniform hypotheses; both follow from the small-signal regime and their initialization event. In particular, the constants used in the upper and lower bounds below are uniform in \(\varepsilon\).

10.3.0.1 Upper bound on \(\tau_\star\) via signal energy.

By the sign-freedom in the teacher direction, replacing \(v_1 \to -v_1\) flips the sign of \(\gamma\) without changing \(f\) or \(\mathcal{L}\), so we may assume \(\gamma(0) > 0\). On the bootstrap interval \([0, \tau_{m_0}]\), combining 2 and 1 gives, using \(|\gamma| \le m_0^r\) and \(|S| \le C |\gamma| T\), the clean lower bound \[\label{eq:dgamma-T} \dot{\gamma} \ge c T(W), \qquad c = \tfrac12 \beta_1 h_\sigma,\tag{42}\] for \(m_0\) small enough; in particular \(\gamma\) is nondecreasing and, with \(\gamma(0) \ge c_\mathrm{nd}\varepsilon^r > 0\), stays positive throughout \([0, \tau_{m_0}]\). The linear-path tensor expansion of 3 gives, uniformly in \((L, r)\), \[\label{eq:Gell-lp} G_\ell = \alpha^{L - 1} B_\ell A_{\ell - 1}^\top + R_\ell, \qquad \|R_\ell\|_F \le C M^2 \|B_\ell\|_2 \|A_{\ell - 1}\|_2,\tag{43}\] with \(A_0 = v_1\), \(A_j = W_j A_{j - 1}\), \(B_L = 1\), \(B_\ell = W_{\ell + 1}^\top B_{\ell + 1}\); consequently \(\|G_\ell\|_F \asymp \|B_\ell\|_2 \|A_{\ell - 1}\|_2 \asymp \prod_{m \ne \ell} y_m\) with \(y_m\) the bottleneck linear-path gain in layer \(m\) (so \(y_m = \Theta(\|W_m\|_\mathrm{op})\) for \(m \le r\) and \(y_m = \Theta(1)\) for \(m > r\), with constants independent of \(\varepsilon\), \(L\), \(r\) by 2). Then \[\label{eq:T-AMGM} T(W) = \sum_{\ell = 1}^{L} \|G_\ell\|_F^2 \asymp \sum_{\ell = 1}^{r} \prod_{m \ne \ell, m \le r} y_m^2.\tag{44}\] To justify the restriction to \(\ell \le r\): by the tensor expansion 43 , for any \(\ell \in \{1, \ldots, L\}\), \[\|G_\ell\|_F^2 \asymp \Bigl(\prod_{m \ne \ell} y_m\Bigr)^2 = \Bigl(\prod_{m \ne \ell,\,m \le r} y_m\Bigr)^2 \cdot \Bigl(\prod_{m \ne \ell,\,m > r} y_m\Bigr)^2.\] For a bottleneck layer \(\ell \le r\), exactly \(r - 1\) bottleneck factors remain in the first product (each \(\Theta(\varepsilon)\)) and all \(L - r\) non-bottleneck factors are \(\Theta(1)\) by 2(i), giving \(\|G_\ell\|_F^2 \asymp M^{2(r - 1)}\). For a non-bottleneck layer \(\ell > r\), all \(r\) bottleneck factors appear in the first product and \(L - r - 1\) non-bottleneck factors in the second, giving \(\|G_\ell\|_F^2 \asymp M^{2r}\): strictly smaller by a factor \(M^2 \le m_0^2 \ll 1\). Summing over the \(L - r \le L\) non-bottleneck layers contributes at most \(L \cdot M^{2r}\), which is negligible compared to the \(r\) bottleneck contributions \(r \cdot M^{2(r - 1)}\) once \(m_0^2 \le 1/L\); this is absorbed into the choice of \(m_0\) in the bootstrap. Hence \(T(W) \asymp \sum_{\ell \le r} \prod_{m \ne \ell, m \le r} y_m^2\), with constants independent of \(\varepsilon\).

AM-GM on the \(r\) bottleneck terms gives \[\sum_{\ell = 1}^{r} \prod_{m \ne \ell} y_m^2 \ge r \Bigl(\prod_{\ell = 1}^{r} \prod_{m \ne \ell} y_m^2\Bigr)^{1/r} = r \Bigl(\prod_{m = 1}^{r} y_m^2\Bigr)^{(r - 1)/r}.\] Since \(\gamma(W) \asymp \prod_{m = 1}^r y_m\) by the Hermite identity (using initial nondegeneracy to bound \(\gamma\) from below and the sign normalization \(\gamma \ge 0\) established above), this gives \(T(W) \ge c' \gamma^{2 (r - 1)/r} = c' \gamma^{2 - 2/r}\). Combining with 42 , the whole upper-bound argument collapses onto a single scalar ODE for the teacher-signal observable, \[\label{eq:scalar-ode-upper} \dot{\gamma}(t) \ge c T(W(t)) \ge c c' \gamma(t)^{2 - 2/r} \qquad \text{on } [0, \tau_{m_0}],\tag{45}\] and integrating this hitting-time comparison from \(\gamma(0) = \Theta(\varepsilon^r)\) to a fixed \(\gamma_\star > 0\) yields \[\tau_\star \le \int_{\gamma_0}^{\gamma_\star} \frac{d\gamma}{c c' \gamma^{2 - 2/r}} = \frac{r}{c c'(r - 2)} \gamma_0^{-(r - 2)/r} (1 - o(1)) = C_+ \varepsilon^{-(r - 2)} (1 - o(1)).\] The integrand \(\gamma^{-(2 - 2/r)}\) has a singularity at \(\gamma = 0\) whenever \(r \ge 3\), but the integration interval \([\gamma_0, \gamma_\star]\) is bounded away from this singularity by \(\gamma_0 = \Theta(\varepsilon^r) > 0\); the integral is therefore finite and the antiderivative \(\frac{r}{2 - r} \gamma^{(2 - r)/r}\) is evaluated at well-defined endpoints. The blow-up of \(\gamma_0^{-(r - 2)/r} = \Theta(\varepsilon^{-(r - 2)})\) as \(\varepsilon \to 0\) is not integrand singularity inside the interval but the polynomial divergence as the lower limit approaches the singular point; this is the mechanism by which \(\tau_\star\) diverges polynomially with \(\varepsilon\).

10.3.0.2 Lower bound on \(\tau_\star\) via bottleneck operator growth.

We estimate \(\|\dot{W}_\ell\|_F \le \|\nabla_{W_\ell} \mathcal{L}\|_F\) in terms of \(M\) alone. From the squared-loss gradient \[\nabla_{W_\ell} \mathcal{L} = \beta_1 \mathbb{E}[\sigma(g) \delta_\ell h_{\ell - 1}^\top]\;-\;\mathbb{E}[f \delta_\ell h_{\ell - 1}^\top],\] the two terms are controlled by the linear-path expansion 43 plus \(\|f\|_{L^2} \lesssim M^r\) (filtered composition applied to \(f\), with \(r\) bottleneck layers and non-bottleneck \(\Theta(1)\)). The teacher term is \(\|\beta_1 \mathbb{E}[\sigma(g) \delta_\ell h_{\ell - 1}^\top]\|_F \lesssim \|G_\ell\|_F \lesssim M^{r - 1}\). The student term is \(\|\mathbb{E}[f \delta_\ell h_{\ell - 1}^\top]\|_F \le \|f\|_{L^2} \|\delta_\ell h_{\ell - 1}^\top\|_{L^2(F)} \lesssim M^r \cdot M^{r - 1} = M^{2 r - 1}\). Together, \[\label{eq:Wdot-bound} \|\dot{W}_\ell\|_F \lesssim M^{r - 1} + M^{2 r - 1} \lesssim M^{r - 1}\tag{46}\] while \(M \le m_0 \ll 1\). Since \(M(t) = \max_\ell \|W_\ell(t)\|_\mathrm{op}\) is the pointwise maximum of finitely many locally Lipschitz functions, it is itself locally Lipschitz and admits an upper Dini derivative \(D^+ M(t) \doteq \limsup_{h \downarrow 0} h^{-1}[M(t + h) - M(t)]\) satisfying \[\label{eq:U-dini} D^+ M(t) \le \max_{\ell \le r} \|\dot{W}_\ell(t)\|_\mathrm{op} \le \max_{\ell \le r} \|\dot{W}_\ell(t)\|_F \le C M(t)^{r - 1}.\tag{47}\] By the a Dini-derivative ODE comparison: let \(V(t)\) solve \(\dot{V} = C V^{r - 1}\) with \(V(0) = M(0)\); then any absolutely continuous \(M\) with \(D^+ M(t) \le C M(t)^{r - 1}\) and \(M(0) = V(0)\) satisfies \(M(t) \le V(t)\) for all \(t \ge 0\) on which \(V\) exists (a consequence of the Petrov–Grönwall argument for upper Dini derivatives, replacing differentiability of \(M\)). The separated integral gives \(V(t) = [V(0)^{-(r - 2)} - C(r - 2) t]^{-1/(r - 2)}\), so the first time \(V\) reaches \(m_0\) satisfies \[\tau_{V, m_0} = \frac{1}{C(r - 2)} \bigl[V(0)^{-(r - 2)} - m_0^{-(r - 2)}\bigr] = \frac{1}{C(r - 2)} \bigl[\Theta(\varepsilon)^{-(r - 2)} - m_0^{-(r - 2)}\bigr] \ge C_- \varepsilon^{-(r - 2)},\] and \(M(t) \le V(t)\) forces \(\tau_{m_0} \doteq \inf\{t : M(t) \ge m_0\} \ge \tau_{V, m_0}\).

We now fix the loss-threshold constants in the order the quantifiers require. Fix any escape threshold \(\mathcal{L}_\star < \mathcal{L}(0)\). The filtered-composition expansion for the output gives the pointwise bound \(\|f(\cdot\,;W(t))\|_{L^2} \le C_0 M(t)^r\) uniformly on the bootstrap interval, hence \(\mathcal{L}(W(t)) \ge \mathcal{L}(0) - C_1 M(t)^r\) for a constant \(C_1 = C_1(\sigma, L, c_\ast, C_0)\) independent of \(\varepsilon\) and \(m_0\). Choose \(m_0 \le m_0^\ast\) small enough that \(C_1 m_0^r < \mathcal{L}(0) - \mathcal{L}_\star\); this is a condition on \(m_0\) alone, since \(C_1\) does not depend on \(m_0\). Then \[\label{eq:loss-threshold} M(t) \le m_0 \quad\Longrightarrow\quad \mathcal{L}(W(t)) \ge \mathcal{L}(0) - C_1 m_0^r > \mathcal{L}_\star,\tag{48}\] so the first time \(\mathcal{L}\) drops to \(\mathcal{L}_\star\) satisfies \(\tau_\star \ge \tau_{m_0} \ge C_- \varepsilon^{-(r - 2)}\). Combined with the upper bound, \(\tau_\star = \Theta(\varepsilon^{-(r - 2)})\). 0◻

10.4 The Shallow-Bottleneck Corners \(r \in \{1, 2\}\)↩︎

The machinery of 10.3 was presented for \(r \ge 3\) because both polynomial integrals \(\int \gamma^{-(2 - 2/r)} d\gamma\) and \(\int M^{-(r - 1)} dM\) diverge polynomially in that range. The proof applies exactly the same at \(r \in \{1, 2\}\) up to the single step of integrating the scalar comparison ODEs; only the antiderivative changes.

10.4.0.1 Concentration at He-normal init.

Fix \(\delta>0\). At He-normal init the non-bottleneck \(W_\ell \in \mathbb{R}^{N \times N}\) have i.i.d.Gaussian entries with variance \(2/N\). Standard Gaussian operator-norm concentration gives \(\|W_\ell\|_\mathrm{op} \in [c_\ast^{-1}, c_\ast]\) simultaneously for \(\ell>r\), and \(M(0)=\max_{\ell\le r}\|W_\ell\|_\mathrm{op}=\Theta(\varepsilon)\) after the bottleneck rescaling, on an event of probability at least \(1-\delta/2\) (for \(N\) in the usual concentration regime). No lower bound on \(\sigma_\mathrm{min}(W_\ell)\) is used or asserted.

For the lower controls, apply the conditional-Gaussian anti-concentration argument of 5 only to the realized teacher-aligned forward and backward chains. With the remaining \(\delta/2\) failure budget, this gives the two-sided chain bounds in 2(i). Intersecting the two events yields a joint event of probability at least \(1-\delta\) on which all linear-path constants are uniform in \(\varepsilon\). This controls the specific forward and backward directions needed here, rather than the minimum singular value of every matrix.

11 Multi-Mode Geometry and Homotopy Proofs↩︎

This appendix develops the multi-mode geometry deferred from 5. 11.1 collects the main statements: the per-stage rescaled-profile reduction, the structural instability of the block-mean saddle, and the escape-time homotopy identity. [app:offblock,app:layer1cross] derive the Duhamel equations for the off-block \(W_2\) coupling and the layer-\(1\) cross-block rotations. 11.6 derives the stage-\(k\) scalar fixed-point equation and verifies the \(\varepsilon^{L - 2}\) escape-rate scaling of 49 . 11.7 proves the escape-time homotopy identity via Liouville duality.

11.1 Main Statements: Modewise Profiles, Structural Instability, Homotopy Identity↩︎

11.1.0.1 Modewise escape and rescaled scalar profiles.

Under the block-aligned stage-\(k\) ansatz (derived in 11.6), the mode-\(k\) breaking amplitude \(c_k\) obeys \[\label{eq:stagek-ode} \frac{d c_k}{d t} = K_L^{(k, \sigma)} \bigl(\rho_\mathrm{op}(k)^2 + c_k^2\bigr)^{(L - 1)/2}, \qquad K_L^{(k, \sigma)} = \frac{\beta_k h_\sigma T_L(\rho_\mathrm{op}(k))}{\sqrt N},\tag{49}\] with chain moment \(T_L(\rho) = \mathbb{E}_g[\prod_{j = 1}^{L - 1} \sigma'(\rho \cdot \mathrm{chain}_j(\rho g))]\). Rescaling by the saddle scale, \(u_k \doteq c_k / \rho_\mathrm{op}(k)\) and \(\Gamma_k \doteq K_L^{(k, \sigma)} \rho_\mathrm{op}(k)^{L - 2}\), yields \(\dot{u}_k = \Gamma_k (1 + u_k^2)^{(L - 1)/2}\), so for any normalized threshold \(\theta > 0\) the escape time is \[\label{eq:stagek-universal} t_k(\theta) = \Gamma_k^{-1} \mathcal{I}_L(\theta), \qquad \mathcal{I}_L(\theta) \doteq \int_0^\theta (1 + s^2)^{-(L - 1)/2} d s,\tag{50}\] with \(\mathcal{I}_L\) depending only on depth; all mode- and configuration-dependence compresses into the single scalar rate \(\Gamma_k\). Because \(\mathcal{I}_L\) is universal, modewise ranking is exact within the reduced model: \(t_j(\theta) < t_k(\theta)\) iff \(\Gamma_j > \Gamma_k\). In the small-\(\varepsilon\) regime, the bottleneck-shell evaluation of 4 gives \(\rho_\mathrm{op}(k)^{L - 2} \asymp \varepsilon^{r_k - 2}\), where \(r_k\) is the critical depth of mode \(k\), i.e., the number of layers along the mode-\(k\) saddle cone at scale \(\varepsilon\) at the start of stage \(k\), the multi-mode generalization of the \(r\) in \(\tau_\star = \Theta(\varepsilon^{-(r - 2)})\) of 3 — so \(t_k(\theta) \asymp C_L^{(\sigma)}(\theta) / [\beta_k \varepsilon^{r_k - 2}]\): teacher weight and critical depth trade off linearly in log-time, and a mode of smaller \(\beta_k\) can escape ahead of a larger one if its critical depth is smaller by at least one layer.

11.1.0.2 Structural instability of the block-mean saddle.

The block-aligned ansatz is exactly flow-invariant from a permutation-symmetric initialization, but the manifold is not attracting at generic Gaussian init. The dominant stage-\(1\) channel in multi-mode simulations is not the off-block \(W_2\) coupling \(C_{kk'}\) but layer-\(1\) cross-block rotations \(\eta_{k, m}\). Augmenting the block-mean ansatz by allowing \(W_1[j, :] = X_{1, k} v_k + \sum_{m \ne k} \eta_{k, m} v_m\) for \(j \in B_k\) and computing the stage-\(1\) Jacobian at the balanced saddle, the reduced-variable equations (derived from the Hermite-moment bookkeeping of [app:offblock,app:layer1cross] — the off-block \(W_2\) and layer-\(1\) cross-block Duhamel derivations, respectively) take the block form \[\label{eq:stage1-aug-linearization} \begin{pmatrix} \dot{x}^{(1)} \\ \dot{\eta}^{(1)} \end{pmatrix} = \varepsilon^{L - 2} \begin{pmatrix} -A_\varepsilon & B_\varepsilon \\ C_\varepsilon & -D_\varepsilon \end{pmatrix} \begin{pmatrix} x^{(1)} \\ \eta^{(1)} \end{pmatrix} + R_\varepsilon,\tag{51}\] with \(A_\varepsilon, D_\varepsilon\) diagonal and positive (the on-diagonal decay rates of each decoupled channel) and \(B_\varepsilon, C_\varepsilon\) entrywise nonnegative. The sign structure of \(B_\varepsilon, C_\varepsilon\) follows directly from the Hermite-moment expansion of [app:offblock,app:layer1cross]: each entry of \(B_\varepsilon\) (resp.\(C_\varepsilon\)) is a leading-order Gaussian expectation of the form \(c \cdot \mathbb{E}_{z}[\sigma'(U z)^2 \phi(z)]\) with \(c > 0\) collecting positive scaling factors (the downstream linear-chain product and the block-size normalization) and \(\phi(z) \ge 0\) a product of squared pre-activations arising from the cross-block coupling; positivity of \(\sigma'(Uz)^2\) and of \(\phi(z)\) under the Gaussian measure forces the entry nonnegative. Structurally: a positive \(\eta\) perturbation feeds additional positive pre-activation mass into the \(x\)-channel’s drive, and symmetrically for \(x \to \eta\); the mutual feedback is manifestly same-sign. By 6 below, whenever the mixed loop gain \(\rho(D_\varepsilon^{-1} C_\varepsilon A_\varepsilon^{-1} B_\varepsilon)\) exceeds \(1\), the decoupled block-mean saddle has a real positive eigenvalue with eigenvector mixing the \(x\) and \(\eta\) blocks, despite all diagonal rates being stabilizing, so it is not structurally stable under layer-\(1\) cross-block rotations. Beyond the block-mean truncation, the stage-\(1\) mean equations couple to arbitrarily high centered row moments \(Q_k^{(p)}\): each Hermite expansion of the within-block row distribution feeds into the mean-channel drift through a moment of order \(q\) in \(\sigma\), so truncating the hierarchy at row-moment order \(p\) leaves residuals from orders \(> p\) that are \(O(\varepsilon^{L - 2})\) at the plateau scale: the same scale as the mean-channel drift itself. No finite-moment truncation in the row-moment hierarchy closes the stage-\(1\) mean equations without uncontrolled error at this scale; fully resolving the coupling therefore requires the full row distribution (or a separate argument bounding the high-moment contribution). Under symmetric population gradient descent the row-variance channel is quiescent by symmetry; it reopens under random or stochastic initialization.

11.1.0.3 The escape-time homotopy identity.

Given that no finite block-mean closure matches the empirical escape time, what we can offer is a principled attribution of the escape-time shift across closure levels. The mechanism is an adjoint-homotopy identity derived from Liouville duality; the proof is in 11.7.

We view 5 as a theoretical attribution tool, not a numerical predictor — a direct forward solve at \(\nu = 1\) already returns \(T(1)\). The identity adds an explicit decomposition of \(T(1) - T(0)\) into per-channel first-variation contributions \(A(\nu)\), exposing which channels (off-block \(W_2\) coupling, cross-block rotations, per-layer amplitudes) drive the shift, and where along the homotopy they cancel. Symmetric quadrature evaluates ?? to high order. 5 illustrates the decomposition on a three-mode tanh teacher.

11.2 Structural Instability of the Block-Mean Saddle: Proof↩︎

The following proposition supplies an analytic proof that the block-mean Jacobian 51 has a real positive eigenvalue whenever the mixed loop gain exceeds unity. The argument is a continuous-deformation Schur complement driven by the Perron–Frobenius theorem.

6 applies to 51 once \(A_\varepsilon, B_\varepsilon, C_\varepsilon, D_\varepsilon\) are identified as the Jacobian blocks extracted from the Hermite-moment expansion of [app:offblock,app:stagek,app:layer1cross]. The required hypotheses (\(A_\varepsilon, D_\varepsilon\) diagonal positive, \(B_\varepsilon, C_\varepsilon\) entrywise nonnegative) are verified structurally from the Hermite-moment definitions in those appendices: diagonal positivity is the decay-rate sign of each decoupled block; nonnegativity of \(B_\varepsilon, C_\varepsilon\) follows from the closed forms 58 and 60 , each of whose coupling entries is \((\mathrm{positive scalar}) \cdot \mathbb{E}[\sigma'(U z)^2 \cdot (\mathrm{nonnegative Hermite polynomial})]\). The residual \(R_\varepsilon\) in 51 is \(O(\varepsilon^{L - 1})\) and strictly higher order in the small-\(\varepsilon\) scaling; perturbing \(J_\varepsilon\) by an \(O(\varepsilon)\)-small term preserves the simple positive eigenvalue guaranteed by the proposition, so the linear-instability conclusion transfers to the full nonlinear flow on a neighborhood of the saddle by the stable-manifold theorem.

11.3 A No-Go Theorem: Strict No-Closure of the Row-Moment Hierarchy↩︎

The two-block form of 6 presumes that the Jacobian has already been reduced to a finite \(2 \times 2\) block structure \(J = \bigl[\begin{smallmatrix} -A & B \\ C & -D \end{smallmatrix}\bigr]\). In the underlying PDE dynamics, however, the coupling blocks \(B, C\) are obtained by truncating an infinite hierarchy of row moments: the within-block row distribution at each layer admits a Hermite expansion \(p_\ell(w) = \sum_{p \ge 0} Q_\ell^{(p)} He_p(w)\), and the Jacobian acts on the graded Hilbert space \(\mathcal{H} = \bigoplus_{p \ge 0} \mathcal{H}_p\), where \(\mathcal{H}_p\) is the block of degree-\(p\) coefficients. The finite-dimensional blocks used in the main text correspond to a principal-submatrix truncation to degrees \(p \le K\) for some fixed \(K\). The natural question is whether this truncation is faithful: that is, does the finite Perron root \(\lambda_K\) agree with, or converge finitely to, the infinite-dimensional Perron root \(\lambda_\infty\)? 8 answers in the negative.

We note that a quantitative rate of the gap \(\lambda_\infty - \lambda_K\) is to our knowledge, open and is left to future work. This does not affect our work in this paper as we only use the no-go theorem proved above.

11.4 Derivation of the Off-Block \(W_2\) Duhamel Equation↩︎

Throughout, the teacher is \(K\)-mode with directions \(\{v_k\}_{k = 1}^K\) and coefficients \(\{\beta_k\}\); the \(K\)-block ansatz partitions the \(N\) hidden neurons of each layer into blocks \(\{B_k\}\) of block size \(N_B \doteq N/K\) (distinct from the bottleneck operator scale \(M\) of 5, which does not enter this appendix); within each block the rows of \(W_1\) point along \(v_k\) and layers \(\ell \ge 2\) carry a scalar \(X_{\ell, k}\). The off-block perturbation is the rank-\(1\) entry \(W_2[i \in B_k, j \in B_{k'}] = C_{k k'} / N\) with \(k \ne k'\). Our goal is the scalar evolution of \(C_{k k'}(t)\) under gradient flow.

11.4.0.1 Why \(W_2\).

We work with the off-block entries of \(W_2\) because every deeper layer \(W_\ell\) (\(\ell \ge 3\)) admits the same type of rank-\(1\) off-block perturbation, but with an additional \(\rho_\mathrm{op}^{\ell - 2}\) suppression from passing through \(\ell - 2\) layers of the balanced chain before the cross-block leak occurs: at layer \(\ell\), the upstream block-\(k'\) pre-activation has already been propagated through \(\ell - 2\) balanced-chain factors of size \(\rho_\mathrm{op}(k') \sim \varepsilon\) each, so the rank-\(1\) coupling coefficient enters the Jacobian blocks \(B, C\) of 51 with an extra factor of \(\rho_\mathrm{op}^{\ell - 2}\) relative to the \(\ell = 2\) case. Layer \(\ell = 2\) is therefore the shallowest and dominant contribution to the off-block channel, and deeper layers contribute strictly subleading corrections that are absorbed into the remainder \(\mathcal{R}_{k k'}\) of 58 . The layer-\(1\) channel is structurally different — a row rotation in \(v\)-space, not an off-block weight entry — and is treated separately in 11.5.

Fix \(k \ne k'\) and write \(C \doteq C_{k k'}\). At layer \(\ell = 3\), block \(k\), the added pre-activation is \[\Delta h_i^{(3)}(x) = \frac{C}{N} \sum_{j \in B_{k'}} \sigma(W_1[j, :] x).\] Every row \(j \in B_{k'}\) equals \(X_{1, k'} v_{k'}^\top\), so \(W_1[j, :] x = X_{1, k'} z_{k'}\) with \(z_{k'} \doteq v_{k'}^\top x\) block-dependent only. Writing \(U_{k'} \doteq X_{1, k'}\), the block sum collapses to \(N_B\) identical copies and \(\Delta h_i^{(3)}(x) = (C / K) \sigma(U_{k'} z_{k'})\); the block-size factor \(N_B = N/K\) cancels against the \(1/N\) normalization, leaving an \(O(1/K)\) per-entry contribution. Expanding \(\sigma(u) = \alpha u + a_q u^q + O(u^{q + 1})\) about \(u = 0\) splits this into a linear piece and a nonlinear defect.

To propagate this perturbation to the output we pass through layers \(\ell = 3, \ldots, L\) of the target block \(k\). Linearizing around the unperturbed trajectory, each layer contributes a multiplicative factor \(X_{\ell, k}\), defining the downstream product \(D_k(X) \doteq \prod_{\ell = 3}^L X_{\ell, k}\). The contribution to the output projected onto block \(k\) is \[\Delta f_k(x) = \frac{C}{K} D_k(X) \bigl[\alpha U_{k'} z_{k'} + a_q U_{k'}^q z_{k'}^q + \cdots\bigr].\]

Projecting the gradient and summing over the rank-\(1\) pair \((i, j) \in B_k \times B_{k'}\), then evaluating the Hermite moments of \(z_{k'} \sigma(U_{k'} z_{k'})\) and \(z_{k'}^q \sigma(U_{k'} z_{k'})\) under \(z_{k'} \sim \mathcal{N}(0, 1)\), yields \[\label{eq:offblock-duhamel} \dot{C}_{k k'} = -\alpha_{k k'} D_k(X) U_{k'} C_{k k'} + \beta_{k k'} D_k(X) U_{k'}^q + \mathcal{R}_{k k'}(X, C),\tag{58}\] with Hermite coefficients \[\alpha_{k k'} = \frac{\alpha^2}{K} \mathbb{E}_{z_{k'}}[\sigma'(U_{k'} z_{k'})^2], \qquad \beta_{k k'} = \frac{a_q \alpha}{K} \mathbb{E}_{z_{k'}}[z_{k'}^q \sigma'(U_{k'} z_{k'})],\] where \(z_{k'} \sim \mathcal{N}(0,1)\), the factor \(\alpha^2/K\) (resp.\(a_q \alpha / K\)) collects the linear-branch contributions from the target-block downstream chain and the \(1/K\) block-size normalization, and the expectations reduce to standard Hermite moments of \(\sigma'\). The remainder is \(\mathcal{R}_{k k'} = O(D_k U_{k'}^2 C) + O(D_k U_{k'}^{q + 1}) + O(C^2)\).

Setting \(\dot{C}_{k k'} = 0\) in the linearized equation gives the quasi-steady balance \(C_{k k'}^\star = (\beta_{k k'}/\alpha_{k k'}) U_{k'}^{q - 1}\); the downstream product \(D_k\) cancels in the ratio, so \(\Phi_{k k'}\) depends on the source block \(k'\) alone, not on the target chain. Variation of constants on \(\dot{C} = -a(t) C + b(t)\) from \(C_{k k'}(0) = 0\), with \(a(s) = \alpha_{k k'} D_k(X(s)) U_{k'}(s)\) and \(b(s) = \beta_{k k'} D_k(X(s)) U_{k'}(s)^q\), gives the Duhamel form \[\label{eq:duhamel} C_{k k'}(t) = e^{-\int_0^t a(s) d s} \int_0^t e^{\int_0^s a(u) d u} b(s) d s + (\text{higher-order}).\tag{59}\] On a pre-escape plateau with \(U_{k'}(s) \sim \varepsilon\) and \(D_k(X(s)) \sim \varepsilon^{L - 2}\), the constant-coefficient approximations \(a \sim \alpha_{k k'} \varepsilon^{L - 1}\) and \(b \sim \beta_{k k'} \varepsilon^{L - 2 + q}\) reduce 59 to the elementary closed form \(C_{k k'}(t) = (\beta_{k k'}/\alpha_{k k'}) U_{k'}^{q - 1}(1 - e^{-a \varepsilon^{L - 1} t})\), agreeing with the quasi-steady graph at long times.

11.5 Derivation of the Layer-\(1\) Cross-Block Duhamel Equation↩︎

The layer-\(1\) cross-block perturbation \(\eta_{k, m}\) replaces \(W_1[j, :] = X_{1, k} v_k^\top\) (\(j \in B_k\)) with \(W_1[j, :] = X_{1, k} v_k^\top + \eta_{k, m} v_m^\top\) for \(m \ne k\). The derivation of \(\dot{\eta}_{k, m}\) is structurally parallel to 11.4: linearizing \(\sigma(z_1[j]) = \sigma(U_k z_k) + \eta z_m \sigma'(U_k z_k) + O(\eta^2)\) (with \(U_k \doteq X_{1, k}\) and \(z_k \doteq v_k^\top x\), \(z_m \doteq v_m^\top x\) independent under the isotropic input law), propagating through the balanced downstream chain \(\bar D_k(X) \doteq \alpha^{L - 2} \prod_{\ell = 2}^{L} X_{\ell, k}\) (starting at layer \(2\) since the perturbation enters at layer \(1\), unlike \(D_k\) of 11.4 which starts at layer \(3\)), and projecting \(-\nabla_\eta \mathcal{L}\) onto the Gaussian input measure (the cross-block pairing \(\mathbb{E}[z_m z_{k''}] = \delta_{m k''}\) selects a single block-\(m\) source term, and the self-interaction contributes a decay) yields \[\label{eq:layer1cross-duhamel} \dot{\eta}_{k, m} = -\tilde{\alpha}_{k, m}(X) \eta_{k, m} + \tilde{\beta}_{k, m}(X) + \tilde{\mathcal{R}}_{k, m}(X, \eta),\tag{60}\] with \(\tilde{\alpha}_{k, m}(X) = \bar D_k(X)^2 \cdot \mathbb{E}_z[\sigma'(U_k z)^2] + O(\varepsilon)\) and \(\tilde{\beta}_{k, m}(X) = \bar D_k(X) \bar D_m(X) \cdot \mu_{k, m}(U) \cdot (\text{positive scalar})\), where \(\mu_{k, m}(U) \doteq \mathbb{E}_{z \sim \mathcal{N}(0, 1)}[z^2 \sigma'(U_k z) \sigma(U_m z)] + U_m\, \mathbb{E}_z[\sigma'(U_k z) \sigma'(U_m z)]\) is the leading-order cross-block Hermite moment extracted from the Stein-pairing expansion. The first factor \(\mathbb{E}_z[\sigma'(U_k z)^2]\) is nonnegative as a square. The second factor \(\mu_{k, m}(U) = \mathbb{E}_z[z^2 \sigma'(U_k z) \sigma(U_m z)] + U_m\, \mathbb{E}_z[\sigma'(U_k z) \sigma'(U_m z)]\) is checked to be positive on the pre-escape plateau (\(U_k, U_m > 0\)), for each of the activations used in the paper. For odd \(\sigma\) (tanh, erf, sin; Class B) the first summand \(\mathbb{E}_z[z^2 \sigma'(U_k z) \sigma(U_m z)]\) vanishes by parity: \(z^2\) is even, \(\sigma'(U_k z)\) is even (derivative of odd), \(\sigma(U_m z)\) is odd, so the integrand is odd and the Gaussian integral is exactly zero. Positivity of \(\mu_{k, m}\) is then inherited from the second summand alone: the integrand \(\sigma'(U_k z) \sigma'(U_m z)\) is a product of two even functions, positive at \(z = 0\) (where both derivatives equal \(\alpha > 0\)), and the Gaussian expectation is strictly positive by continuity for small \(U_k, U_m\) (explicitly, \(\mathbb{E}_z[\sigma'(U_k z) \sigma'(U_m z)] = \alpha^2 + O(U_k^2 + U_m^2)\)), so \(\mu_{k, m}(U) = U_m \alpha^2 (1 + O(\varepsilon^2)) > 0\) on the plateau since \(U_m \sim \varepsilon > 0\). For non-odd \(\sigma\) (GELU, Swish, softplus; Classes C/D) both summands contribute nontrivially; direct integration against the Hermite basis at \(U_k, U_m = \varepsilon\) gives a positive leading coefficient. This is a finite calculation, not a monotonicity hypothesis on \(\sigma\), and in particular does not require \(\sigma\) to be monotone (sin is covered by the odd-parity argument above). Comparing 60 with 58 , both are of the form \(\dot{\theta} = -a(X) \theta + b(X) + \mathcal{R}\) with \(a(X) \ge 0\) and \(b(X) \ge 0\), so the Jacobian block \(B_\varepsilon, C_\varepsilon\) of 51 inherits entrywise nonnegativity and \(A_\varepsilon, D_\varepsilon\) are diagonal positive. On the pre-escape plateau \(\bar D_k \asymp \varepsilon^{L - 2}\), so every entry of \(B_\varepsilon, C_\varepsilon\) is \(\Theta(\varepsilon^{L - 2})\) and the Perron loop-gain matrix \(\mathcal{M}_\varepsilon \doteq D_\varepsilon^{-1} C_\varepsilon A_\varepsilon^{-1} B_\varepsilon\) (notation of 6) is \(O(1)\) independent of \(\varepsilon\) at leading order, verifying the hypotheses of 6.

11.6 Stage-\(k\) Saddle Fixed-Point Equations↩︎

After stages \(1, \ldots, k - 1\) have escaped, the first \(k - 1\) modes sit on a post-escape manifold on which the scalars \(U_j = X_{1, j}\) (for \(j < k\)) are dependent on the teacher coefficients via the deep-linear balance law \(U_j^{L - 1} = \beta_j / K^{(\sigma)} + O(\varepsilon^{q - 1})\), and the downstream chain along each of these modes has relaxed to the balanced ray. The \(k\)th active mode carries \(U_k = U \sim \varepsilon\) that has not yet escaped, and the downstream chain along this mode sits near the small-balanced cone. Modes \(j > k\) are also at \(O(\varepsilon)\) but decouple from the \(k\)th stage to leading order through the quasi-steady off-block coupling of 58 . The effective gradient flow for \(U_k\) therefore decomposes as \(\dot{U}_k = -\partial_{U_k} \mathcal{L}_\mathrm{eff}(U_k; U_{1:k - 1})\).

Expand \(\mathcal{L}_\mathrm{eff}\) in powers of \(U_k\) at fixed upstream \(U_{1:k - 1}\): the three leading contributions are a residual forcing \(F_k(U_{1:k - 1})\) from the un-fitted modes \(\ge k\) (gradient of the squared loss at \(U_k = 0\)), a quadratic Hessian coupling with coefficient \(H_k(U_{1:k - 1})\), and the deep-linear self-drive of the \(k\)th mode’s own imbalance. Collecting higher-order terms into \(R_k(U)\), \[\label{eq:stagek-drift} \dot{U}_k = F_k(U_{1:k - 1}) - H_k(U_{1:k - 1}) U_k + K^{(\sigma)} U_k^{L - 1} + R_k(U),\tag{61}\] with \(R_k(U) = O(U_k^{L + q - 2}) + O(U_k^2 \max_{j < k} U_j) + O(U_k^3)\).

11.6.0.1 Scaling chain.

Abbreviate \(\rho_k \doteq \rho_\mathrm{op}(k)\). We use the following three scaling relations: \[\label{eq:stagek-scaling} (\mathrm{S1})\; \rho_k^{L - 2} \asymp \varepsilon^{r_k - 2}, \qquad (\mathrm{S2})\; U_k \sim \rho_k, \qquad (\mathrm{S3})\; F_k, H_k \text{ scale as powers of } \rho_k,\tag{62}\] where (S1) is the bottleneck-shell evaluation (4) with \(r_k\) the critical depth of mode \(k\), (S2) is the normalized-profile condition \(U_k = \rho_k u_k\) with \(u_k \in (0, \theta)\) (\(u_k = O(1)\) on the plateau), and (S3) is established next. All exponents below are expressed in \(\rho_k\) and converted to \(\varepsilon\) via (S1) only at the final step.

Derivation of (S3). On the stage-\(k\) block-aligned ansatz, the derivative of the network output in the first-layer scale propagates through the \(k\)th mode’s chain linearly at \(U_k = 0\): \(\partial_{U_k} f|_{U_k = 0}\) equals \(\alpha^{L - 1} \cdot (\text{block-size factor}) \cdot (v_k^\top x) \cdot \prod_{\ell = 2}^{L} X_{\ell, k}\), each of the \(L - 1\) downstream layers contributing a factor of scale \(\rho_k\), so \[\partial_{U_k} f\big|_{U_k = 0} = \Theta(\rho_k^{L - 1}) \cdot (v_k^\top x).\] The residual forcing is the Gaussian expectation of this feature against the un-fitted teacher residual \(\sum_{j \ge k} \beta_j \sigma(v_j^\top x)\); orthogonality \(v_k \perp v_j\) for \(j > k\) selects \(j = k\) alone, and Stein’s identity gives \[\label{eq:Fk-scaling} F_k(U_{1:k - 1}) = c_k \beta_k \rho_k^{L - 1} \bigl(1 + O(\varepsilon^{q - 1})\bigr), \qquad c_k > 0\tag{63}\] (the correction term absorbs \(O(\varepsilon^{q - 1})\) contributions from the fitted residual \(U_j^{L - 1} - \beta_j/K^{(\sigma)}\)). The Hessian coefficient is the second variation, \[\label{eq:Hk-scaling} H_k(U_{1:k - 1}) = \mathbb{E}[(\partial_{U_k} f|_{0})^2] + O(\text{residual} \cdot \partial^2_{U_k} f|_{0}) = O(\rho_k^{2(L - 1)}),\tag{64}\] the dominant contribution being the square of the same feature. Both \(F_k\) and \(H_k\) are independent of \(U_k\).

11.6.0.2 Ordering of the three terms on the plateau.

Using (S2) to put \(U_k \sim \rho_k\) and (S3) to put \(F_k = \Theta(\rho_k^{L - 1})\), \(H_k = O(\rho_k^{2(L - 1)})\), each of the three terms in 61 is a power of \(\rho_k\):

The drag is higher order by a factor of \(\rho_k^L\) relative to both the forcing and the self-drive, i.e.\(H_k U_k = O(\rho_k^{2L - 1}) = O(\rho_k^L) \cdot \rho_k^{L - 1}\); since \(\rho_k \to 0\) as \(\varepsilon \to 0\) by (S1), the drag is asymptotically negligible relative to the other two, and \(F_k + K^{(\sigma)} U_k^{L - 1}\) is the effective right-hand side. The forcing \(F_k > 0\) has no dependence on \(U_k\) and seeds escape from any \(U_k \in [0, \theta \rho_k]\); in particular \(U_k = 0\) is not locally attracting, because the linearization at \(U_k = 0\) is \(\dot{U}_k = F_k + O(\rho_k^{2L - 1}) > 0\) with \(F_k > 0\) bounded below in \(\rho_k^{L - 1}\).

11.6.0.3 Escape time.

On the rescaled state \(u_k = U_k / \rho_k \in (0, \theta)\), the effective RHS \(F_k + K^{(\sigma)} U_k^{L - 1}\) rescales (using 63 and \(U_k^{L - 1} = \rho_k^{L - 1} u_k^{L - 1}\)) to \[\rho_k \dot{u}_k = c_k \beta_k \rho_k^{L - 1} + K^{(\sigma)} \rho_k^{L - 1} u_k^{L - 1} + (\text{drag}, O(\rho_k^{2L - 1})),\] i.e.\(\dot{u}_k = \Gamma_k (1 + u_k^2)^{(L - 1)/2} \cdot (1 + O(\rho_k^L))\) after absorbing the forcing into the \(+1\) and the self-drive into the \(u_k^2\) term inside the bracket (the chain-moment identity of 49 fixes the form of the bracket; the same-order forcing term supplies the constant \(\Gamma_k = K_L^{(k, \sigma)} \rho_k^{L - 2}\)). Integrating from \(u_k(0) = 0\) to \(u_k = \theta\), \[t_k = \int_0^{\theta} \frac{d u_k}{\Gamma_k (1 + u_k^2)^{(L - 1)/2}} \bigl(1 + O(\rho_k^L)\bigr) = \Gamma_k^{-1} \mathcal{I}_L(\theta) \bigl(1 + O(\rho_k^L)\bigr).\] Applying (S1) at the final step, \[t_k = \Theta\bigl(\Gamma_k^{-1}\bigr) = \Theta\bigl(\rho_k^{-(L - 2)}\bigr) = \Theta\bigl(\varepsilon^{-(r_k - 2)}\bigr),\] with \(\mathcal{I}_2(\theta) = \mathrm{arcsinh}(\theta)\), \(\mathcal{I}_3(\theta) = \arctan(\theta)\), and \(\mathcal{I}_L(\theta) = \theta {}_2 F_1\bigl(\tfrac12, \tfrac{L - 1}{2}; \tfrac32; -\theta^2\bigr)\) for \(L \ge 4\). The drag correction is \(O(\rho_k^L) = O(\varepsilon^{L (r_k - 2)/(L - 2)})\) relative, absorbed into the \((1 + o(1))\) of the escape law 4. The successive ratio \(t_{k + 1}/t_k = \Gamma_{k + 1}^{-1}/\Gamma_k^{-1} = (\beta_k / \beta_{k + 1}) \cdot (\rho_k / \rho_{k + 1})^{L - 2}\) depends on mode weights and critical depths alone, through (S1).

11.7 Proof of the Escape-Time Homotopy Identity↩︎

We prove 5 via Liouville duality. Let \(\Theta_\nu\) denote the escape-time function of the \(f_\nu\)-flow: the unique solution of the Liouville transport equation \[\label{eq:liouville} f_\nu \cdot \nabla \Theta_\nu = -1\tag{65}\] with boundary data \(\Theta_\nu \equiv 0\) on the threshold hypersurface \(\{h = 0\}\). Along any \(f_\nu\)-trajectory \(x_\nu(t)\) starting from the shared initial state \(x_0\), integrating 65 along characteristics gives the orbit identity \(\Theta_\nu(x_\nu(t)) = T(\nu) - t\). Evaluating at \(t = 0\), \[T(\nu) = \Theta_\nu(x_0), \qquad \text{so} \qquad T'(\nu) = \partial_\nu \Theta_\nu(x_0).\]

Transport equation for \(\partial_\nu \Theta_\nu\). Differentiating 65 in \(\nu\) gives \(f_\nu \cdot \nabla(\partial_\nu \Theta_\nu) = -(\partial_\nu f_\nu) \cdot \nabla \Theta_\nu\), with boundary data \(\partial_\nu \Theta_\nu \equiv 0\) on \(\{h = 0\}\) (inherited from \(\Theta_\nu \equiv 0\) there). Along the \(f_\nu\)-flow, \(\frac{d}{dt}[\partial_\nu \Theta_\nu(x_\nu(t))] = -[(\partial_\nu f_\nu) \cdot \nabla \Theta_\nu](x_\nu(t))\). Integrating from \(0\) to \(T(\nu)\) and using \(\partial_\nu \Theta_\nu(x_\nu(T(\nu))) = 0\), \[\label{eq:T-prime} T'(\nu) = \partial_\nu \Theta_\nu(x_0) = \int_0^{T(\nu)} \bigl[(\partial_\nu f_\nu) \cdot \nabla \Theta_\nu\bigr](x_\nu(t)) d t.\tag{66}\]

Identification with the escape-time adjoint. Define the escape-time adjoint along the trajectory by \[\label{eq:p-def} p_\nu(t) \doteq -\nabla \Theta_\nu(x_\nu(t)).\tag{67}\] Then \(p_\nu\) satisfies the three stated conditions:

1. Adjoint ODE. Differentiating \(\nabla(f_\nu \cdot \nabla \Theta_\nu) = \nabla(-1) = 0\) gives \((D^2 \Theta_\nu) f_\nu = -(\partial_x f_\nu)^\top \nabla \Theta_\nu\), so \(\frac{d}{dt}[\nabla \Theta_\nu(x_\nu(t))] = (D^2 \Theta_\nu) f_\nu = -(\partial_x f_\nu)^\top \nabla \Theta_\nu\). Negating, \(\dot{p}_\nu = -(\partial_x f_\nu)^\top p_\nu\).

2. Normalization. From 65 , \(p_\nu(t) \cdot f_\nu(x_\nu(t)) = -\nabla \Theta_\nu \cdot f_\nu = +1\) along the trajectory.

3. Terminal condition. On \(\{h = 0\}\), \(\nabla \Theta_\nu\) is proportional to the unit conormal \(\nabla h / \|\nabla h\|\) (because \(\Theta_\nu \equiv 0\) on \(\{h = 0\}\) forces \(\nabla \Theta_\nu\) tangent to its level set to vanish), and the normalization \(\nabla \Theta_\nu \cdot f_\nu = -1\) fixes the proportionality: \[\nabla \Theta_\nu(x_\nu(T(\nu))) = \frac{-\nabla h}{\nabla h \cdot f_\nu}\Big|_{T(\nu)}, \qquad \text{hence} \qquad p_\nu(T(\nu)) = \frac{\nabla h}{\nabla h \cdot f_\nu}\Big|_{T(\nu)},\] matching the terminal condition in 5.

Substituting \(\nabla \Theta_\nu = -p_\nu\) into 66 , \[T'(\nu) = -\int_0^{T(\nu)} p_\nu(t) \cdot (\partial_\nu f_\nu)(x_\nu(t)) d t = A(\nu),\] and integration on \([0, 1]\) yields ?? .

12 Experimental and Numerical Details↩︎

All experiments train feedforward networks on the squared loss \(\mathcal{L} = \frac{1}{2}\mathbb{E}_{x}[(f(x) - y(x))^2]\) with isotropic Gaussian inputs \(x \sim \mathcal{N}(0, I_d)\) and a teacher network of the same activation as the student. Population expectations are approximated by drawing a fresh Gaussian batch at every gradient step (no fixed dataset). Unless otherwise noted, training uses the normalized-metric convention: the first-layer learning rate is \(\eta\) and all deeper layers use \(\eta / N\), matching the gradient flow whose fixed-point manifold is the symmetric balanced ansatz of 1. Three initialization schemes appear across the figures: symmetric-aligned (1), modewise-blocks (partitioning hidden units into teacher-aligned blocks), and He-normal with bottleneck rescaling (standard signal-propagating init with selected layers compressed by \(\varepsilon\)).

12.0.0.1 1 (Exactness of the ansatz reduction).

We train a \(4\)-layer tanh network of width \(N = 64\) on a single-mode teacher with \(d = 16\) and \(\beta_1 = 1\), using symmetric-aligned initialization at the unbalanced scale \(X_0 = (0.03, 0.05, 0.07, 0.09)\). Full-network gradient descent uses batch size \(32 768\) with the normalized-metric per-layer learning rates, run for \(120 000\) steps with \(25\) evenly spaced snapshots recorded. Panel (a) overlays the population loss from full GD (circles) against the reduced ODE trajectory of 2 (solid curve); panel (b) does the same for the per-layer scales \(X_\ell\). The reduced ODE is integrated via LSODA [37] with relative tolerance \(10^{-12}\) and absolute tolerance \(10^{-14}\); Gaussian expectations in the ODE right-hand side are computed by \(128\)-node Gauss–Hermite quadrature.

12.0.0.2 2 (Depth scaling and critical-depth law).

We evaluate the closed-form escape integral and the reduced ODE for a tanh teacher–student pair with \(N = 64\), \(d = 16\), and \(\beta_1 = 1\). Panel (a) uses balanced initialization \(X_\ell = \varepsilon\) for all layers, with \(\varepsilon\) swept over \(20\) log-uniform points in \([10^{-2}, 10^{-0.75}]\) and \(L \in \{3, \ldots, 6\}\). Panel (b) fixes \(L = 6\) and places \(r \in \{3, \ldots, 6\}\) layers at scale \(\varepsilon\) with the remaining \(L - r\) layers at \(\Theta(1)\), over the same \(\varepsilon\) grid. The closed-form leading-order integral of 3 is evaluated by adaptive Gauss–Kronrod quadrature with absolute and relative tolerances of \(10^{-10}\); the integrand singularity at \(Y = \varepsilon^2\) is regularized by the substitution \(Y = \varepsilon^2 e^{2s}\). The reduced-ODE escape times integrate the exact flow of 4 via LSODA, terminated at the first crossing of \(X_1 = 0.3\).

12.0.0.3 3 (Universality collapse).

We test five activations — three Class B (tanh, erf, sin) and two Class C (GELU, Swish) — on a \(4\)-layer, width-\(64\) network with \(d = 16\) and a single-mode teacher (\(\beta_1 = 1\)), all under symmetric-aligned initialization. Theory curves are the closed-form escape integral evaluated on a \(14\)-point log-uniform \(\varepsilon\) grid over \([3 \times 10^{-3}, 10^{-1}]\). Full-network GD empirics use batch size \(1024\), step size \(0.05\), and \(5\) independent seeds per \(\varepsilon\) value; Class B activations are tested at \(\varepsilon \in [0.03, 0.1]\) and Class C at \(\varepsilon \in [0.1, 0.316]\) (shifted range, since Class C escapes faster at matched scale). Escape is defined as the first crossing of \(X_1 = 0.5\). Reduced-ODE escape times (diamonds) are computed at the same \(\varepsilon\) values. Panel (b) rescales all curves by the analytically computed activation constant \(K^{(\sigma)}\) of 1, demonstrating the claimed collapse.

12.0.0.4 4 (Critical-depth law under random initialization).

We train an \(8\)-layer tanh network of width \(N = 128\) on a single-mode teacher with \(d = 16\) and \(\beta_1 = 1\), under He-normal initialization: all layers are initialized with entries drawn i.i.d.from \(\mathcal{N}(0, 2/\mathrm{fan\_in})\), then the first \(r\) layers are rescaled by \(\varepsilon\) to create a bottleneck of critical depth \(r\). Training uses SGD with batch size \(512\), a uniform learning rate of \(0.01\) across all layers, and fresh Gaussian inputs drawn each step. Escape is defined as the first step at which the instantaneous loss drops below \(0.02\), approximately \(10\%\) of the uninformative baseline \(\tfrac{1}{2}\mathrm{Var}(y) \approx 0.19\) (loss-based detection is used because, off the symmetric manifold, the network learns a distributed representation that does not align \(W_1\) with the teacher direction).

Threshold sensitivity. To verify that the fitted slopes are not artefacts of the choice of \(0.02\) we repeated the regression at \(0.01\) (early escape) and \(0.05\) (late escape). Across \(r \in \{3, 5, 8\}\) the slope estimates shift by \(|\Delta \mathrm{slope}| \le 0.08\) and the \(r\)-ordering of the three slopes is preserved. We report the intermediate value \(0.02\) in the main figure.

Disjoint \(\varepsilon\) windows. We test \(r \in \{3, 5, 8\}\) with per-\(r\) initialization scales chosen so that escape is feasible within \(800 000\) SGD steps: \(\varepsilon \in [0.01, 0.3]\) for \(r = 3\), \([0.06, 0.5]\) for \(r = 5\), and \([0.15, 0.5]\) for \(r = 8\), with \(5\) seeds per configuration. The windows are partially disjoint — smaller \(r\) reaches smaller \(\varepsilon\) before the escape budget is exhausted, while larger \(r\) requires larger \(\varepsilon\) to fit in the budget — so the three slopes are measured on non-identical \(\varepsilon\) ranges, and the figure does not directly compare absolute escape times across \(r\), only the slope of \(\log t_\mathrm{esc}\) in \(\log \varepsilon\). The theory reference lines of slope \(-(r - 2)\) are anchored at the largest-\(\varepsilon\) data point for each \(r\), to isolate the scaling exponent from prefactor variation.

Fitted slopes. Ordinary least-squares regression of \(\log t_\mathrm{esc}\) on \(\log \varepsilon\) over each per-\(r\) window, on the seed-mean escape times:

Fitted slopes agree with the predicted \(-(r-2)\) law to within \(\le 0.6\) across all three \(r\), with the deviation consistent with the finite-\(\varepsilon\) corrections one expects from the bootstrap interval \([0, \tau_{m_0}]\) of 5.

12.0.0.5 5 (Closure hierarchy).

We train a \(4\)-layer tanh network of width \(N = 60\) on a \(3\)-mode teacher with \(d = 16\), orthonormal directions, and amplitudes \(\beta = (1.0, 0.3, 0.08)\). Initialization uses modewise blocks: the \(60\) hidden units are partitioned into \(3\) blocks of \(20\), each placed on the symmetric-aligned ansatz along its teacher direction at scale \(\varepsilon = 0.05\). Training runs for \(150 000\) steps with batch size \(4096\) and normalized-metric learning rates (\(\eta / N_B\) with block size \(N_B = N/K = 20\)). The two vertical predictions are: (i) the leading-order single-mode escape integral of 3, and (ii) the escape-time homotopy identity of 5, evaluated by \(2\)-node Gauss–Legendre quadrature at \(\nu = \frac{1}{2} \pm \frac{1}{2\sqrt{3}}\). At each quadrature node, the forward trajectory \(\dot{x}_\nu = f_\nu(x_\nu)\) and backward adjoint \(\dot{p}_\nu = -(\partial_x f_\nu)^\top p_\nu\) are integrated jointly. The Jacobian \(\partial_x f_\nu\) is constructed analytically in closed form from the reduced-variable equations of motion (layer scales, off-block amplitudes, cross-block rotations): each Jacobian entry is a polynomial in the reduced state with Gaussian moments of \(\sigma\) and \(\sigma'\) appearing as coefficients, and these moments are precomputed by Gauss–Hermite quadrature. We cross-validated the derived Jacobian against JAX reverse-mode autodiff on three random reduced states at tolerance \(10^{-10}\), obtaining agreement to \(\le 3 \times 10^{-11}\) in entrywise \(\ell_\infty\) norm. The augmented closure \((X, C_{kk'}, \eta_{k,m})\) has \(24\) degrees of freedom at \(L = 4\), \(K = 3\); extending to per-layer cross-block amplitudes adds \(12\) further variables. All ODE systems use the same adaptive RK4 integrator with local truncation error below \(10^{-9}\).

References↩︎