The Mathematics of AI Winters: The mathematical Taxonomy of Paradigm Fragility in AI Winter
June 10, 2026
Abstract
The two major periods of reduced funding and confidence in artificial intelligence research, usually described as the first and second AI winters, are often explained in terms of engineering failure, commercial disappointment, and inflated expectations. This article develops a complementary thesis. It argues that the dominant paradigms of the relevant periods also encountered genuine formal barriers: limitations of representation, optimisation, computational complexity, statistical learnability, and high-dimensional approximation. The contribution is synthetic rather than archival. We do not claim that particular theorems mechanically caused the winters. Rather, we show that several central disappointments of early AI were aligned with mathematically precise bottlenecks. We analyse those bottlenecks through the perceptron impossibility results associated with [1], the complexity-theoretic hardness of exact neural-network training [2], minimax rates for nonparametric estimation in high dimension [3], vanishing-gradient analyses [4], [5], and classical statistical learning theory [6]–[8]. We then relate these barriers to the later breakthroughs that mitigated, rather than eliminated, them.
Keywords: AI winter, PAC learning, VC dimension, NP-hardness, curse of dimensionality, vanishing gradients, universal approximation, double descent, bias–variance trade-off.
The Mathematics of AI Winters
A Mathematical Taxonomy of Paradigm Fragility in AI Winter
Miquel Noguer i Alonso (AIFI), David Pacheco Aznar (Staq.io)
June 2026
1 Introduction↩︎
The history of artificial intelligence is marked by repeated cycles of expansion and retrenchment. Two episodes are conventionally identified as AI winters: the first, associated with the mid-1970s, followed the critical reception of ambitious symbolic and perceptron-based programmes, most visibly after the Lighthill Report [9], the philosophical critique of [10], and the reception of Perceptrons [1]; the second, associated with the late 1980s and early 1990s, followed the collapse of the expert-systems market — whose intellectual programme is best represented by [11] and [12] — and the limited practical performance of shallow connectionist systems. The standard explanations emphasise commercial overpromising, insufficient hardware, brittle engineering, and institutional disappointment. Those explanations are important, but they are not the whole story.
This article studies a narrower claim. It argues that the dominant paradigms of those periods also faced mathematically identifiable bottlenecks. Some were representational, as in the inability of single-layer threshold models to capture non-linearly separable relations. Some were computational, as in the combinatorial explosion of symbolic search and the worst-case hardness of exact training. Some were statistical, as in the weakness of then-available generalisation guarantees for flexible models. Others were optimisation-theoretic, as in the instability of deep gradient propagation under saturating nonlinearities. The aim is therefore not to reduce intellectual history to theorem-proving. It is to show that the disappointments of those eras were, in part, consistent with formal limits rather than merely with poor execution.
The contribution is mainly synthetic. None of the underlying theorems is new. What is new is their integration into a unified mathematical interpretation of AI winters.
1.0.0.1 Methodological clarification.
At the level of history, our claim is one of compatibility and necessary vulnerability, not monocausal explanation. At the level of mathematics, the novelty of the paper lies not in new lower bounds but in a budgeted synthesis: representational limits, optimisation hardness, and finite-sample fragility are analysed as jointly binding constraints under shared data and compute budgets. This narrower claim is also the stronger one, because it is precisely the claim for which the cited theory can supply genuine support.
Throughout, we distinguish three notions that are often conflated in informal discussion: representational capacity, learnability from finite samples, and tractable trainability. A model class may be expressive without being efficiently trainable; it may be trainable without carrying non-vacuous finite-sample guarantees; and it may possess asymptotic guarantees while remaining practically unusable at realistic scales. This separation is crucial for interpreting both the failures of early AI and the later successes of deep learning.
A further qualification is necessary. The theorems discussed below did not by themselves cause the winters. The historical episodes were also shaped by economics, hardware, programming practice, benchmark choice, and the sociology of research communities. Our claim is therefore deliberately limited: formal barriers materially contributed to the fragility of the leading paradigms, and later progress became durable only when those barriers were mitigated by new mathematics, algorithms, and architectures.
The article proceeds as follows. Section 2 introduces the formal learning framework. Section 3 studies representational barriers and the classical perceptron results. Section 4 examines computational hardness, combinatorial explosion, and high-dimensional estimation. Section 5 derives the vanishing-gradient phenomenon. Section 6 reviews the statistical barriers associated with generalisation. Section 7 considers the mathematical developments that mitigated these difficulties. Section 8 summarises the argument in comparative form, and Section 9 concludes.
1.0.0.2 Status of results.
To avoid overstating what is proved here, we classify the mathematical content of the article into four categories. (i) Classical theorems cited from the literature: the perceptron impossibility results, the Blum–Rivest NP-hardness theorem, the Stone minimax rate, the vanishing-gradient analyses, and the foundational PAC/VC bounds. These are used as given and credited to their original sources. (ii) Framework synthesis: Definition 1 (the barrier triple) and Proposition 28 (the Joint-Binding condition) organise the classical results into a single joint-binding condition parameterised by a data and compute budget. The proposition is a structural reorganisation of the preceding sections, not a new lower bound; its proof is labelled a synthesis for this reason. (iii) Interpretive remarks: the connections drawn between the barrier triple and modern developments (scaling laws, neural collapse, lottery tickets, implicit bias) are post-hoc lenses, not historical-causal claims. (iv) Historical context: all attributions of winter episodes to formal barriers are stated as necessary-condition arguments, never as sufficient ones; the actual historical interventions (multilayer architectures, non-saturating activations, improved initialisation, large corpora, hardware scaling) are named explicitly where relevant.
1.1 A Formal Notion of an AI Winter Barrier↩︎
The central thesis of this article can be stated more formally by identifying the mathematical structure underlying paradigm fragility.
Definition 1 (AI Winter Barrier). Let \(\mathcal{A}\) be a class of AI architectures and \(\mathcal{T}\) a class of learning or reasoning tasks. A formal AI winter barrier is a triple \[\mathcal{B}(\mathcal{A},\mathcal{T}) = \bigl(R(\mathcal{A},\mathcal{T}),\; C(\mathcal{A},\mathcal{T}),\; S(\mathcal{A},\mathcal{T})\bigr),\] where:
\(R\) is a representational constraint limiting the set of functions or relations realisable by \(\mathcal{A}\);
\(C\) is a computational or optimisation constraint implying that training, inference, or search over \(\mathcal{A}\) is intractable in the worst case or scales prohibitively in practice;
\(S\) is a statistical constraint under which finite-sample guarantees become vacuous or sample requirements become unrealistic at relevant scales.
We say that a paradigm operates in a winter regime if at least two of these constraints bind simultaneously under realistic data and compute budgets.
Remark 1. We will make the joint-binding condition precise in Proposition 28, where the data budget \(n\) and compute budget \(\kappa\) appear as explicit parameters and each constraint is tied to a quantitative obstruction. Definition 1 itself remains at a structural level: it fixes vocabulary, not thresholds.
Remark 2. This definition is not intended as a historical law. It is a structural device that organises the various theorems discussed below into a common analytical framework. Its purpose is to make precise what it means for an AI paradigm to be mathematically fragile.
1.1.0.1 Central insight.
AI winters can be interpreted as periods in which the dominant paradigms faced a joint failure regime: representational restrictions, computational intractability, and weak statistical control were not isolated problems but interacting constraints. The formal synthesis is given in Proposition 28 (Section 8).
1.1.0.2 Notation.
We write \(\mathbb{R}\) for the reals, \(\mathbb{N}\) for the natural numbers, \([n] = \{1,\ldots,n\}\). Vectors are bold lowercase (\(\mathbf{x}\)), matrices bold uppercase (\(\mathbf{W}\)). \(\left\lVert \cdot \right\rVert\) denotes the Euclidean norm unless otherwise specified. \(\mathbb{P}[\cdot]\) and \(\mathop{\mathrm{\mathbb{E}}}[\cdot]\) are probability and expectation with respect to a distribution clear from context.
2 The Formal Learning Framework↩︎
2.1 The PAC Model↩︎
We work throughout in the Probably Approximately Correct (PAC) learning framework of [7].
Definition 2 (Learning Problem). Let \(\mathcal{X}\) be an instance space, \(\mathcal{Y}\) an output space, and \(\mathcal{D}\) an unknown joint distribution over \(\mathcal{X}\times \mathcal{Y}\). Given a hypothesis class \(\mathcal{H}\subseteq \mathcal{Y}^{\mathcal{X}}\) and a loss function \(\ell: \mathcal{Y}\times \mathcal{Y}\to \mathbb{R}_{\geq 0}\), the true risk of \(h \in \mathcal{H}\) is \[L_{\mathcal{D}}(h) \;=\; \mathop{\mathrm{\mathbb{E}}}_{(x,y) \sim \mathcal{D}}[\ell(h(x), y)],\] and the empirical risk on a sample \(S = \{(x_i, y_i)\}_{i=1}^m\) is \[L_S(h) \;=\; \frac{1}{m}\sum_{i=1}^m \ell(h(x_i), y_i).\]
Definition 3 (Realizable PAC Learnability). A binary hypothesis class \(\mathcal{H}\) is PAC learnable in the realizable setting if there exists an algorithm \(\mathcal{A}\) and a function \(m^{\mathrm{real}}_{\mathcal{H}}:(0,1)^2 \to \mathbb{N}\) such that whenever the data-generating distribution \(\mathcal{D}\) satisfies \(\inf_{h \in \mathcal{H}} L_{\mathcal{D}}(h) = 0\), then for every \(\varepsilon,\delta \in (0,1)\) and every \(m \geq m^{\mathrm{real}}_{\mathcal{H}}(\varepsilon,\delta)\), \[\mathbb{P}_{S \sim \mathcal{D}^m}\!\left[L_{\mathcal{D}}(\mathcal{A}(S)) \le \varepsilon\right] \ge 1-\delta.\]
Definition 4 (Agnostic PAC Learnability). A binary hypothesis class \(\mathcal{H}\) is agnostically PAC learnable if there exists an algorithm \(\mathcal{A}\) and a function \(m^{\mathrm{agn}}_{\mathcal{H}}:(0,1)^2 \to \mathbb{N}\) such that for every distribution \(\mathcal{D}\) over \(\mathcal{X}\times \{0,1\}\), every \(\varepsilon,\delta \in (0,1)\), and every \(m \ge m^{\mathrm{agn}}_{\mathcal{H}}(\varepsilon,\delta)\), \[\mathbb{P}_{S \sim \mathcal{D}^m}\!\left[L_{\mathcal{D}}(\mathcal{A}(S)) \le \inf_{h \in \mathcal{H}} L_{\mathcal{D}}(h) + \varepsilon\right] \ge 1-\delta.\]
The distinction matters. Realizable PAC learning assumes that the target relation is representable inside \(\mathcal{H}\), whereas agnostic PAC learning competes only with best-in-class risk. In modern learning theory the agnostic setting is the more robust formulation, but the cleanest classical VC equivalence is usually first stated in the realizable case; the agnostic characterization proceeds through uniform convergence and empirical risk minimisation [8], [13], [14].
2.2 VC Dimension and the Fundamental Theorem↩︎
Definition 5 (Shattering and VC Dimension). A set \(\mathcal{C} = \{x_1,\ldots,x_m\} \subset \mathcal{X}\) is shattered by \(\mathcal{H}\) if for every labelling \(\mathbf{y} \in \{0,1\}^m\) there exists \(h \in \mathcal{H}\) with \(h(x_i) = y_i\) for all \(i\). The Vapnik–Chervonenkis dimension is \[\mathop{\mathrm{VCdim}}(\mathcal{H}) \;=\; \sup\bigl\{m \in \mathbb{N}: \exists\, \mathcal{C} \subset \mathcal{X}, \; \left\lvert \mathcal{C} \right\rvert = m,\; \mathcal{H}\text{ shatters } \mathcal{C}\bigr\}.\]
Theorem 3 (Fundamental Theorem in the Realizable Case). [6], [8], [13] For binary classification in the realizable setting, a hypothesis class \(\mathcal{H}\) is PAC learnable if and only if \(d = \mathop{\mathrm{VCdim}}(\mathcal{H}) < \infty\). Moreover, the realizable sample complexity satisfies1 \[C_1\,\frac{d + \log(1/\delta)}{\varepsilon} \;\leq\; m^{\mathrm{real}}_{\mathcal{H}}(\varepsilon,\delta) \;\leq\; C_2\,\frac{d\log(1/\varepsilon) + \log(1/\delta)}{\varepsilon}\] for universal constants \(C_1,C_2>0\).
In the agnostic setting, finite VC dimension still characterises learnability, but the standard sample-complexity scale becomes \[m^{\mathrm{agn}}_{\mathcal{H}}(\varepsilon,\delta) = \Theta\!\left(\frac{d + \log(1/\delta)}{\varepsilon^2}\right),\] with the proof proceeding through uniform convergence and empirical risk minimisation rather than through the realizable reduction alone [8], [13], [14]. The key message is unchanged: classes with \(\mathop{\mathrm{VCdim}}(\mathcal{H}) \gg m\) do not admit non-vacuous finite-sample control.
Lemma 1 (Sauer–Shelah). Let \(d = \mathop{\mathrm{VCdim}}(\mathcal{H}) < \infty\). The growth function \[\Pi_{\mathcal{H}}(m) \;=\; \max_{\mathcal{C} \subset \mathcal{X},\,\left\lvert \mathcal{C} \right\rvert=m} \left\lvert \mathcal{H}_{\mathcal{C}} \right\rvert\] satisfies \[\Pi_{\mathcal{H}}(m) \;\leq\; \sum_{i=0}^{d} \binom{m}{i} \;\leq\; \left(\frac{em}{d}\right)^d.\]
Proof. By induction on \(m + d\). For \(m = 0\) or \(d = 0\) the bound is trivial. Partition any \(m\)-element set by taking \(x_m\) and noting that restrictions of \(\mathcal{H}\) to the remaining \(m-1\) elements satisfy both \(\Pi_{\mathcal{H}}(m-1) \leq \sum_{i=0}^{d}\binom{m-1}{i}\) and that pairs forced by \(x_m\) satisfy \(\Pi_{\mathcal{H}'}(m-1) \leq \sum_{i=0}^{d-1}\binom{m-1}{i}\). Adding via Pascal’s identity gives the result. ◻
3 The Perceptron Crisis: Representational Barriers↩︎
3.1 The Rosenblatt Perceptron↩︎
The perceptron [16] defines a linear classifier \[f_{\mathbf{w},b}(\mathbf{x}) \;=\; \mathop{\mathrm{sign}}\!\bigl(\langle \mathbf{w},\,\mathbf{x} \rangle + b\bigr), \qquad \mathbf{w} \in \mathbb{R}^d,\; b \in \mathbb{R}.\] The perceptron algorithm, introduced by Rosenblatt, was later placed on firm mathematical ground by Novikoff’s convergence theorem, which established that if the training data is linearly separable, the perceptron learning rule converges in finite steps.
Theorem 4 (Perceptron Convergence). [17] Let \(\{(\mathbf{x}_i, y_i)\}_{i=1}^m \subset \mathbb{R}^d \times \{-1,+1\}\) be linearly separable with margin \(\gamma = \min_i y_i \langle \mathbf{w}^*,\,\mathbf{x}_i \rangle/\left\lVert \mathbf{w}^* \right\rVert > 0\) and \(\left\lVert \mathbf{x}_i \right\rVert \leq R\) for all \(i\). Then the Perceptron algorithm makes at most \[T \;\leq\; \left(\frac{R}{\gamma}\right)^2\] mistakes before converging to a correct classifier.
Proof. Let \(\mathbf{w}_0 = \mathbf{0}\) and suppose mistake \(t\) occurs at \((\mathbf{x}_{i_t}, y_{i_t})\), updating \(\mathbf{w}_{t+1} = \mathbf{w}_t + y_{i_t}\mathbf{x}_{i_t}\). For the inner product with \(\mathbf{w}^*\): \[\langle \mathbf{w}_{t+1},\,\mathbf{w}^* \rangle = \langle \mathbf{w}_t,\,\mathbf{w}^* \rangle + y_{i_t}\langle \mathbf{x}_{i_t},\,\mathbf{w}^* \rangle \geq \langle \mathbf{w}_t,\,\mathbf{w}^* \rangle + \gamma\left\lVert \mathbf{w}^* \right\rVert.\] Thus \(\langle \mathbf{w}_T,\,\mathbf{w}^* \rangle \geq T\gamma\left\lVert \mathbf{w}^* \right\rVert\). For the squared norm: since \(y_{i_t}\langle \mathbf{w}_t,\,\mathbf{x}_{i_t} \rangle \leq 0\) on a mistake, \[\left\lVert \mathbf{w}_{t+1} \right\rVert^2 = \left\lVert \mathbf{w}_t \right\rVert^2 + 2y_{i_t}\langle \mathbf{w}_t,\,\mathbf{x}_{i_t} \rangle + \left\lVert \mathbf{x}_{i_t} \right\rVert^2 \leq \left\lVert \mathbf{w}_t \right\rVert^2 + R^2,\] giving \(\left\lVert \mathbf{w}_T \right\rVert^2 \leq TR^2\). Combining the two bounds via Cauchy–Schwarz, \[T^2\gamma^2\left\lVert \mathbf{w}^* \right\rVert^2 \;\leq\; \langle \mathbf{w}_T,\,\mathbf{w}^* \rangle^2 \;\leq\; \left\lVert \mathbf{w}_T \right\rVert^2\left\lVert \mathbf{w}^* \right\rVert^2 \;\leq\; TR^2\left\lVert \mathbf{w}^* \right\rVert^2,\] and dividing through by \(T\gamma^2\left\lVert \mathbf{w}^* \right\rVert^2\) yields \(T \leq (R/\gamma)^2\). ◻
3.2 The Minsky–Papert Impossibility↩︎
Despite this guarantee, the classical critique associated with [1] made clear that single-layer perceptrons are restricted to linearly separable decision rules. The XOR example is the standard elementary illustration. It should not be read as proving that multilayer networks are impossible, but rather as identifying a genuine representational bottleneck for the architectural class then most prominently discussed.
Theorem 5 (XOR Impossibility). [1] The XOR function \(f:\{0,1\}^2 \to \{-1,+1\}\) defined by \(f(x_1,x_2) = 1 \iff x_1 \neq x_2\) is not linearly separable. That is, there exist no \(w_1, w_2, b \in \mathbb{R}\) such that \[\mathop{\mathrm{sign}}(w_1 x_1 + w_2 x_2 + b) = f(x_1, x_2) \quad \forall\, (x_1,x_2) \in \{0,1\}^2.\]
Proof. Suppose for contradiction such \(w_1, w_2, b\) exist. The four conditions are: \[\begin{align} b &< 0 \tag{1} \\ w_1 + b &> 0 \tag{2} \\ w_2 + b &> 0 \tag{3} \\ w_1 + w_2 + b &< 0. \tag{4} \end{align}\] Adding 2 and 3 : \(w_1 + w_2 + 2b > 0\), so \(w_1 + w_2 > -2b > 0\) (using 1 ). Therefore \(w_1 + w_2 + b > -2b + b = -b > 0\), contradicting 4 . ◻
The XOR example is the most elementary case of a broader phenomenon. Minsky and Papert systematically characterised which Boolean functions are computable by a single perceptron, showing the class to be severely limited. Crucially, they demonstrated that adding more neurons in a single layer — while possibly increasing the number of computable functions — could not overcome the fundamental linear-separability constraint.
3.3 VC Dimension of Linear Classifiers↩︎
Theorem 6 (VC Dimension of Halfspaces). For the class of halfspace classifiers \(\mathcal{H}= \{(\mathbf{x} \mapsto \mathop{\mathrm{sign}}(\langle \mathbf{w},\,\mathbf{x} \rangle+b)) : \mathbf{w} \in \mathbb{R}^d, b \in \mathbb{R}\}\): \[\mathop{\mathrm{VCdim}}(\mathcal{H}) = d + 1.\]
Proof sketch. Lower bound (\(\mathop{\mathrm{VCdim}}\ge d+1\)): Consider the \(d+1\) affinely independent points \(\{\mathbf{0},\mathbf{e}_1,\ldots,\mathbf{e}_d\} \subset \mathbb{R}^d\). For any target labelling \((y_0,y_1,\ldots,y_d) \in \{-1,+1\}^{d+1}\), take \(b = \tfrac{1}{2}y_0\) and, for \(i = 1,\ldots,d\), set \(w_i = y_i - b\). Then \(\mathop{\mathrm{sign}}(b) = y_0\) and \(\mathop{\mathrm{sign}}(w_i + b) = \mathop{\mathrm{sign}}(y_i) = y_i\) for each \(i\), so every labelling is realised.
Upper bound (\(\mathop{\mathrm{VCdim}}\le d+1\)): By Radon’s theorem, any set of \(d+2\) points in \(\mathbb{R}^d\) admits a partition into two non-empty subsets \(A,B\) whose convex hulls intersect. Label the points in \(A\) by \(+1\) and those in \(B\) by \(-1\). If a halfspace separated this labelling, convexity would force the intersection point of \(\operatorname{conv}(A)\) and \(\operatorname{conv}(B)\) to be assigned both signs, a contradiction. Hence no halfspace can shatter \(d+2\) points. ◻
The consequence is fundamental: the realizable PAC sample complexity for halfspace classifiers in \(\mathbb{R}^d\) is linear in \(d\), while the agnostic rate is of order \(d/\varepsilon^2\). For XOR in \(\mathbb{R}^2\), under the uniform distribution on the four points of \(\{0,1\}^2\), the minimum classification error of any halfspace is \(1/4\). Thus the best attainable risk inside the class is strictly positive, a class-specific irreducible error arising from insufficient representational power.
4 Computational Complexity Barriers↩︎
4.1 NP-Hardness of Neural Network Training↩︎
Even if an architecture has sufficient representational power, one must still find the weights. This problem is provably hard.
Theorem 7 (Blum–Rivest Hardness). [2] The following decision problem is NP-complete: given a set of labelled examples \(\{(\mathbf{x}_i, y_i)\}_{i=1}^m \subset \{0,1\}^n \times \{0,1\}\), decide whether there exist weights \(\mathbf{W}\) for a neural network with \(2\) hidden units and one output unit (all with threshold activations) that correctly classifies all examples.
The proof proceeds by reduction from the 3-Dimensional Matching problem, which is NP-complete. The key insight is that two hidden threshold units can encode two hyperplanes whose intersection region has sufficient combinatorial richness to simulate Boolean constraints.
Remark 8 (What the hardness result does and does not imply). Theorem 7 is a worst-case statement. It does not imply that every practically arising instance of network training is hard. Its relevance is more precise: representational sufficiency does not by itself entail efficient trainability. In particular, the existence of weights realising a target relation gives no general reason to expect that a reliable optimisation procedure will find them within realistic resource budgets.
4.2 The Lighthill Report and Combinatorial Explosion↩︎
The Lighthill Report [9] identified what it termed “combinatorial explosion” as the central failure mode of AI systems. In modern mathematical language, this is the exponential blowup of the search space in tree- and graph-based reasoning.
For a state-space search tree with branching factor \(b\) and depth \(d\), the number of nodes at depth \(d\) is \(b^d\). The total search space is \[\left\lvert \text{Search Tree} \right\rvert = \sum_{k=0}^{d} b^k = \frac{b^{d+1}-1}{b-1} = \Theta(b^d).\] For the game of chess with \(b \approx 35\) and \(d \approx 80\), this is \(35^{80} \approx 10^{123}\) — far beyond any computational budget. Alpha-beta pruning reduces this to \(\Theta(b^{d/2})\) in the best case, still \(10^{61}\) for chess.
Expert systems faced an analogous problem. In the worst case, matching rule antecedents against a working memory of \(p\) atomic propositions involves a search over subsets of conditions whose size can grow as \(2^p\); the precise behaviour depends on the match algorithm and on how many of the \(n\) rules are simultaneously active, so the bound should be read as an illustrative worst case rather than a tight complexity figure. The frame problem [18] — the difficulty of representing what does not change after an action — is a further source of knowledge-base bloat: one natural representation requires \(\mathcal{O}(a \cdot f)\) persistence axioms for \(a\) actions and \(f\) facts. As both grow, engineering and maintaining such systems becomes increasingly costly, though the precise complexity depends on the representation language and inference strategy; the \(\mathcal{O}(a^2 f^2)\) figure sometimes cited informally reflects the cross-product of action–fact pairs and is not a formal worst-case theorem.
4.3 The Curse of Dimensionality↩︎
[19] first articulated the curse of dimensionality in the context of dynamic programming. The phenomenon manifests across all areas of machine learning.
Theorem 9 (Stone’s Minimax Rate). [3] Let \(\Sigma(s, L; [0,1]^d)\) denote the Hölder class of functions on \([0,1]^d\) whose \(\lfloor s\rfloor\)-th order partial derivatives are \((s - \lfloor s\rfloor)\)-Hölder continuous with constant \(L\). The minimax optimal estimation error over \(m\) i.i.d.samples satisfies2 \[\inf_{\hat{f}}\; \sup_{f \in \Sigma(s,L)} \mathop{\mathrm{\mathbb{E}}}\!\left[\left\lVert \hat{f} - f \right\rVert_{L^2}^2\right] \;\asymp\; m^{-2s/(2s+d)}.\]
This rate has a stark implication for sample requirements. Suppose we target mean squared \(L^2\) error at most \(\delta\), i.e. \(\mathop{\mathrm{\mathbb{E}}}[\|\hat{f} - f\|_{L^2}^2] \leq \delta\). Setting \(m^{-2s/(2s+d)} \lesssim \delta\) and solving for \(m\) gives \[m \;=\; \Omega\!\left(\delta^{-(2s+d)/(2s)}\right).\] Parametrising by the root-MSE \(\varepsilon = \sqrt{\delta}\), the exponent becomes \((2s+d)/s\) in \(\varepsilon\): \(m = \Omega(\varepsilon^{-(2s+d)/s})\). For \(s = 2\) and \(d = 10\) this exponent is \(14/2 = 7\), giving \(m = \Omega(\varepsilon^{-7})\); for \(d = 100\) it is \(104/2 = 52\), giving \(m = \Omega(\varepsilon^{-52})\).
To see at what dimension the minimax sample requirement exceeds a budget of \(m_0\) samples (for fixed \(\varepsilon\) and \(s\)), invert \(m_0 = \varepsilon^{-(2s+d)/s}\) to obtain \[d^* \;=\; s\,\frac{\log m_0}{\log(1/\varepsilon)} - 2s.\] For \(s = 2\) and \(m_0 = 10^4\): with \(\varepsilon = 0.1\) (ten-percent root-MSE) one gets \(d^* = 4\); with \(\varepsilon = 0.01\) one gets \(d^* = 0\), meaning the budget is already insufficient for the one-dimensional problem at that accuracy. The precise threshold depends sensitively on the target accuracy, but the qualitative conclusion is unambiguous: even modest dimension exhausts any practical sample budget.
Proposition 10 (Volume Concentration). In the unit ball \(B_1^d \subset \mathbb{R}^d\), for any \(\varepsilon \in (0,1)\): \[\frac{\mathrm{Vol}(B_1^d) - \mathrm{Vol}(B_{1-\varepsilon}^d)}{\mathrm{Vol}(B_1^d)} = 1 - (1-\varepsilon)^d \;\xrightarrow{d \to \infty}\; 1.\] That is, all volume concentrates near the surface* in high dimensions.*
Proof. The volume of a \(d\)-dimensional ball of radius \(r\) is \(V_d r^d\) where \(V_d = \pi^{d/2}/\Gamma(d/2+1)\). Thus \(\mathrm{Vol}(B_{1-\varepsilon}^d) = V_d(1-\varepsilon)^d\). Dividing: \(1 - (1-\varepsilon)^d \to 1\) since \((1-\varepsilon)^d \to 0\) exponentially for any fixed \(\varepsilon > 0\). ◻
This means that in high dimensions, nearest-neighbour methods, kernel density estimators, and all other methods that rely on local geometry effectively break down. It is important to distinguish two complementary phenomena: the volume concentration established above, and the related but distinct distance concentration.
Remark 11 (Distance Concentration). For i.i.d.\(\mathbf{x}, \mathbf{y} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_d)\), the strong law of large numbers gives \(\left\lVert \mathbf{x} \right\rVert^2/d \to 1\) and \(\left\lVert \mathbf{x}-\mathbf{y} \right\rVert^2/(2d) \to 1\) almost surely as \(d \to \infty\). More precisely, the Laurent–Massart concentration inequality for chi-squared random variables [21], [22] yields, for any \(t \in (0,1)\), \[\mathbb{P}\!\bigl[\,|\left\lVert \mathbf{x} \right\rVert^2/d - 1| > t\,\bigr] \;\leq\; 2\exp(-dt^2/8).\] A union bound over the \(\binom{m}{2}\) pairs then yields the stronger statement that if \(m=m(d)\) satisfies \(\log m = o(d)\), the relative spread of all pairwise distances converges to zero in probability: \[\frac{\max_{i \neq j}\left\lVert \mathbf{x}_i - \mathbf{x}_j \right\rVert - \min_{i \neq j}\left\lVert \mathbf{x}_i - \mathbf{x}_j \right\rVert}{\min_{i \neq j}\left\lVert \mathbf{x}_i - \mathbf{x}_j \right\rVert} \;\xrightarrow[d \to \infty]{\mathbb{P}}\; 0.\] Thus, whenever sample size grows subexponentially in the dimension, the geometry of pairwise distances becomes nearly degenerate, weakening the distinction between “near” and “far” for nearest-neighbour and kernel-based methods [23]. This is distinct from the volume-concentration proposition above, which concerns volume distribution within a ball; together, the two results give a complete picture of geometric degeneracy in high dimensions.
5 The Vanishing-Gradient Problem↩︎
5.1 Backpropagation and Gradient Flow↩︎
Backpropagation [24] provided an efficient method to compute gradients in multilayer networks. For a network with \(L\) layers: \[\begin{align} \mathbf{z}^{(\ell)} &= \mathbf{W}^{(\ell)}\mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)}, \qquad \ell = 1, \ldots, L, \\ \mathbf{a}^{(\ell)} &= \sigma(\mathbf{z}^{(\ell)}), \end{align}\] where \(\sigma\) is applied componentwise. The per-layer Jacobian of activations is \[\mathbf{J}^{(\ell)} \;:=\; \frac{\partial \mathbf{a}^{(\ell)}}{\partial \mathbf{a}^{(\ell-1)}} \;=\; \operatorname{diag}\!\bigl(\sigma'(\mathbf{z}^{(\ell)})\bigr)\,\mathbf{W}^{(\ell)}, \qquad \ell = 2,\ldots,L,\] and the gradient of the loss with respect to the first-layer activations is obtained by repeated application of the chain rule, composing Jacobians from output to input: \[\label{eq:chain} \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(1)}} \;=\; \underbrace{\frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(L)}}}_{\boldsymbol{\delta}^{(L)\top}} \,\mathbf{J}^{(L)}\,\mathbf{J}^{(L-1)}\,\cdots\,\mathbf{J}^{(2)}.\tag{5}\] The first-layer weight gradient \(\partial \mathcal{L}/\partial \mathbf{W}^{(1)}\) is then obtained by composing 5 with \(\partial \mathbf{a}^{(1)}/\partial \mathbf{W}^{(1)}\). The object of interest for gradient magnitude is therefore the product of \(L-1\) Jacobian matrices, each of which combines the spectral properties of the weight matrices with the derivative of the activation function.
5.2 Formal Analysis of Gradient Decay↩︎
Theorem 12 (Vanishing Gradient in the Contractive Regime). [4], [5] Let \(\sigma:\mathbb{R}\to\mathbb{R}\) be Lipschitz with \(\sup_z|\sigma'(z)| =: \kappa < \infty\) and suppose the weight matrices \(\{\mathbf{W}^{(\ell)}\}\) satisfy \(\left\lVert \mathbf{W}^{(\ell)} \right\rVert_2 \le \rho\) for every layer, with \(\rho\kappa < 1\). Then the backward Jacobian product satisfies \[\left\lVert \mathbf{J}^{(L)}\,\mathbf{J}^{(L-1)}\cdots\mathbf{J}^{(2)} \right\rVert_2 \;\le\; \prod_{\ell=2}^{L} \left\lVert \mathbf{J}^{(\ell)} \right\rVert_2 \;\le\; (\rho\kappa)^{L-1} \;=:\; c^{L-1}\] with \(c<1\), so the backpropagated gradient decays exponentially in depth.
Proof. By submultiplicativity of the operator norm, \(\left\lVert AB \right\rVert_2 \le \left\lVert A \right\rVert_2\left\lVert B \right\rVert_2\). Since \(\mathbf{J}^{(\ell)} = \operatorname{diag}(\sigma'(\mathbf{z}^{(\ell)}))\mathbf{W}^{(\ell)}\), \[\left\lVert \mathbf{J}^{(\ell)} \right\rVert_2 \le \left\lVert \operatorname{diag}(\sigma'(\mathbf{z}^{(\ell)})) \right\rVert_2\,\left\lVert \mathbf{W}^{(\ell)} \right\rVert_2 \le \kappa\,\rho.\] Multiplying over the \(L-1\) factors gives the result. For \(\sigma=\tanh\) one has \(\kappa = \sup_z|\sigma'(z)| = 1\), so the hypothesis reduces to the explicitly contractive condition \(\rho<1\). ◻
The theorem isolates one mechanism of gradient decay: spectral contraction. A second mechanism is saturation. Even when \(\left\lVert \mathbf{W}^{(\ell)} \right\rVert_2\) is not strictly below \(1\), the factor \(\left\lVert \operatorname{diag}(\sigma'(\mathbf{z}^{(\ell)})) \right\rVert_2\) becomes small if many pre-activations lie in regions where \(|\sigma'(z)| \ll 1\). For sigmoids and hyperbolic tangents this occurs when \(|z|\) is large. Hence vanishing gradients can arise either because the linear part is contractive or because the nonlinearity is locally saturated; historically, both mechanisms mattered.
5.3 The Edge of Chaos: Mean-Field Analysis↩︎
A sharper, distribution-level account of gradient propagation is given by the mean-field theory of random initialisations. For \(W_{ij}^{(\ell)} \sim \mathcal{N}(0, \sigma_w^2/n)\) and \(b_i^{(\ell)} \sim \mathcal{N}(0, \sigma_b^2)\), define the order parameter \[q^{(\ell)} \;=\; \frac{1}{n}\sum_{i=1}^n \mathop{\mathrm{\mathbb{E}}}\!\left[(a_i^{(\ell)})^2\right].\]
Proposition 13 (Mean-Field Recursion). [25] To keep the variance bookkeeping precise, it is convenient to distinguish pre-activation and activation second moments. Define \[q_z^{(\ell)} := \frac{1}{n}\sum_{i=1}^n \mathop{\mathrm{\mathbb{E}}}[(z_i^{(\ell)})^2], \qquad q_a^{(\ell)} := \frac{1}{n}\sum_{i=1}^n \mathop{\mathrm{\mathbb{E}}}[(a_i^{(\ell)})^2].\] Under the random initialisation \(W_{ij}^{(\ell)}\sim\mathcal{N}(0,\sigma_w^2/n)\) and \(b_i^{(\ell)}\sim\mathcal{N}(0,\sigma_b^2)\), the mean-field recursion takes the form \[q_z^{(\ell+1)} \;=\; \sigma_w^2 q_a^{(\ell)} + \sigma_b^2, \qquad q_a^{(\ell+1)} \;=\; \int_{-\infty}^{\infty} \sigma\!\left(\sqrt{q_z^{(\ell+1)}}\,u\right)^2 \frac{e^{-u^2/2}}{\sqrt{2\pi}}\,\mathrm{d}u.\] Equivalently, if one prefers a one-variable recursion, then the quantity denoted \(q^{(\ell)}\) must be interpreted as a pre-activation variance rather than as an activation variance. At a fixed point \(q_z^\ast = \sigma_w^2 q_a^\ast + \sigma_b^2\), the scalar \[\chi \;=\; \sigma_w^2 \int_{-\infty}^{\infty} \bigl(\sigma'(\sqrt{q_z^\ast}\,u)\bigr)^2 \frac{e^{-u^2/2}}{\sqrt{2\pi}}\,\mathrm{d}u\] controls the linearised propagation of correlations and typical back-propagated signals. If \(\chi < 1\), correlations contract and gradients vanish; if \(\chi > 1\), they expand and gradients explode; if \(\chi = 1\), the network is on the edge of chaos.
This analysis shows that the vanishing (and exploding) gradient problems are typical for random initialisations that ignore the \(\chi = 1\) constraint. The set \(\{(\sigma_w, \sigma_b) : \chi(\sigma_w, \sigma_b) = 1\}\) is a one-dimensional curve in the two-dimensional parameter space: not pathological, but not automatically reached either. Without principled initialisation schemes (which did not exist until the work of [26] and [27]), networks were unlikely to begin in the stable propagation regime, especially as layer depth increased.
6 Generalisation Theory: The Statistical Barriers↩︎
6.1 Uniform Convergence Bounds↩︎
Theorem 14 (Uniform Convergence Bound). [8], [14] Let \(\mathcal{H}\) be a binary hypothesis class with \(\mathop{\mathrm{VCdim}}(\mathcal{H})=d\). Then with probability at least \(1-\delta\) over a sample of size \(m\), \[\sup_{h\in\mathcal{H}} \left\lvert L_{\mathcal{D}}(h)-L_S(h) \right\rvert \;\leq\; C\sqrt{\frac{d\log(em/d)+\log(1/\delta)}{m}}\] for a universal constant \(C>0\).
The point is qualitative rather than constant-sensitive. To make the right-hand side at most \(0.05\), requiring \(C\sqrt{(d\log(em/d)+\log(1/\delta))/m} \leq 0.05\), one needs sample size roughly \(m \gtrsim 3200\,d\) (absorbing log-factors and universal constants). For a network with \(W = 10^4\) weights, using the rough heuristic \(\mathop{\mathrm{VCdim}}\approx W\) [28], this requires \(m \approx 3.2 \times 10^7\) labelled examples — far beyond what was available in the 1980s. The bound is therefore non-explanatory for any practical dataset size of that era.
6.2 The Bias–Variance Tradeoff↩︎
For regression under squared loss, the mean squared error decomposes as: \[\label{eq:bv} \mathop{\mathrm{\mathbb{E}}}\!\left[(\hat{f}(x) - f(x))^2\right] \;=\; \underbrace{\bigl(\mathop{\mathrm{\mathbb{E}}}[\hat{f}(x)] - f(x)\bigr)^2}_{\text{Bias}^2(\hat{f})} + \underbrace{\mathop{\mathrm{Var}}[\hat{f}(x)]}_{\text{Variance}(\hat{f})} + \underbrace{\sigma_\varepsilon^2}_{\text{Irreducible noise}}.\tag{6}\]
In Sobolev-type regularisation settings one often writes the complexity trade-off in the stylised form
\(\text{Bias}^2 \asymp \lambda^{2s}\),
\(\text{Variance} \asymp d_\mathrm{eff}(\lambda)/m\),
where \(\lambda\) is a regularisation scale, \(s\) is a smoothness index, and \(d_\mathrm{eff}(\lambda) = \mathop{\mathrm{tr}}\!\bigl(\mathbf{H}(\mathbf{H} + \lambda\mathbf{I})^{-1}\bigr)\) denotes the effective dimension associated with the kernel or sample-covariance matrix \(\mathbf{H}\). The optimal complexity balances \[\lambda^* = \mathop{\mathrm{arg\,min}}_\lambda \left[\lambda^{2s} + \frac{d_\mathrm{eff}(\lambda)}{m}\right].\] For neural networks the analogue of \(\lambda\) may arise through explicit weight decay, architectural restriction, or implicit mechanisms such as early stopping. The mapping is therefore heuristic rather than literal. Even so, the lesson remains: when effective model flexibility is large relative to sample size, variance dominates and out-of-sample performance becomes unstable.
6.3 Rademacher Complexity↩︎
A tighter modern tool is Rademacher complexity.
Definition 6 (Rademacher Complexity). The empirical Rademacher complexity of \(\mathcal{H}\) with respect to sample \(S = \{x_i\}_{i=1}^m\) is \[\hat{\mathfrak{R}}_S(\mathcal{H}) \;=\; \mathop{\mathrm{\mathbb{E}}}_{\boldsymbol{\sigma}}\!\left[ \sup_{h \in \mathcal{H}} \frac{1}{m}\sum_{i=1}^m \sigma_i h(x_i) \right],\] where \(\boldsymbol{\sigma} = (\sigma_1,\ldots,\sigma_m)\) are i.i.d.Rademacher variables (\(\mathbb{P}[\sigma_i = \pm 1] = 1/2\)).
Theorem 15 (Rademacher Generalisation Bound). With probability at least \(1-\delta\): \[L_{\mathcal{D}}(h) \;\leq\; L_S(h) + 2\hat{\mathfrak{R}}_S(\mathcal{H}) + \sqrt{\frac{\log(2/\delta)}{2m}}.\]
For a single-hidden-layer network with \(k\) hidden units, weight matrix \(\mathbf{W}^{(1)} \in \mathbb{R}^{k \times d}\) with \(\left\lVert \mathbf{W}^{(1)} \right\rVert_F \leq B_1\), output weights \(\mathbf{w}^{(2)} \in \mathbb{R}^k\) with \(\left\lVert \mathbf{w}^{(2)} \right\rVert_1 \leq B_2\), inputs satisfying \(\left\lVert \mathbf{x} \right\rVert_2 \leq X\) almost surely, and a \(1\)-Lipschitz activation \(\sigma\) with \(\sigma(0)=0\), the Ledoux–Talagrand contraction inequality [29] yields \[\hat{\mathfrak{R}}_S(\mathcal{H}) \;\leq\; \frac{B_1 B_2 X}{\sqrt{m}}.\] The \(1/\sqrt{k}\) factor sometimes quoted in informal presentations is not a general consequence of this setup. The reason \(k\) disappears from the bound is that the contraction inequality absorbs it: writing the network output as \(f(\mathbf{x}) = \mathbf{w}^{(2)\top}\sigma(\mathbf{W}^{(1)}\mathbf{x})\), one applies contraction to the \(\ell_1\)-weighted sum over hidden units, which yields a factor \(\left\lVert \mathbf{w}^{(2)} \right\rVert_1 \leq B_2\) regardless of \(k\). Increasing \(k\) while keeping \(B_1\) and \(B_2\) fixed therefore does not worsen the bound, but at the cost that those norm constraints become harder to satisfy in practice. Sharper bounds incorporating activation geometry and width may yield \(k\)-dependent factors, but these are not universal consequences of the setup above. For multilayer networks, the analogous spectral-norm-based bounds [30] accumulate one spectral-norm factor per layer, so the bound grows multiplicatively with depth — making classical generalisation guarantees essentially vacuous for deep networks unless weight norms are tightly controlled.
7 Resolutions: Mathematical Breakthroughs That Ended the Winters↩︎
The modern era is often associated with the deep-learning synthesis catalogued by [31] and [32], in which architectural innovations, algorithmic refinements, and the availability of large labelled corpora combine to push performance past the regimes in which the barriers of Sections 3–6 bind decisively. We organise the underlying mathematical resolutions by the barrier each one addresses.
7.1 Universal Approximation and Depth Separation↩︎
The first key insight was that sufficiently wide networks overcome the representational barrier.
Theorem 16 (Universal Approximation). [33]–[35] Let \(\sigma:\mathbb{R}\to\mathbb{R}\) be continuous. Single-hidden-layer networks with activation \(\sigma\) are universal approximators on compact subsets of \(\mathbb{R}^d\) — meaning that for every \(f \in C([0,1]^d)\) and \(\varepsilon > 0\) there exist \(N \in \mathbb{N}\), coefficients \(\alpha_i \in \mathbb{R}\), biases \(b_i \in \mathbb{R}\), and weights \(\mathbf{w}_i \in \mathbb{R}^d\) with \[\sup_{\mathbf{x} \in [0,1]^d} \left\lvert f(\mathbf{x}) - \sum_{i=1}^N \alpha_i\,\sigma\!\left(\langle \mathbf{w}_i,\,\mathbf{x} \rangle + b_i\right)\right\rvert \;<\; \varepsilon\] — if and only if \(\sigma\) is not a polynomial. The sufficiency results of [33] (for sigmoidal \(\sigma\)) and [34] (for bounded non-constant \(\sigma\)) were sharpened to this necessary-and-sufficient form by [35].
Universal approximation is therefore a statement about expressivity in principle, not about efficient representation. A shallow network may approximate any continuous function on a compact domain, yet require enormous width to do so. The deeper insight is that depth can change representational efficiency dramatically, a fact foreshadowed by [36] for sum–product networks and made precise for ReLU networks by the depth-separation results of the mid-2010s.
Proposition 17 (Universal Approximation versus Efficient Representation). Universal approximation and depth separation are not in tension. The former says that shallow networks are dense in suitable function spaces; the latter says that for some function families, achieving a fixed approximation error with a shallow network may require exponential width, whereas a deeper network achieves the same error with polynomial size. The issue is therefore not expressivity in principle, but representational efficiency.
Theorem 18 (Depth Separation for ReLU Networks). [37], [38]
(Telgarsky, 2016.) For any \(k \in \mathbb{N}\), there exists a function computable by a ReLU network of depth \(3k\) and \(\mathcal{O}(1)\) width that cannot be approximated to constant error by any network of depth \(k\) and sub-exponential width \(2^{k^{1/3}}\).
(Eldan & Shamir, 2016.) There exists a radial function \(f:\mathbb{R}^d \to \mathbb{R}\) that is efficiently computable by a 3-layer network of polynomial size but requires exponential size (in \(d\)) to approximate to constant error by any 2-layer network.
Remark 19 (Linear-Region Counting). [39] An earlier and complementary expressivity argument is the linear-region count of [39], which predates the approximation-theoretic separations above. They showed that depth-\(k\) ReLU networks of width \(n\) can produce at least \[\Omega\!\left(\left(\lfloor n/d \rfloor\right)^{(k-1)d} \cdot n\right)\] linear regions — exponentially more than depth-2 networks of the same width. This expressivity gap is consistent with, but does not directly imply, the approximation-theoretic separations of Theorem 18, which concern approximation error for specific function families.
7.2 The Representer Theorem and Kernel Methods↩︎
The 1990s saw the emergence of kernel methods [40], [41], which circumvent intractable weight-space search by recasting optimisation in a reproducing kernel Hilbert space (RKHS).
Theorem 20 (Representer Theorem). [42] Let \(\mathcal{H}_k\) be an RKHS with kernel \(k\). For any loss function \(\ell\) and regularisation \(\Omega(\left\lVert f \right\rVert_{\mathcal{H}_k})\) with \(\Omega\) strictly monotone increasing, the solution to \[\min_{f \in \mathcal{H}_k} \;\frac{1}{m}\sum_{i=1}^m \ell(y_i, f(x_i)) + \Omega(\left\lVert f \right\rVert_{\mathcal{H}_k})\] takes the form \(f^*(\cdot) = \sum_{i=1}^m \alpha_i k(x_i, \cdot)\).
Proof. Decompose any \(f \in \mathcal{H}_k\) as \(f = f_\| + f_\perp\) where \(f_\| \in \text{span}\{k(x_i,\cdot)\}_{i=1}^m\) and \(f_\perp\) is orthogonal to this span. Then \(f(x_i) = \langle f,\,k(x_i,\cdot) \rangle_{\mathcal{H}_k} = f_\|(x_i)\) for all \(i\), so \(f_\perp\) affects the regularisation term only (since \(\left\lVert f \right\rVert^2 = \left\lVert f_\| \right\rVert^2 + \left\lVert f_\perp \right\rVert^2 \geq \left\lVert f_\| \right\rVert^2\)). Setting \(f_\perp = 0\) therefore cannot increase the loss while it strictly decreases the regularisation, contradicting optimality of any \(f_\perp \neq 0\). ◻
The kernel trick \(k(\mathbf{x},\mathbf{z}) = \phi(\mathbf{x})^\top\phi(\mathbf{z})\) allows implicit computation of dot products in potentially infinite-dimensional feature spaces, without explicitly computing \(\phi\). For the Gaussian RBF: \(k(\mathbf{x},\mathbf{z}) = \exp(-\left\lVert \mathbf{x}-\mathbf{z} \right\rVert^2/2\sigma^2)\), the corresponding feature map is infinite-dimensional.
Remark 21 (The Neural Tangent Kernel). [43] A remarkable theoretical bridge between kernel methods and modern deep learning is the Neural Tangent Kernel (NTK). For a network \(f_{\boldsymbol{\theta}}\) with parameter vector \(\boldsymbol{\theta}\), define \[K_{\mathrm{NTK}}(\mathbf{x}, \mathbf{z}) \;=\; \left\langle \frac{\partial f_{\boldsymbol{\theta}}(\mathbf{x})}{\partial \boldsymbol{\theta}},\; \frac{\partial f_{\boldsymbol{\theta}}(\mathbf{z})}{\partial \boldsymbol{\theta}} \right\rangle.\] [43] showed that in the infinite-width limit, \(K_{\mathrm{NTK}}\) converges to a deterministic kernel at initialisation and remains constant throughout gradient descent training. In this limit, training the network is equivalent to kernel regression with \(K_{\mathrm{NTK}}\), providing an exact RKHS interpretation of infinite-width neural networks. The NTK therefore closes a conceptual loop: the representational barriers of the perceptron era (Section 3) are circumvented by depth and nonlinearity, and the resulting function class admits a kernel description in the lazy-training regime. Whether practical finite-width networks operate near this kernel regime remains an active research question.
7.3 Optimisation: Activations, Initialisation, and Residual Connections↩︎
The vanishing gradient problem was resolved by a combination of:
7.3.0.1 (i) ReLU activations.
The rectified linear unit \(\sigma(z)=\max\{0,z\}\) avoids the small-derivative saturation that makes sigmoid and \(\tanh\) networks especially vulnerable to gradient decay in their tails. This does not mean that gradient gating disappears: \(\sigma'(z)=\mathbf{1}_{\{z>0\}}\), so inactive units can still block gradient flow. The practical gain is that active units contribute a derivative of order one rather than an exponentially small factor, making stable signal propagation compatible with variance-preserving initialisation. Combined with the He scheme [27], the typical spectral factor \(\rho\kappa\) entering Theorem 12 is brought close to unity, so gradients propagate without exponential decay across realistic depths.
7.3.0.2 (ii) He/Xavier Initialisation [26], [27].
In the scaling \(W_{ij}^{(\ell)}\sim\mathcal{N}(0,\sigma_w^2/n)\) used above, He initialisation for ReLU corresponds to \(\sigma_w^2 = 2\) (so that \(\mathrm{Var}(W_{ij})=2/n\) with \(n\) the fan-in), which places the network at the edge of chaos \(\chi = 1\) at initialisation and yields well-conditioned gradient flow. Xavier initialisation corresponds to \(\sigma_w^2 = 1\) and is the analogous variance-preserving choice for \(\tanh\) networks.
7.3.0.3 (iii) Residual connections [44].
Adding skip connections \(\mathbf{a}^{(\ell+1)} = \sigma(\mathbf{W}^{(\ell)}\mathbf{a}^{(\ell)}) + \mathbf{a}^{(\ell)}\) modifies the per-layer Jacobian to \[\mathbf{J}^{(\ell)}_{\mathrm{res}} \;=\; \frac{\partial \mathbf{a}^{(\ell+1)}}{\partial \mathbf{a}^{(\ell)}} \;=\; \mathbf{I} + \operatorname{diag}(\sigma'(\mathbf{z}^{(\ell)}))\mathbf{W}^{(\ell)}.\] The presence of \(\mathbf{I}\) does not guarantee a unit lower bound on singular values in general: if \(\mathbf{A} = \operatorname{diag}(\sigma')\mathbf{W}\) has a singular value close to \(-1\) (possible when \(\mathbf{W}\) has a large negative eigencomponent), then \(\mathbf{I} + \mathbf{A}\) may have singular values well below \(1\). A simple scalar instance: \(A = -\tfrac{1}{2}\) gives \(I + A = \tfrac{1}{2}\). The practical benefit of residuals is more subtle: they create additive gradient paths through the network — the gradient of the loss with respect to early-layer parameters receives a direct contribution \(\partial \mathcal{L}/\partial \mathbf{a}^{(L)}\) bypassing all intermediate weight matrices. This bypassed path is not subject to the multiplicative decay in 5 , and empirically dominates when intermediate layers are small or near-zero, maintaining meaningful gradient signal even at extreme depth [44].
7.4 The Double Descent Phenomenon↩︎
A later development, important for retrospective interpretation though not a direct historical cause of the winters themselves, is the double-descent phenomenon. It helps explain why classical statistical intuition was incomplete in highly overparameterised regimes.
Theorem 22 (Asymptotic Risk in Overparameterised Regression). [45] Consider the minimum-norm least-squares interpolant \(\hat{\mathbf{w}} = \mathbf{X}^\dagger\mathbf{y}\) for design matrix \(\mathbf{X} \in \mathbb{R}^{m \times p}\) with i.i.d.isotropic Gaussian rows, and labels \(\mathbf{y} = \mathbf{X}\mathbf{w}^* + \boldsymbol{\varepsilon}\), \(\varepsilon_i \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,\sigma^2)\). Let \(\gamma = p/m\) (the overparameterisation ratio). As \(m, p \to \infty\) with \(\gamma > 1\) fixed, the test risk decomposes as \[\mathop{\mathrm{\mathbb{E}}}\!\left[L_\mathcal{D}(\hat{f})\right] \;\to\; \underbrace{\left\lVert \mathbf{w}^* \right\rVert^2\!\left(1 - \frac{1}{\gamma}\right)}_{\text{Bias}^2} \;+\; \underbrace{\sigma^2 \cdot \frac{1}{\gamma - 1}}_{\text{Variance}}.\] As \(\gamma \to \infty\), the variance term \(\sigma^2/(\gamma-1)\) vanishes, but the bias term converges to \(\left\lVert \mathbf{w}^* \right\rVert^2\) in general. Thus the test risk does not converge to zero in the isotropic Gaussian model unless the signal itself shrinks in a way that makes the bias vanish.
Remark 23 (Interpreting Double Descent). The original empirical observation of [46] — that test risk decreases again after the interpolation threshold \(p = m\) — is robust and important. The precise asymptotic formula above shows that the behaviour in the overparameterised regime depends on both the signal strength \(\left\lVert \mathbf{w}^* \right\rVert\) and the noise \(\sigma^2\), not on noise alone. The practical success of overparameterised networks is better understood through the lens of implicit regularisation: gradient descent preferentially selects low-complexity interpolating solutions in many settings [29], [30], [47]–[49]. The key conceptual point stands: classical bias–variance intuition, which treats overparameterisation as uniformly harmful, is incomplete.
7.5 Modern Theory: Four Post-Hoc Lenses on the Barrier Triple↩︎
7.5.0.1 Methodological note.
The four items below are epistemically heterogeneous. Neural scaling laws and neural collapse are primarily empirical regularities with partial theoretical explanation; the lottery ticket hypothesis is, as its name indicates, a hypothesis supported by substantial experiments rather than a general theorem; implicit-bias theory contains theorem-level results, but mostly in restricted model classes. They are grouped here not as coequal formal results, but as post-hoc explanatory lenses on different coordinates of the barrier triple.
The four theoretical developments below are not themselves the historical interventions that ended the winters; those were, more prosaically, multilayer architectures, non-saturating activations, improved initialisation, large labelled corpora, and hardware scaling. The developments discussed here are later theoretical lenses on the modern era: each illuminates a specific component of the barrier triple \((R, C, S)\) from Definition 1 and helps explain, after the fact, why contemporary large-scale systems tend to sit outside the joint-binding regime of Proposition 28.
Neural Scaling Laws (S-barrier)↩︎
[50] demonstrated that the test loss of large language models follows precise power-law relationships in model size \(N\), dataset size \(D\), and compute budget \(C_{\mathrm{flop}}\): \[L(N,D) \;\approx\; \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty,\] with empirically fitted exponents \(\alpha_N \approx 0.076\), \(\alpha_D \approx 0.095\), and irreducible loss \(L_\infty\). [51] refined this picture with a compute-optimal (Chinchilla) result: for a fixed FLOP budget \(C_{\mathrm{flop}}\), the optimal allocation satisfies \(N^* \propto C_{\mathrm{flop}}^{a}\) and \(D^* \propto C_{\mathrm{flop}}^{b}\) with empirically estimated exponents \(a, b \approx 0.5\) (point estimates fall in the range \(0.46\)–\(0.54\) depending on the fitting approach), i.e.model size and data should scale in roughly equal proportion.
Remark 24 (Scaling Laws and the Statistical Barrier). The classical statistical barrier \(S\) of Definition 1 asserts that finite-sample guarantees require unrealistic data volumes. Scaling laws give this assertion a quantitative inversion: they specify how many samples and parameters are needed to reach a target loss, under the assumption that both are available. They do not eliminate the barrier — the Stone minimax rate (Theorem 9) still governs pointwise estimation — but they show that the barrier is metrically navigable: smooth power-law curves, rather than hard walls, control performance. This is precisely the shift that distinguishes the modern era from the second winter, when VC-style bounds gave only vacuous guarantees with no actionable guidance on scale.
Neural Collapse (R-barrier)↩︎
[52] identified a striking geometric phenomenon in the terminal phase of deep-network training. As the loss approaches zero, the within-class variability of last-layer features \(\mathbf{h}_i^{(c)}\) collapses: \[\frac{1}{n_c}\sum_{i=1}^{n_c}\left\lVert \mathbf{h}_i^{(c)} - \boldsymbol{\mu}_c \right\rVert^2 \;\to\; 0,\] and the class means \(\boldsymbol{\mu}_1,\dots,\boldsymbol{\mu}_K\) converge to a simplex equiangular tight frame (ETF): \[\cos\angle(\boldsymbol{\mu}_c - \boldsymbol{\mu}_0,\; \boldsymbol{\mu}_{c'} - \boldsymbol{\mu}_0) = -\frac{1}{K-1}, \quad c \neq c',\] the maximally spread configuration in \(\mathbb{R}^d\) for \(K\) unit vectors. Simultaneously, the classifier weight vectors \(\mathbf{w}_c\) align with the class means.
Remark 25 (Neural Collapse and the Representational Barrier). The representational barrier \(R\) of Definition 1 captures the inability of a model class to realise target functions. Neural collapse implies that deep networks do more than find some representation: they converge to a geometry-optimal one — the ETF — that maximises inter-class separation given the ambient dimension. This offers a partial geometric lens on the puzzle of generalisation in deep networks: the terminal-phase representation is structured rather than arbitrary, and under symmetric Gaussian class-conditional models the ETF geometry coincides with that of Bayes-optimal classifiers. The evidence base is, however, largely empirical and restricted to the final training phase; neural collapse should be read as a suggestive regularity that aligns with the representational component \(R\) of Definition 1, not as a derivation of generalisation from first principles.
The Lottery Ticket Hypothesis (C-barrier)↩︎
[53] proposed the lottery ticket hypothesis: a randomly initialised dense network \(f(\mathbf{x};\boldsymbol{\theta})\) of size \(n\) contains a sparse subnetwork (a “winning ticket”) \(f(\mathbf{x};\boldsymbol{\theta}_0 \odot \mathbf{m})\), where \(\mathbf{m}\in\{0,1\}^n\) is a binary mask, such that when trained from the original initialisation \(\boldsymbol{\theta}_0 \odot \mathbf{m}\) (not from a fresh random start), it matches the full network’s accuracy in at most the same number of iterations. Formally, with target accuracy \(a^\ast\) and iteration budget \(j^\ast\) for the full network: \[\exists\,\mathbf{m}:\;|\mathbf{m}|_1 \ll n,\quad \text{accuracy}\bigl(f(\cdot;\boldsymbol{\theta}_{j^\ast}\odot\mathbf{m})\bigr) \;\ge\; a^\ast.\]
Remark 26 (Lottery Tickets and the Computational Barrier). The computational barrier \(C\) of Definition 1 rests on the worst-case intractability of finding correct weights (Theorem 7). The lottery ticket hypothesis — still a hypothesis, supported by substantial but not universal empirical evidence and sensitive to architecture, dataset, and pruning protocol — does not contradict this hardness result: worst-case intractability survives. What it suggests, at the level of an empirical regularity, is that in the regimes where winning tickets have been observed, useful sparse structure is already latent in the dense initialisation, so that practical training resembles identification of a pre-existing subnetwork rather than search through an exponential space from scratch. Read this way, lottery tickets sit alongside the resolutions of Section 7 as a post-hoc lens on why the \(C\)-barrier does not bind in practice for the architectures studied, rather than as a theorem about trainability in general.
Implicit Bias Theory (S-barrier)↩︎
[54] proved the first rigorous result on the implicit bias of gradient descent: for linearly separable data, gradient descent on the logistic loss with any step-size schedule satisfying \(\sum_t \eta_t = \infty\), \(\sum_t \eta_t^2 < \infty\) converges in direction to the hard-margin SVM solution — the maximum-margin classifier \(\hat{\mathbf{w}}/\left\lVert \hat{\mathbf{w}} \right\rVert\): \[\frac{\mathbf{w}(t)}{\left\lVert \mathbf{w}(t) \right\rVert} \;\to\; \hat{\mathbf{w}}_{\mathrm{SVM}} \quad \text{as } t \to \infty.\] [49] extended this analysis to matrix factorisation and linear convolutional networks, showing that gradient descent imposes a nuclear-norm minimisation bias in the matrix case and a structured frequency-domain sparsity bias in the convolutional case.
Remark 27 (Implicit Bias and the Statistical Barrier). The statistical barrier \(S\) of Definition 1 is most acute because classical bounds (VC, Rademacher) apply to the worst-case hypothesis in a class, while gradient descent does not explore the class uniformly. The implicit bias results of [54] and [49] make this precise: the algorithm itself performs implicit regularisation, converging to minimum-complexity solutions (maximum margin, minimum nuclear norm) that happen to generalise well. This bridges the gap between the vacuous worst-case statistical bounds of Section 6 and the empirical success of unregularised gradient descent on overparameterised models. The statistical barrier is not eliminated — it is circumvented by the algorithm’s inductive bias, which provides an effective regulariser that classical theory did not anticipate.
7.5.0.2 Connection to the barrier model.
Each of the four developments above offers a post-hoc lens on a specific component of the barrier triple from Definition 1: scaling laws quantify the regime in which the statistical barrier \(S\) becomes metrically navigable; neural collapse describes a geometric structure that partially accounts for why the representational barrier \(R\) does not bite in practice; lottery tickets reframe the computational barrier \(C\) from exhaustive search to latent structure identification; and implicit bias theory shows how the statistical barrier \(S\) can be circumvented by the algorithm’s inductive geometry rather than by tighter worst-case bounds. Proposition 28 characterises joint binding of \((R, C, S)\) as the structural signature of paradigm fragility; these four lenses provide complementary explanations for why the modern era, shaped primarily by architectural and hardware progress, appears to avoid that regime.
8 A Mathematical Taxonomy of AI Winter Causes↩︎
8.1 A Structural Synthesis↩︎
We now state explicitly the structural logic implicit in the previous sections, unifying Definition 1 with the barriers established in Sections 3–6.
Proposition 28 (Joint-Binding Condition for Structural Fragility). Let \(\mathcal{A}\) be an architecture class, \(\mathcal{T}\) a family of tasks equipped with a task distribution of interest, and fix a data budget \(n\) and compute budget \(\kappa\). Suppose the following three conditions hold for a target error \(\varepsilon>0\):
Representational limitation: there exists a subset \(\mathcal{T}_R \subseteq \mathcal{T}\) of positive measure whose target functions are not \(\varepsilon\)-approximable by any element of \(\mathcal{A}\);
Computational hardness: locating an \(\varepsilon\)-optimal hypothesis in \(\mathcal{A}\) is worst-case intractable, or its cost grows super-polynomially in the relevant problem parameters, so that no algorithm respecting budget \(\kappa\) is known to succeed reliably;
Statistical fragility: the finite-sample guarantees available for \(\mathcal{A}\) require a sample size that exceeds \(n\) by a super-constant factor at the scales of interest.
If at least two of these conditions bind simultaneously under the budget \((n,\kappa)\), then any learning system based on \(\mathcal{A}\) must exhibit at least one of the following pathologies on \(\mathcal{T}\):
irreducible approximation error on a positive-measure subset,
computational infeasibility of reliable training within budget \(\kappa\),
generalisation guarantees that remain vacuous at sample size \(n\).
We call any paradigm satisfying the hypotheses structurally fragile with respect to the budget \((\mathcal{A},\mathcal{T},n,\kappa)\).
Synthesis. The statement reorganises the preceding sections rather than establishing a new lower bound. If (i) binds, approximation error is bounded away from zero on \(\mathcal{T}_R\) regardless of optimisation or data (Section 3). If (ii) binds, expressive hypotheses may exist in principle but cannot be located reliably within budget \(\kappa\) (Section 4). If (iii) binds, empirical fit does not transfer into controlled generalisation at sample size \(n\) (Section 6). When any two bind jointly, the paradigm has no simultaneous route to expressivity, trainability, and sample-efficient generalisation within the budget, so at least one of (a)–(c) must occur. ◻
Remark 29 (Framework status). Proposition 28 should be read as a budgeted synthesis rather than as a new impossibility theorem. Its force is logical: once a paradigm requires simultaneous expressivity, trainability, and non-vacuous finite-sample control, the joint failure of any two of these conditions under the same resource budget is enough to render the paradigm structurally unstable. The proposition therefore packages the earlier obstructions into a single fragility criterion instead of asserting a fresh lower bound. It makes no claim about the historical sufficiency of these mathematical conditions: economic, institutional, and engineering factors lie outside its scope. Structural fragility is a necessary condition for the kind of paradigm collapse associated with historical AI winters, not a sufficient one.
8.2 Barrier Diagram↩︎
8.3 Comparative Taxonomy↩︎
Table 1 summarises the main barriers discussed in the article. The final column refers to later mitigations rather than complete eliminations. In each case, the point is not that a single theorem “caused” an historical winter, but that the dominant methods of the time were exposed to a structurally important formal bottleneck.
3pt 3em
| Barrier | Primary relevance | Formal object | Mathematical implication | Later mitigation |
|---|---|---|---|---|
| Representational | First winter | Single-layer threshold classifiers | Non-linearly separable functions (e.g.XOR) are not representable | Multilayer architectures; universal approximation; depth separation |
| Computational | Mostly first; partly both | Exact training of threshold networks; symbolic search trees | Worst-case hardness and exponential search growth | Stochastic optimisation; heuristics; hardware scale; approximate learning |
| Statistical | Mostly second; also retrospective | VC dimension and uniform convergence | Classical bounds become loose or vacuous in flexible regimes | Margin methods; norm control; implicit regularisation; interpolation theory |
| Optimisation | Second winter | Products of Jacobians in deep networks | Exponential decay or explosion of gradients under saturating dynamics | ReLU activations; careful initialisation; residual connections |
| Dimensionality | Both | Minimax rates for nonparametric estimation | Sample complexity deteriorates rapidly with ambient dimension | Learned hierarchical representations; task structure; invariances |
| Symbolic knowledge | Second winter | Rule bases and persistence constraints | Brittleness and combinatorial maintenance burden | Representation learning and data-driven feature formation |
9 Conclusion↩︎
This article has presented a mathematical interpretation of AI winter that is narrower and more defensible than a monocausal historical thesis: not that specific theorems caused historical retrenchment, but that the leading paradigms of the relevant eras were already structurally exposed before external shocks arrived. The central claim — formalised in Definition 1 and Proposition 28 — is that several of the most visible disappointments of early AI aligned with genuine formal bottlenecks. Single-layer perceptrons were representationally limited. Exact training of small threshold networks was already worst-case hard. Symbolic search suffered exponential growth. High-dimensional smooth-function estimation obeyed unfavourable minimax rates. Deep gradient propagation under saturating nonlinearities was unstable. Classical finite-sample generalisation theory, meanwhile, did not explain why large flexible models should perform well in practice. None of these facts by itself determines an historical outcome, but together they help explain why major paradigms proved brittle when ambitious claims met finite compute, limited data, and real engineering constraints. That is the precise sense in which formal barriers contributed to winter: not as sufficient causes, but as mathematically articulable sources of fragility.
The second lesson is equally important. The end of an AI winter did not come from the disappearance of these barriers, but from the discovery of architectures and mathematical viewpoints that mitigated them. Multilayer networks overcame the strict linear-separability bottleneck. Kernel and margin methods supplied new forms of statistical control. Improved initialisation, non-saturating activations, and residual architectures altered optimisation geometry. Modern representation learning exploited compositional structure and invariances to soften the curse of dimensionality in practice. Recent theory on interpolation and implicit regularisation has further clarified why overparameterisation need not be fatal, even if many foundational questions remain open.
The broader implication is methodological. AI advances become durable when claims of intelligence are matched by explicit analysis of representational, computational, statistical, and optimisation constraints. The history of AI winter is therefore not only a cautionary tale about hype but also a reminder that progress in artificial intelligence depends on mathematics as much as on engineering.
9.0.0.1 Limitations.
The article has focused on formal bottlenecks and has not attempted a complete intellectual history of the two winters. It has also treated later developments mainly through theorem-level lenses, leaving aside many empirical and institutional factors. The account should therefore be read as a mathematical anatomy of fragility, not as a comprehensive explanation of every historical event.
References↩︎
The \(\log(1/\varepsilon)\) gap between the classical upper and lower bounds was closed by [15], who established the tight rate \(m^{\mathrm{real}}_{\mathcal{H}}(\varepsilon,\delta) = \Theta((d + \log(1/\delta))/\varepsilon)\) via a one-inclusion-graph majority-vote algorithm. The classical bounds above remain the historically relevant statement for the period covered in this article.↩︎
The same exponent \(-2s/(2s+d)\) extends to Sobolev classes \(\mathcal{W}^{s,2}([0,1]^d)\) via wavelet-projection arguments [20], and more generally to Besov classes; the underlying obstruction — the curse of dimensionality — is identical across these scales.↩︎