Theory of the Frequency Principle for General Deep Neural Networks
June 21, 2019
Abstract
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.
1 Introduction↩︎
Deep learning has achieved great success as in many fields [1], e.g., speech recognition [2], object recognition [3], natural language processing [4] and computer game control [5]. It has also been adopted into algorithms to solve scientific computing problems [6]–[9]. In principle, the universal approximation theorem states that a commonly-used Deep Neural Network (DNN) of sufficiently large width can approximate any function to a desired precision [10]. However, it remains a mystery that how a DNN finds a minimum corresponding to such an approximation through the gradient-based training process. To understand the learning behavior of DNNs for the approximation problem, recent works model the gradient flow of parameters in a two-layer ReLU neural networks by a partial differential equation (PDE) in the mean-field limit [11]–[13]. However, it is not clear whether this PDE approach, which describes a neural network of one hidden layer of infinite width, can be extended to general DNNs of multiple hidden layers and limited neuron number.
In this work, we take another approach that uses Fourier analysis to study the learning behavior of DNNs based on the phenomenon of Frequency Principle (F-Principle), i.e., a DNN tends to learn a target function from low to high frequencies during the training [14]–[19]. Empirically, the F-Principle can be widely observed in general DNNs for both benchmark and synthetic data [14], [18]. Conceptually, it provides a qualitative explanation of the success and failure of DNNs [18]. Based on the F-Principle, a series of works has been done. For example, it is used as an important phenomenon to pursue fundamentally different learning trajectories of meta-learning [20]. It is also used as a tool to observe the performance of adaptive activation function [21]. Based on the F-Principle, a numerical algorithm is developed to accelerate the DNN fitting of high frequency functions by shifting high frequencies to lower ones [22]. Theoretically, an effective model of linear F-Principle dynamics [19], which accurately predicts the learning results of two-layer ReLU neural networks of large widths, leads to an apriori estimate of the generalization bound. In addition, a theorem is provided for the characterization of the initial training stage of a two-layer \(\tanh\) network [18]. The same theoretical analysis in [18] is also adopted in the analysis of DNNs with ReLU activation function [15] and a nonlinear collaborative scheme of loss functions for DNN training [23]. These subsequent works show the importance of the F-Principle. However, a theory of the F-Principle for general DNNs is still missing.
Following the same direction as in [18], in this work, we propose a theoretical framework of Fourier analysis for the study of the training behavior of general DNNs in the following three stages: the initial stage, the intermediate stage, and the final stage. At all stages, we rigorously characterize the F-Principle by estimating some proper quantities. At the initial and final stages with the MSE loss (mean-squared error, also known as \(L^{2}\) loss), we show that the change of MSE is dominated by low frequencies. Furthermore, in these two stages with general \(L^{p}\) (\(2\leq p<\infty\)) loss, we show that the change of the DNN output is dominated by the low-frequency part. A key contribution of this work is on the intermediate stage — with \(L^{p}\) loss, the difference of the MSE over a certain period, in which the MSE is reduced by half, is dominated by the low frequencies. In summary, we verify that the F-Principle is universal in the sense that our results not only work for DNNs of multiple layers with any commonly-used activation function, e.g., ReLU, sigmoid, and tanh, but also work for a general population density of data and for a general class of loss functions. The key insight unraveled by our analysis is that the regularity of DNN converts into the decay rate of a loss function in the frequency domain.
2 Preliminaries↩︎
We start with a brief introduction to DNNs and its training dynamics. Under very mild assumptions, we provide some regularity results which are crucial to the proof of the main theorems summarized in the next section.
2.1 Deep Neural Networks↩︎
Consider a DNN with \((H-1)\)-hidden layers and general activation functions. We regard the \(d\)-dimensional input as the \(0\)-th layer and the one-dimensional output as the \(H\)-th layer. Let \(n_l\) be the number of neurons in the \(l\)-th layer. In particular, \(n_0=d\) and \(n_H=1\).
The hypothesis space \(\mathcal{H}\) is a family of hypothesis functions parametrized by the parameter vector \(\theta\in\mathbb{R}^N\) whose entries are called parameters \(W^{(l)}_i\)’s (also known as weight) and \(b^{(l)}_i\)’s (also known as bias). More precisely, we set \[\begin{align} \theta=\Big(W^{(l)},b^{(l)}\Big)_{l=1}^{H}, \end{align}\] where for \(l=1,\cdots,H\), \[\begin{align} W^{(l)} &= \Big(W^{(l)}_i\Big)_{i=1}^{n_l},\quad W^{(l)}_i\in\mathbb{R}^{n_{l-1}}\\ b^{(l)} &= \Big(b^{(l)}_i\Big)_{i=1}^{n_l},\quad b^{(l)}\in\mathbb{R}. \end{align}\] The size \(N\) of the network is the number of the parameters, i.e., \[N=\sum_{l=0}^{H-1}(n_l+1) n_{l+1}.\label{eq4646size}\tag{1}\]
To define the hypothesis functions in \(\mathcal{H}\), we need some nonlinear functions which are known as activation functions: \[\sigma_i^{(l)}:\mathbb{R}\to\mathbb{R},\quad l=1,\cdots,H-1,\quad i=1,\cdots,n_l.\] Given \(\theta\in \mathbb{R}^N\), the corresponding function \(h\) in \(\mathcal{H}\) is defined by a series of function compositions. First, we set \(h^{(0)}=\mathrm{id}:\mathbb{R}^{d}\to\mathbb{R}^{d}\), i.e., \(h^{(0)}(x)=x\) for all \(x\in\mathbb{R}^d\). Then for \(l=1,\cdots,H-1\), \(h^{(l)}\) is defined recursively as \[\begin{align} &h^{(l)}:\mathbb{R}^{d}\to\mathbb{R}^{n_l},\\ &(h^{(l)}(x))_i=\sigma_{i}^{(l)} (W_i^{(l)}\cdot h^{(l-1)}(x)+b_i^{(l)}),\quad i=1,\cdots,n_l. \end{align}\] Finally, we denote \[h^{(H)}(x)=W^{(H)}\cdot h^{(H-1)}(x)+b^{(H)}.\] We remark that for the most applications, the activation functions \(\sigma_i^{(l)}\) are chosen to be the same, i.e., \(\sigma_i^{(l)}=\sigma\), \(l=1,\cdots,H-1\), \(i=1,\cdots,n_l\).
Example 1. For instance, if a one-hidden layer neural network is used, then \(H=2\) and the hypothesis function can be written into the following form: \[h^{(2)}(x,\theta)=\sum_{i=1}^{n}w_{i}^{(2)}\sigma(w_{i}^{(1)}\cdot x+b_{i}^{(1)}),\quad w_{i}^{(2)},b_{i}^{(1)}\in{\rm \mathbb{R}}, w_{i}^{(1)}\in\mathbb{R}^d.\] Thus the size of the network \(N=(d+2)n\) which is consistent with 1 .
We are only interested in the target function \(f_\mathrm{target}\) in a compact domain \(\Omega\), i.e., \(\Omega\subset\subset\mathbb{R}^d\). A bump function \(\chi\) is used to truncate both hypothesis and target functions: \[\begin{align} h(x,\theta)&= h^{(H)}(x,\theta)\chi(x),\\ f(x)&= f_\mathrm{target}(x)\chi(x). \end{align}\] In the sequel, we will also refer to \(h\) and \(f\) as the hypothesis and target functions, respectively.
2.2 Loss Function and Training Dynamics↩︎
In this work, we investigate the training dynamics of parameters in DNNs with two cases of loss functions:
(i) The MSE loss function with population measure \(\mu\), i.e., \[L_\rho(\theta)=\int_{\mathbb{R}^d}\lvert h(x,\theta)-f(x)\rvert^2\mathop{}\!\mathrm{d}{\mu}.\] In this case, the training dynamics of \(\theta\) follows the gradient flow: \[\left\{ \begin{array}{l} \dfrac{\mathrm{d}\theta}{\mathrm{d}t}=-\nabla_\theta L_\rho\\ \theta(0)=\theta_0. \end{array} \right.\label{eq4646TrainingL2Loss}\tag{2}\]
(ii) A general loss function with population measure \(\mu\), i.e., \[\tilde{L}_\rho(\theta)=\int_{\mathbb{R}^d}\ell(h(x,\theta)-f(x))\mathop{}\!\mathrm{d}{\mu},\] where the function \(\ell\) satisfies some mild assumptions to be explained later. In this case, the training dynamics of \(\theta\) becomes: \[\left\{ \begin{array}{l} \dfrac{\mathrm{d}\theta}{\mathrm{d}t}=-\nabla_\theta \tilde{L}_\rho\\ \theta(0)=\theta_0. \end{array} \right.\label{eq4646TrainingGeneralLoss}\tag{3}\]
In the case of MSE loss function, we have \[\begin{align} L_\rho &= \int_{\mathbb{R}^d}\lvert h_\rho(x)-f_\rho(x)\rvert^2\mathop{}\!\mathrm{d}{x}\\ &= \int_{\mathbb{R}^d}\lvert\hat{h}_\rho(\xi)-\hat{f}_\rho(\xi)\rvert^2\mathop{}\!\mathrm{d}{\xi}, \end{align}\] where \(\rho\), satisfying \(\mathrm{d}\mu=\rho\mathrm{d}x\), is called the population density and \[h_\rho=h\sqrt{\rho},\quad f_\rho=f\sqrt{\rho}.\] The second equality is due to the Plancherel theorem. Here and in the sequel, we use the following conventions for the Fourier transform and its inverse transform on \(\mathbb{R}^d\): \[\mathcal{F}[g](\xi)=\hat{g}(\xi)=\int_{\mathbb{R}^d}g(x)\mathrm{e}^{-2\pi\mathrm{i}\xi\cdot x}\mathop{}\!\mathrm{d}{x},\quad g(x)=\int_{\mathbb{R}^d}\hat{g}(\xi)\mathrm{e}^{2\pi\mathrm{i}\xi\cdot x}\mathop{}\!\mathrm{d}{\xi}.\] For the convenience of proofs, we denote \[\begin{align} L_\rho(\theta) &= \int_{\mathbb{R}^d}q_\rho(\xi,\theta)\mathop{}\!\mathrm{d}{\xi},\\ q_\rho(\xi,\theta) &= \lvert\hat{h}_\rho(\xi,\theta)-\hat{f}_\rho(\xi)\rvert^2. \end{align}\]
2.3 Assumptions↩︎
The requirements on \(\chi\), \(f\), \(\sigma\), and \(\mu\) are summarized here.
Assumption 1 (regularity). The bump function \(\chi\) satisfies \(\chi(x)=1\), \(x\in\Omega\) and \(\chi(x)=0\), \(x\in\mathbb{R}^d\backslash\Omega'\) for domains \(\Omega\) and \(\Omega'\) with \(\Omega\subset\subset\Omega'\subset\subset\mathbb{R}^d\). There is a positive integer \(k\) (can be \(\infty\)) such that \(f_\mathrm{target}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\), \(\chi\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;[0,+\infty))\) , and \(\sigma_i^{(l)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R};\mathbb{R})\) for \(l=1,\cdots,H-1\), \(i=1,\cdots,n_l\).
Assumption 2 (bounded population density). There exists a function \(\rho\in L^{\infty}(\mathbb{R}^d;[0,+\infty))\) satisfying \(\mathrm{d}\mu=\rho\mathop{}\!\mathrm{d}{x}\).
Example 2. Here we list some commonly-used activation functions:
(1) ReLU (Rectified Linear Unit): \(\mathrm{ReLU}(x)=\max(0,x)\), \(x\in\mathbb{R}\);
(2) tanh (hyperbolic tangent): \(\tanh(x)=\frac{\mathrm{e}^x-\mathrm{e}^{-x}}{\mathrm{e}^x+\mathrm{e}^{-x}}\), \(x\in\mathbb{R}\);
(3) sigmoid function (also known as logistic function): \(S(x)=\frac{1}{1+\mathrm{e}^{-x}}\), \(x\in\mathbb{R}\).
Remark 1. It is also allowed that \(k=\infty\) where the functions \(f\) and \(\sigma_i^{(l)}\) are all \(C^\infty\) by Sobolev embedding inequalities. This case includes \(\tanh\) and \(\mathrm{sigmoid}\) activation functions.
Remark 2. If an activation function is ReLU, then \(k=1\).
Remark 3. For \(x\in\Omega\), we have \(h(x,\theta)-f(x)=h^{(H)}(x,\theta)-f_\mathrm{target}(x)\).
For the training dynamics 2 or 3 , we suppose the parameters are bounded.
Assumption 3 (bounded trajectory). The training dynamics is nontrivial, i.e., \(\theta(t)\not\equiv\mathrm{const}\). There exists a constant \(R>0\) such that \(\sup_{t\geq 0}\lvert\theta(t)\rvert\leq R\) where the parameter vector \(\theta(t)\) is the solution to 2 or 3 .
Remark 4. The bound \(R\) depends on initial parameter \(\theta_0\).
In the case of MSE loss function, we will further take the following assumption.
Assumption 4. The density \(\rho\) satisfies \(\sqrt{\rho}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;[0,+\infty))\).
The general loss function considered in this work satisfies the following assumption.
Assumption 5 (general loss function). The function \(\ell\) in the general loss function \(\tilde{L}_\rho(\theta)\) satisfies \(\ell\in C^{2}(\mathbb{R};[0,+\infty))\) and there exist positive constants \(C\) and \(r_0\) such that \(C^{-1}[\ell'(z)]^2\leq \ell(z)\leq C \lvert z\rvert^2\) for \(\lvert z\rvert\leq r_0\).
Example 3. The \(L^p\) (\(2\leq p<\infty\)) loss function satisfies Assumption 5. Here the \(L^p\) (\(1\leq p<\infty\)) loss functions used in machine learning are defined as \(L_\rho(\theta)=\int_{\mathbb{R}^d}\lvert h(x,\theta)-f(x)\rvert^p\rho(x)\mathop{}\!\mathrm{d}{x}\) which is a little bit different from the \(L^p\) norm used in mathematics.
2.4 Regularity↩︎
We begin with the integrability of the hypothesis function. To achieve this, we use the “Japanese bracket” of \(\xi\): \[\langle\xi\rangle:=(1+\lvert\xi\rvert^2)^{1/2}.\]
Lemma 1. Suppose that the Assumption 1 holds. Given any \(\theta\in\mathbb{R}^N\), the hypothesis function \(h\in W^{k,2}(\mathbb{R}^d;\mathbb{R})\) and its gradient with respect to the parameters \(\nabla_\theta h\in W^{k-1,2}(\mathbb{R}^d;\mathbb{R}^N)\). Also, we have \(f\in W^{k,2}(\mathbb{R}^d;\mathbb{R})\).
Proof. Recall that \(f(x)=f_\mathrm{target}(x)\chi(x)\) and \(h(x)=h^{(H)}(x)\chi(x)\) given \(\theta\in\mathbb{R}^N\). By Assumption 1, \(f_\mathrm{target}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\) and \(\chi(x)\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;[0,+\infty))\) with a compact support. Thus \(f\in W^{k,2}(\mathbb{R}^d;\mathbb{R})\). In order to show \(h\in W^{k,2}(\mathbb{R}^d;\mathbb{R})\), it is sufficient to prove that \(h^{(H)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\). Indeed, we prove \(h^{(l)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R}^{n_l})\) for \(l=0,1,\cdots,H\) by induction. For \(l=0\), \(h^{(0)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R}^{n_0})\) because \(h^{(0)}(x)=x\) and \(n_0=d\). Suppose that for \(l\) (\(0\leq l\leq H-2\)) we have \(h^{(l)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R}^{n_l})\). Now let us consider \(h^{(l+1)}\) with \((h^{(l+1)}(x))_i=\sigma_{i}^{(l+1)} (W_i^{(l+1)}\cdot h^{(l)}(x)+b_i^{(l+1)})\), \(i=1,\cdots,n_{l+1}\). By the induction assumption, we have \(W_i^{(l+1)}\cdot h^{(l)}+b_i^{(l+1)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\). By Assumption 1, \(\sigma_i^{(l)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R};\mathbb{R})\). Note that \(\sigma^{(l)}_{i}\in C^{k-1}(\mathbb{R};\mathbb{R})\) by Sobolev embedding. Then \((h^{(l+1)})_i\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\) because of the chain rule and the fact that the composition of continuous functions is still continuous. Finally, for \(l=H-1\), we have \(h^{(H)}=W^{(H)}\cdot h^{(H-1)}+b^{(H)}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\).
The proof for \(\nabla_\theta h\) is similar if we note that \((\sigma_i^{(l)})' \in W^{k-1,\infty}_\mathrm{loc}(\mathbb{R};\mathbb{R})\). ◻
Remark 5. The continuity of \(\sigma_i^{(l)}\) is neccesary because the composition of two Lebesgue measurable functions need not be Lebesgue measurable.
Lemma 2. Suppose that the Assumptions 1 and 2 hold. Then
(a). For any \(0\leq m\leq k\), we have \[\begin{align} \langle\cdot\rangle^m\lvert\hat{h}(\cdot,\theta)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}), \label{eq4646hhatL2}\\ \langle\cdot\rangle^m\lvert\hat{f}(\cdot)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}).\label{eq4646fhatL2} \end{align}\] {#eq: sublabel=eq:eq4646hhatL2,eq:eq4646fhatL2} (b). For any \(0\leq m\leq k-1\), we have \[\langle\cdot\rangle^{m}\lvert\nabla_\theta \hat{h}(\cdot,\theta)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}).\label{eq4646gradienthhatL2}\qquad{(1)}\] (c). For any \(0\leq m\leq 2k-1\), we have \[\langle\cdot\rangle^{m}\lvert\nabla_\theta q(\cdot,\theta)\rvert \in L^1(\mathbb{R}^d;\mathbb{R}).\label{eq4646gradientqL1}\qquad{(2)}\]
Proof. (a). Let \(0\leq m\leq k\). Given \(\theta\in\mathbb{R}^N\), we have \(f, h \in W^{k,2}(\mathbb{R}^d;\mathbb{R})\) by Lemma 1. It is well known that for any function \(g\in W^{k,2}(\mathbb{R}^d)\), for \(0\leq m\leq k\), \[C\lVert g\rVert_{W^{m,2}(\mathbb{R}^d)}\leq \lVert\langle\cdot\rangle^m \lvert\hat{g}\rvert\rVert_{L^2(\mathbb{R}^d)} \leq \tilde{C}\lVert g\rVert_{W^{m,2}(\mathbb{R}^d)},\] where the positive constants \(C\) and \(\tilde{C}\) only depend on \(d\) and \(m\). The statements ?? and ?? follow this.
(b). Let \(0\leq m\leq k-1\). Given \(\theta\in\mathbb{R}^N\), we have \(\nabla_\theta h\in W^{k-1,2}(\mathbb{R}^d;\mathbb{R}^N)\) by Lemma 1. Similar to part (a), this leads to ?? .
(c). Let \(m_1=m-m_2\) and \(m_2=\min\{m, k\}\). Then \(0\leq m_1\leq k-1\) and \(0\leq m_2\leq k\). Combining the inequalities in parts (a) and (b), we have \[\begin{align} \lVert\langle\cdot\rangle^{m} \lvert\nabla_\theta q(\cdot,\theta)\rvert\rVert_{L^1(\mathbb{R}^d)} &= \left\lVert\langle\cdot\rangle^{m}\left\lvert\left(\nabla_\theta \hat{h}(\cdot,\theta)\right) \overline{\hat{h}(\cdot,\theta)-\hat{f}(\cdot)}+\mathrm{c.c.}\right\rvert\right\rVert_{L^1(\mathbb{R}^d)}\nonumber\\ &\leq 2\left\lVert\langle\cdot\rangle^{m_1}\lvert\nabla_\theta\hat{h}(\cdot,\theta)\rvert\right\rVert_{L^2(\mathbb{R}^d)}\left\lVert\langle\cdot\rangle^{m_2}\lvert\hat{h}(\cdot,\theta)-\hat{f}(\cdot)\rvert\right\rVert_{L^2(\mathbb{R}^d)}\nonumber\\ &< \infty. \end{align}\] ◻
Lemma 3. Suppose that the Assumptions 1, 2, and 4 hold. Then
(a). For any \(0\leq m\leq k\), we have \[\begin{align} \langle\cdot\rangle^m\lvert\hat{h}_\rho(\cdot,\theta)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}), \label{eq4646hhatL2rho}\\ \langle\cdot\rangle^m\lvert\hat{f}_\rho(\cdot)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}).\label{eq4646fhatL2rho} \end{align}\] {#eq: sublabel=eq:eq4646hhatL2rho,eq:eq4646fhatL2rho} (b). For any \(0\leq m\leq k-1\), we have \[\langle\cdot\rangle^{m}\lvert\nabla_\theta \hat{h}_\rho(\cdot,\theta)\rvert \in L^2(\mathbb{R}^d;\mathbb{R}).\label{eq4646gradienthhatL2rho}\qquad{(3)}\] (c). For any \(0\leq m\leq 2k-1\), we have \[\langle\cdot\rangle^{m}\lvert\nabla_\theta q_\rho(\cdot,\theta)\rvert \in L^1(\mathbb{R}^d;\mathbb{R}).\label{eq4646gradientqL1rho}\qquad{(4)}\]
Proof. The proof is similar to the one of Lemma 2. The only new ingredient is assumption that \(\sqrt{\rho}\in W^{k,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R})\). ◻
3 Main Results↩︎
In this section, we first propose several quantitative characterization for the F-Principle. Main results are then summarized with numerical illustrations at the end of this section.
3.1 Characterization of F-Principle↩︎
For the MSE loss function, a natural quantity to characterize the F-principle is the ratio of the loss function decrements caused by low frequencies and the total loss function decrements. To achieve this, we devide the MSE loss function into two parts, contributed by low and high frequencies, respectively, i.e., \[L^-_{\rho,\eta}(\theta)=\int_{B_\eta}q_\rho(\xi,\theta)\mathop{}\!\mathrm{d}{\xi},\quad L^+_{\rho,\eta}(\theta)=\int_{B_\eta^c}q_\rho(\xi,\theta)\mathop{}\!\mathrm{d}{\xi},\] where \(B_\eta\) and \(B_\eta^c=\mathbb{R}^d\backslash B_\eta\) are a ball centered at the origin with radius \(\eta>0\) and its complement. Thus \(L_\rho=L^-_{\rho,\eta}+L^+_{\rho,\eta}\) for any \(\eta>0\). The ratio considered for characterizing the F-Principle is \[\frac{\lvert\mathrm{d}L_{\rho,\eta}^-/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\quad \mathrm{and}\quad \frac{\lvert\mathrm{d}L_{\rho,\eta}^+/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}.\label{eq4646CharacterizationRatioRho}\tag{4}\]
For a general loss function, the training dynamics leads to \[\frac{\mathrm{d}\tilde{L}_\rho}{\mathrm{d}t}=-\lvert\nabla_\theta \tilde{L}_\rho\rvert^2.\] In this case, we study \[L(\theta)=\int_{\mathbb{R}^d}\lvert\hat{h}(\xi,\theta)-\hat{f}(\xi)\rvert^2\mathop{}\!\mathrm{d}{\xi}.\] We remark that for a given \(\theta\), \(L(\theta)=\int_{\mathbb{R}^d}\lvert h(x,\theta)-f(x)\rvert^2\mathop{}\!\mathrm{d}{x}\) has nothing to do with \(\mu\). We still take the decomposition \(L=L^-_\eta+L^+_\eta\) with \[L^-_{\eta}(\theta)=\int_{B_\eta}q(\xi,\theta)\mathop{}\!\mathrm{d}{\xi},\quad L^+_{\eta}(\theta)=\int_{B_\eta^c}q(\xi,\theta)\mathop{}\!\mathrm{d}{\xi},\] where \[q(\xi,\theta) =\lvert\hat{h}(\xi,\theta)-\hat{f}(\xi)\rvert^2.\] One can simply mimic 4 and consider \[\frac{\lvert\mathrm{d}L_{\eta}^-/\mathrm{d}t\rvert}{\lvert\mathrm{d}L/\mathrm{d}t\rvert}\quad \mathrm{and}\quad \frac{\lvert\mathrm{d}L_{\eta}^+/\mathrm{d}t\rvert}{\lvert\mathrm{d}L/\mathrm{d}t\rvert}.\label{eq4646CharacterizationRatio}\tag{5}\] However, there is an issue in this characterization: \(L\) may not be monotonically decreasing and the denominator in 5 may be zero. To overcome this, a time averaging is required. Indeed, we investigate the following ratio where integrals are taken for both numerator and denominator in 5 : \[\frac{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^-}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} \quad \mathrm{and}\quad \frac{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^+}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}.\label{eq4646CharacterizationRatioIntegral}\tag{6}\] For the general loss function, we also propose another quantity to characterize the F-Principle: \[\frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}} \quad\mathrm{and}\quad \frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta^c)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}}.\]
3.2 Main Theorems↩︎
As we mentioned in the introduction, the training dynamics of a DNN has three stages: initial stage, intermediate stage, and final stage. For each stage, we provide a theorem to characterize the F-Principle.
Initial Stage
We start with the F-Principle in the initial stage. Clearly, the constants \(C\) in the estimates depend on the initial parameter \(\theta_0\) and the time \(T\).
Theorem 1. [F-Principle in the initial stage]
(\(L^2\) loss function)* Suppose that Assumptions 1, 2, 3, and 4 hold. We consider the training dynamics 2 . Then for any \(1\leq m\leq 2k-1\) and any \(T>0\) satifying \(\lvert\nabla_\theta L_\rho(\theta(T))\rvert>0\) (if \(k=1\), we further require that \(\inf_{t\in(0,T]}\lvert\nabla_\theta L_\rho(\theta(t))\rvert>0\)), there is a constant \(C>0\) such that \[\frac{\lvert\mathrm{d}L^+_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\leq
C\eta^{-m}\quad\mathrm{and}\quad\frac{\lvert\mathrm{d}L^-_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\geq 1-C\eta^{-m},\quad t\in(0,T].\] (general loss function) Suppose that Assumptions 1, 2, 3, and 5 hold. We consider the training
dynamics 3 . Then for any \(1\leq m\leq k-1\) and any \(T>0\) satifying \(\lvert\nabla_\theta
\tilde{L}_\rho(\theta(T))\rvert>0\), there is a constant \(C>0\) such that \[\frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta^c)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}} \leq C\eta^{-m}\quad\mathrm{and}\quad
\frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}} \geq 1-C\eta^{-m},\quad t\in(0,T].\]*
Intermediate Stage
The theorem of intermediate stage is superior to the other results (initial/final stage) in three aspects. First, for a general loss function considered here, Plancherel theorem is not helpful. It is even more challenging to show the F-Principle based on the \(L^2\)-characterization \(L^{-}_\eta(\theta)=\int_{B_\eta}\lvert\hat{h}(\xi,\theta)-\hat{f}(\xi)\rvert^2\mathop{}\!\mathrm{d}{\xi}\) in the training dynamics which is a gradient flow of a non-\(L^2\) loss function: \[\frac{\mathrm{d}\tilde{L}_\rho}{\mathrm{d}t}=-\lvert\nabla_\theta \tilde{L}_\rho\rvert^2.\] Secondly, although \(\tilde{L}_\rho(\theta(t))\) decays as \(t\) increases, \(L(\theta(t))\) may not be monotonically decreasing. As a result, \(\frac{\mathrm{d}L}{\mathrm{d}t}\) might vanish and should not be used in the denominator of the ratio \(\frac{\mathrm{d}L_\eta/\mathrm{d}t}{\mathrm{d}L/\mathrm{d}t}\). However, the ratio still makes sense if we replace the infinitesimal change by a finite decrements in both numerator and denominator (see the precise meaning in Eq. 7 ). The particular choice of a finite decrement is indeed related to the time-scale of the training dynamics. A proper time-scale is the half-life \(T_2-T_1\) satisfying \(\frac{1}{2}L(\theta(T_1))=L(\theta(T_2))\). Thirdly, we obtain an upper bound for the dependence of training period \(T_2-T_1\). This bound works for all the situations. If the non-degenerate global minimizer is obtained, the dependence on \(T_2-T_1\) in Eq. 7 can also be removed and leads to a consistent result to the results for the final stage.
Theorem 2. [F-Principle in the intermediate stage]
(general loss function)* Suppose that Assumptions 1, 2, 3, and 5 hold. We consider the training dynamics 3 . Then for any \(1\leq m\leq k-1\), there is a constant \(C>0\) such that for any \(0<T_1<T_2\) satisfying \(\frac{1}{2}L(\theta(T_1))\geq L(\theta(T_2))\), we have
\[\frac{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^+}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}\leq
C\sqrt{T_2-T_1}\eta^{-m}.\label{eq4646SqrtTGeneralLoss}\tag{7}\] *
Final Stage
If non-degenerate global minimizers are achieved in the training dynamics, we can obtain global-in-time result which characterizing the training dynamics in the final stage. Here we give the definition for non-degenerate minimizers:
Definition 1. A minimizer \(\theta^*\) of \(L_\rho\) (or \(\tilde{L}_\rho\), respectively) is global if \(L_\rho(\theta^*)=0\) (or \(\tilde{L}_\rho(\theta^*)=0\), respectively). The minimizer is non-degenerate if the Hessian matrix \(\nabla_\theta^2 L_\rho(\theta^*)\) (or \(\nabla_\theta^2\tilde{L}_\rho(\theta^*)\), respectively) exists and is positive definite.
Theorem 3. [F-Principle in the final stage]
(\(L^2\) loss function)* Suppose that Assumptions 1, 2, 3, and 4 hold. We consider the training dynamics 2 . If the solution \(\theta\) converges to a non-degenerate global
minimizer \(\theta^*\), then for any \(1\leq m\leq k-1\), there is a constant \(C>0\) such that \[\frac{\lvert\mathrm{d}L^+_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\leq
C\eta^{-m}\quad\mathrm{and}\quad\frac{\lvert\mathrm{d}L^-_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\geq 1-C\eta^{-m},\quad t\in(0,+\infty).\] (general loss function) Suppose that Assumptions 1, 2, 3, and 5 hold. We consider the training
dynamics 3 . If the solution \(\theta\) converges to a non-degenerate global minimizer \(\theta^*\), then for any \(1\leq m\leq k-1\), there is a constant \(C>0\) such that \[\frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta^c)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}} \leq C\eta^{-m}\quad\mathrm{and}\quad
\frac{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_\eta)}}{\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}} \geq 1-C\eta^{-m},\quad t\in(0,+\infty).\]*
3.3 Discussion and Illustrations↩︎
To help the readers get some intuitions of the above theorems, we present a numerical example using the following target function \[f(x)=\sum_{j=1}^{500}\sin(jx/10)/j.\]
The training data are uniformly sampled from \([-3.14,3.14]\) with sample size \(300\). The discrete Fourier transform of \(f(x)\) is shown in Fig. 1 (a), in which we focus on the peak frequencies marked by black squares. First, we use the MSE as the training loss function.
Initial stage in Fig. 1 (b). The ratio of the change of the loss function, \(\lvert\mathrm{d}L_{\eta}^{+}/\mathrm{d}t\rvert/\lvert\mathrm{d}L/\mathrm{d}t\rvert\) in the upper panel, and the ratio of the change of the DNN output, \(\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^{2}(B_{\eta}^c)}/\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^{2}(\mathbb{R}^{d})}\) in the middle panel, both decreases as frequency increases. At such initial stage, only the relative error of the first peak frequency, \(\lvert\hat{h}-\hat{f}\rvert/\lvert\hat{f}\rvert\), decreases to a small value.
Intermediate stage in Fig. 1 (c). The ratio of the change of the loss function in a certain period, \(\lvert L_{\eta}^{+}(\theta(T_1))-L_{\eta}^{+}(\theta(T_2))\rvert/\lvert L_{\eta}(\theta(T_1))-L_{\eta}(\theta(T_2))\rvert\), increases with \(\lvert T_2-T_1\rvert\) for a fixed \(\eta\).
Final stage in Fig. 1 (d). There exists a frequency \(\eta_{0}\) — when \(\eta>\eta_{0},\) the ratio of the change of the loss function, \(\lvert\mathrm{d}L_{\eta}^{+}/\mathrm{d}t\rvert/\lvert\mathrm{d}L/\mathrm{d}t\rvert\) in the upper panel, and the ratio of the change of the DNN output, \(\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(B_{\eta}^c)}/\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^2(\mathbb{R}^d)}\) in the middle panel, both decreases as frequency increases. At such final stage, only peak frequencies corresponding to high frequencies have not converged yet.
Secondly, we use the \(L^{4}\) training loss \(\frac{1}{M}\sum_{i=1}^M(h(x_i,\theta)-y_i)^4\) as shown in Fig. 2. We obtain similar results.
Figure 1: Numerical understanding of theorems of MSE training loss. (a) Amplitudeof DFT of the training samples against frequency index. Frequenciesmarked by black squares are analyzed in the second row. (b, d) upper:\(\lvert\mathrm{d}L_{\geq\eta}/\mathrm{d}t\rvert/\lvert\mathrm{d}L/\mathrm{d}t\rvert\)vs. frequency index. Middle: \(\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^{2}(B_{\eta}^{c})}/\lVert\mathrm{d}\hat{h}/\mathrm{d}t\rVert_{L^{2}(\mathbb{R}^{d})}\)vs. frequency index. Lower: Relative error of each selected frequency,\(\lvert\hat{h}-\hat{f}\rvert/\lvert\hat{f}\rvert\) vs. frequency index. Each sub-figureis plotted at one training epoch. (c) \(\lvert L_{\eta}^+(\theta(T_1))-L_{\eta}^+(\theta(T_2))\rvert/\lvert L_{\eta}(\theta(T_1))-L_{\eta}(\theta(T_2))\rvert\)vs. \(\lvert T_2-T_1\rvert\) with \(\eta\) is selected as the fourth frequencypeak. We use a tanh-DNN with widths 1-200-50-1 with full batch trainingby Adam optimizer. The learning rate is \(2\times10^{-5}\)..
Figure 2: Numerical understanding of theorems of \(L^{4}\) training loss. Theillustrations are same as Fig. 1 (b, c, d), respectively.We use a tanh-DNN with widths 1-500-500-500-500-1 with full batchtraining..
4 Proof of Theorems↩︎
4.1 F-Principle: Initial Stage (Theorem 1)↩︎
In this section, we focus on the initial stage of the training dynamics. The first result shows that the change of loss function concentrates on low frequencies.
In general, \(C\) may depend on \(T\). In the next section, we will provide a similar result in some situation where \(C\) does not depend on \(T\).
Proof of Theorem 1 (\(L^2\) loss function). The dynamics for the loss function contributed by high frequency reads as: \[\begin{align} \frac{\mathrm{d}L^+_{\rho,\eta}(\theta)}{\mathrm{d}t} &= \left(\int_{B_\eta^c}\nabla_\theta q_\rho(\xi,\theta)\mathop{}\!\mathrm{d}{\xi}\right)\cdot\frac{\mathrm{d}\theta}{\mathrm{d}t}\nonumber\\ &= -\left(\int_{B_\eta^c}\nabla_\theta q_\rho(\xi,\theta)\mathop{}\!\mathrm{d}{\xi}\right)\cdot\nabla_\theta L_\rho(\theta). \end{align}\] The dynamics for the total loss function is \[\frac{\mathrm{d}L_\rho(\theta)}{\mathrm{d}t}=-\lvert\nabla_\theta L_\rho(\theta)\rvert^2.\] Therefore \[\begin{align} \frac{\lvert\mathrm{d}L^+_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert} &\leq \frac{\Big(\int_{B_\eta^c}\lvert\nabla_\theta q_\rho(\xi,\theta)\rvert\mathop{}\!\mathrm{d}{\xi}\Big)\lvert\nabla_\theta L_\rho(\theta)\rvert}{\lvert\nabla_\theta L_\rho(\theta)\rvert^2}\nonumber\\ &= \frac{\lVert\nabla_\theta q_\rho(\cdot,\theta)\rVert_{L^1(B_\eta^c)}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}.\label{eq4646DecayFiniteTime} \end{align}\tag{8}\] Note that \(\eta\leq \langle\xi\rangle\) for all \(0<\eta\leq \lvert\xi\rvert\). Therefore \[\begin{align} \lVert\nabla_\theta q_\rho(\cdot,\theta)\rVert_{L^1(B_\eta^c)} &\leq \eta^{-m}\int_{B_\eta^c}\langle\xi\rangle^{m}\lvert\nabla_\theta q_\rho(\xi,\theta)\rvert\mathop{}\!\mathrm{d}{\xi}\nonumber\\ &\leq \eta^{-m}\lVert\langle\cdot\rangle^{m} \lvert\nabla_\theta q_\rho(\cdot,\theta)\rvert\rVert_{L^1(\mathbb{R}^d)}. \end{align}\] By Assumption 3, \(\sup_{t\geq 0}\lvert\theta(t)\rvert\leq R\) and \[\sup_{t\in(0,T]}\lVert\langle\cdot\rangle^m\nabla_\theta q_\rho(\cdot,\theta)\rVert_{L^1(\mathbb{R}^d)}<+\infty.\] For \(k=1\), we take the assumption that \(\inf_{t\in(0,T]}\lvert\nabla_\theta L_\rho(\theta(t))\rvert>0\). For \(k\geq 2\), according to Assumption 1, we have \(\nabla_\theta L_\rho(\cdot)\in W^{1,\infty}_\mathrm{loc}(\mathbb{R}^d;\mathbb{R}^N)\), and hence \(\nabla_\theta L_\rho(\theta)\) is locally Lipschitz in \(\theta\). This together with Assumption 3 implies that \(\nabla_\theta L_\rho(\theta(t))\) is continuous on \(t\in[0,T]\). If \(\inf_{t\in(0,T]}\lvert\nabla_\theta L_\rho(\theta(t))\rvert=0\), then there is a \(t_0\in [0,T]\) such that \(\lvert\nabla_\theta L_\rho(\theta(t_0))\rvert=0\). By the uniqueness of ordinary differential equation, we have \(\lvert\nabla_\theta L_\rho(\theta(T))\rvert=0\) which contridicts with the assumption that \(\lvert\nabla_\theta L_\rho(\theta(t))\rvert>0\). Therefore \(\inf_{t\in(0,T]}\lvert\nabla_\theta L_\rho(\theta(t))\rvert>0\). Thus for \(k\geq 1\) the following ratio is bounded from above: \[C:=\sup_{t\in(0,T]} \frac{\lVert\langle\cdot\rangle^{m} \nabla_\theta q_\rho(\cdot,\theta)\rVert_{L^1(\mathbb{R}^d)}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}<+\infty.\] Therefore \[\frac{\lvert\mathrm{d}L^+_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert}\leq C\eta^{-m},\quad t\in(0,T].\] ◻
Corollary 1 (dissipation). In the situation of Theorem 1 for \(L^2\) loss function, we have that for sufficiently large \(\eta\) \[\frac{\mathrm{d}L^-_{\rho,\eta}}{\mathrm{d}t} \leq -(1-C\eta^{-m})\lvert\nabla_\theta L_\rho\rvert^2 \leq 0\label{eq4646LowFrequencyLossFunctionDecay}.\qquad{(5)}\]
Proof. For sufficiently large \(\eta\), the dynamics of \(L^-_{\rho,\eta}\) is dissipative because \[\frac{\mathrm{d}L^-_{\rho,\eta}(\theta)}{\mathrm{d}t} = \frac{\mathrm{d}L_\rho(\theta)}{\mathrm{d}t}-\frac{\mathrm{d}L^+_{\rho,\eta}(\theta)}{\mathrm{d}t} \leq -(1-C\eta^{-m})\lvert\nabla_\theta L_\rho(\theta)\rvert^2 \leq 0.\] ◻
Next we prove the case of general loss function.
Proof of Theorem 1 (general loss function). On the one hand, we estimate the numerator by studying the dynamics for \(\hat{h}\): \[\frac{\mathrm{d}\hat{h}(\xi,\theta)}{\mathrm{d}t} =\nabla_\theta \hat{h}(\xi,\theta)\cdot\frac{\mathrm{d}\theta}{\mathrm{d}t} =-\nabla_\theta \hat{h}(\xi,\theta)\cdot\nabla_\theta \tilde{L}_\rho(\theta).\label{eq4646dynamicsFtildeh}\tag{9}\] Taking square and integrating both sides on \(B_\eta^c\) leads to the upper bound on the numerator \[\left\lVert\frac{\mathrm{d}\hat{h}(\cdot,\theta)}{\mathrm{d}t}\right\rVert_{L^2(B_\eta^c)} \leq \lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert \lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)}. \label{eq4646hhatGeneralLossNumeratorEstimate}\tag{10}\] On the other hand, note the dynamics for the hypothesis function \[\frac{\mathrm{d}h(x,\theta)}{\mathrm{d}t}=\nabla_\theta h(x,\theta)\cdot \frac{\mathrm{d}\theta}{\mathrm{d}t}=-\nabla_\theta \tilde{L}_\rho(\theta)\cdot \nabla_\theta h(x,\theta)\] and the dynamics for the total loss function \[\frac{\mathrm{d}\tilde{L}_\rho(\theta)}{\mathrm{d}t}=-\lvert\nabla_\theta\tilde{L}_\rho(\theta)\rvert^2.\] Thus we have \[\begin{align} \lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert^2 &=\left\lvert\frac{\mathrm{d}\tilde{L}_\rho(\theta)}{\mathrm{d}t}\right\rvert\nonumber\\ &=\left\lvert\frac{\mathrm{d}}{\mathrm{d}t}\int_{\mathbb{R}^d}\ell(h(x,\theta)-f(x))\rho(x)\mathop{}\!\mathrm{d}{x}\right\rvert\nonumber\\ &=\left\lvert\int_{\mathbb{R}^d} \frac{\mathrm{d}h(x,\theta)}{\mathrm{d}t}\ell'(h(x,\theta)-f(x))\rho(x)\mathop{}\!\mathrm{d}{x}\right\rvert\nonumber\\ &\leq \lVert\sqrt{\rho}\rVert_{L^\infty}\left\lVert\frac{\mathrm{d}h(\cdot,\theta)}{\mathrm{d}t}\right\rVert_{L^2(\mathbb{R}^d)}\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}, \label{eq4646hhatGeneralLossDenominatorEstimate} \end{align}\tag{11}\] where we used the Cauchy–Schwarz inequality in the last step. Combining Eqs. 10 and 11 , we obtain \[\begin{align} \frac{\lVert\frac{\mathrm{d}\hat{h}}{\mathrm{d}t}\rVert_{L^2(B_\eta^c)}}{\lVert\frac{\mathrm{d}\hat{h}}{\mathrm{d}t}\rVert_{L^2(\mathbb{R}^d)}} &\leq \frac{\lVert\sqrt{\rho}\rVert_{L^\infty}\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)} \lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert\lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)} }{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert^2}\nonumber\\ &\leq \lVert\sqrt{\rho}\rVert_{L^\infty}\lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)}\frac{\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert}. \end{align}\] Similar to the case of \(L^2\) loss, \[\begin{align} \lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)} &\leq \eta^{-m} \left(\int_{B_\eta^c}\langle\xi\rangle^{2m}\lvert\nabla_\theta \hat{h}(\xi,\theta)\rvert^2\mathop{}\!\mathrm{d}{\xi}\right)^{1/2}\nonumber\\ &\leq \eta^{-m} \lVert\langle\cdot\rangle^{m} \nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(\mathbb{R}^d)}. \end{align}\] Again, by Assumption 3, \(\sup_{t\geq 0}\lvert\theta(t)\rvert\leq R\) and \[\sup_{t\in(0,T]}\lVert\langle\cdot\rangle^{m} \nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(\mathbb{R}^d)}<+\infty.\] For \(k\geq 2\), the same argument of the \(L^2\) loss case leads to \(\inf_{t\in(0,T]}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert>0\). The proof is completed by the following bound \[\begin{align} \sup_{t\in(0,T]}\frac{\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert} &\leq \sup_{t\in(0,T]}\frac{\left(C\int_{\mathbb{R}^d}\ell(h(x,\theta)-f(x))\rho(x)\mathop{}\!\mathrm{d}{x}\right)^{1/2}}{\lvert\nabla_\theta\tilde{L}_\rho(\theta)\rvert}\nonumber\\ &< +\infty, \nonumber \end{align}\] where we used Assumption 5. ◻
4.2 F-Principle: Intermediate Stage (Theorem 2)↩︎
In this section, we prove the key theorem for the intermediate stage. This theorem then implies several useful corollaries.
Proof of Theorem 2. The numerator can be controlled as follows \[\begin{align} &~~\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^+(\theta)}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}\nonumber\\ &= \int_{T_1}^{T_2}\left\lvert\left(\int_{B_\eta^c}\nabla_\theta q(\xi,\theta(t))\mathop{}\!\mathrm{d}{\xi}\right)\cdot\frac{\mathrm{d}\theta(t)}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}\nonumber\\ &\leq \int_{T_1}^{T_2}\left( \int_{B_\eta^c}\lvert\nabla_\theta q(\xi,\theta(t))\rvert\mathop{}\!\mathrm{d}{\xi}\right) \lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert\mathop{}\!\mathrm{d}{t}\nonumber\\ &=\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert\int_{B_\eta^c}\left\lvert\nabla_\theta \hat{h}(\xi,\theta(t))\overline{\hat{h}(\xi,\theta(t))-\hat{f}(\xi)}+\mathrm{c.c.}\right\rvert\mathop{}\!\mathrm{d}{\xi}\mathop{}\!\mathrm{d}{t}\nonumber\\ &\leq 2\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert\lVert\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(B_\eta^c)}\lVert h(\cdot,\theta(t))-f(\cdot)\rVert_{L^2(\mathbb{R}^d)}\mathop{}\!\mathrm{d}{t}\nonumber\\ &\leq 2\eta^{-m}\left(\sup_{t\in[T_1,T_2]}\lVert\langle\cdot\rangle^{m}\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(\mathbb{R}^d)}\right)\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert L(\theta(t))^{1/2}\mathop{}\!\mathrm{d}{t}, \end{align}\] where in the second-to-last step we used the Cauchy–Schwarz inequality and the Plancherel theorem, and in the last step we used the following \[\lVert\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(B_\eta^c)}\leq \eta^{-m}\lVert\langle\cdot\rangle^{m}\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(\mathbb{R}^d)}.\] By Assumption 3, \(\sup_{t\geq 0}\lvert\theta(t)\rvert\leq R\) and \[C_1:=\sup_{t\geq 0}\lVert\langle\cdot\rangle^{m}\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(\mathbb{R}^d)}<\infty.\] By the assumption that \(\frac{1}{2}L(\theta(T_1))\geq L(\theta(T_2))\) , we have \[\begin{align} \int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t} \geq \lvert L(\theta(T_1))-L(\theta(T_2))\rvert \geq \frac{1}{2}L(\theta(T_1)). \end{align}\] Therefore, \[\begin{align} \frac{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^+}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} &\leq \frac{2C_1\eta^{-m}\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert L(\theta(t))^{1/2}\mathop{}\!\mathrm{d}{t}}{\lvert\frac{1}{2}L(\theta(T_1))\rvert^{1/2}(\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L(\theta(t))}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t})^{1/2}}\nonumber\\ &\leq \frac{2\sqrt{2}C_1\eta^{-m}(\int_{T_1}^{T_2}\lvert\nabla_\theta\tilde{L}_\rho(\theta(t))\rvert^2\mathop{}\!\mathrm{d}{t})^{1/2}}{\lvert L(\theta(T_1))\rvert^{1/2}}\times\frac{(\int_{T_1}^{T_2}L(\theta(t))\mathop{}\!\mathrm{d}{t})^{1/2}}{(\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L(\theta(t))}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t})^{1/2}}. \end{align}\] Recall the training dynamics \[\frac{\mathrm{d}\theta}{\mathrm{d}t} = -\nabla_\theta \tilde{L}_\rho(\theta),\] where for \(k\geq 2\), \(\nabla_\theta\tilde{L}_\rho(\cdot)\in W^{1,\infty}_\mathrm{loc}(\mathbb{R}^N)\subset C^{0,1}(\mathbb{R}^N)\). Hence \(\theta(\cdot)\in C^{0,1}([0,+\infty))\). Taking further time derivative of \(\theta\), we obtain \[\frac{\mathrm{d}^2 \theta}{\mathrm{d}t^2} = -\nabla^2_\theta\tilde{L}_\rho(\theta)\cdot\frac{\mathrm{d}\theta}{\mathrm{d}t} =\nabla^2_\theta\tilde{L}_\rho(\theta)\cdot\nabla_\theta\tilde{L}_\rho(\theta).\] Since \(\nabla_\theta\tilde{L}_\rho(\cdot), \nabla^2_\theta\tilde{L}_\rho(\cdot)\in L^\infty_\mathrm{loc}(\mathbb{R}^N)\) and \(\theta(\cdot)\) is continuous, we have \(\frac{\mathrm{d}^2 \theta(\cdot)}{\mathrm{d}t^2}\in L^{\infty}_\mathrm{loc}([0,+\infty))\). Taking time derivatives of the \(L^2\) loss function \(L\), we obtain \[\begin{align} \frac{\mathrm{d}L}{\mathrm{d}t} &= \nabla_\theta L\cdot\frac{\mathrm{d}\theta}{\mathrm{d}t},\\ \frac{\mathrm{d}^2 L}{\mathrm{d}t^2} &= (\nabla^2_\theta L\cdot\frac{\mathrm{d}\theta}{\mathrm{d}t})\cdot \frac{\mathrm{d}\theta}{\mathrm{d}t}+\nabla_\theta L\cdot\frac{\mathrm{d}^2 \theta}{\mathrm{d}t^2}. \end{align}\] The facts that \(\nabla^2_\theta L(\cdot), \nabla_\theta L(\cdot)\in L^\infty_\mathrm{loc}(\mathbb{R}^N)\) and that \(\theta(\cdot)\) is continuous lead to \(\nabla^2_\theta L(\theta(\cdot)), \nabla_\theta L(\theta(\cdot))\in L^\infty_\mathrm{loc}([0,+\infty))\). This with \(\frac{\mathrm{d}\theta(\cdot)}{\mathrm{d}t},\frac{\mathrm{d}^2 \theta(\cdot)}{\mathrm{d}t^2}\in L^{\infty}_\mathrm{loc}([0,+\infty))\) implies that \(\frac{\mathrm{d}^2 L(\theta(\cdot))}{\mathrm{d}t^2}\in L^{\infty}_\mathrm{loc}([0,+\infty))\). Therefore \(\frac{\mathrm{d}L(\theta(\cdot))}{\mathrm{d}t}\in C^{0,1}([0,+\infty))\). Thus \(M:=\max_{t\in[T_1,T_2]}L(\theta(t))\) is finite. If \(M\leq 2 L(\theta(T_1))\), we have \[\begin{align} \frac{\int_{T_1}^{T_2}L(\theta)\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L(\theta)}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} &\leq \frac{(T_2-T_1)M}{\lvert L(\theta(T_1))-L(\theta(T_2))\rvert}\nonumber\\ &\leq \frac{(T_2-T_1)2L(\theta(T_1))}{\frac{1}{2}L(\theta(T_1))}\nonumber\\ &=4(T_2-T_1).\label{eq4646ratioM1} \end{align}\tag{12}\] If \(M>2L(\theta(T_1))\), then we choose \(t_{M}\in[T_1,T_2]\) such that \(L(\theta(t_M))=M\). We have \[\begin{align} \frac{\int_{T_1}^{T_2}L(\theta)\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L(\theta)}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} &\leq \frac{(T_2-T_1)M}{\lvert L(\theta(T_1))-L( \theta(t_M))\rvert+\lvert L(\theta(t_M))-L(\theta(T_2))\rvert}\nonumber\\ &= \frac{(T_2-T_1)M}{M-L(\theta(T_1))+M-L(\theta(T_2))}\nonumber\\ &\leq \frac{(T_2-T_1)M}{2M-\frac{3}{2}L(\theta(T_1))}\nonumber\\ &\leq \frac{4}{5}(T_2-T_1).\label{eq4646ratioM2} \end{align}\tag{13}\] Combining Eqs. 12 and 13 , we have \[\frac{\int_{T_1}^{T_2}L(\theta)\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L(\theta)}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} \leq 4(T_2-T_1).\] Therefore \[\begin{align} \frac{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L_{\eta}^+}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{T_1}^{T_2}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}} &\leq \frac{2\sqrt{8}C_1\eta^{-m}\sqrt{T_2-T_1}\lvert\tilde{L}_\rho(\theta(T_1))-\tilde{L}_\rho(\theta(T_2))\rvert^{1/2}}{\lvert L(\theta(T_1))\rvert^{1/2}}\nonumber\\ &\leq 4\sqrt{2}C_1\eta^{-m}\sqrt{T_2-T_1}\frac{\lvert\tilde{L}_\rho(\theta(T_1))\rvert^{1/2}}{\lvert L(\theta(T_1))\rvert^{1/2}}\nonumber\\ &= C\sqrt{T_2-T_1}\eta^{-m}, \end{align}\] where \(C=4\sqrt{2}C_1C_2^{1/2}\) and \[C_2:=\sup_{t\geq 0}\frac{\lvert\tilde{L}_\rho(\theta(t))\rvert}{\lvert L(\theta(t))\rvert}.\] Now it is sufficient to show that \(C_2<+\infty\). In fact, there is a constant \(C_3\) such that \(\sup_{\lvert\theta\rvert\leq R}\sup_{x\in\mathbb{R}^d}\lvert h(x,\theta)-f(x)\rvert\leq C_3\). This with Assumption 5 implies that \(\ell(z)\leq C_4\lvert z\rvert^2\) for \(\lvert z\rvert\leq C_3\). Therefore \[\begin{align} C_2 &\leq \sup_{t\geq 0}\frac{\lvert\int_{\mathbb{R}^d}\ell(h(x,\theta(t))-h(x,\theta^*))\rho(x)\mathop{}\!\mathrm{d}{x}\rvert}{\lvert\int_{\mathbb{R}^d}(h(x,\theta(t))-h(x,\theta^*))^2\mathop{}\!\mathrm{d}{x}\rvert}\nonumber\\ &\leq \lVert\rho\rVert_{L^\infty} \sup_{t\geq 0}\frac{\lvert\int_{\mathbb{R}^d}C_4(h(x,\theta(t))-h(x,\theta^*))^2\mathop{}\!\mathrm{d}{x}\rvert}{\lvert\int_{\mathbb{R}^d}(h(x,\theta(t))-h(x,\theta^*))^2\mathop{}\!\mathrm{d}{x}\rvert}<+\infty. \end{align}\] ◻
Remark 6. If the condition \(\frac{1}{2}L(\theta(T_1))\geq L(\theta(T_2))\) is replaced by \(\delta L(\theta(T_1))\geq L(\theta(T_2))\) for any \(\delta\in(0,1)\), the estimates in Theorem 2 and the following corollaries still hold.
Corollary 2. Under the same assumptions in Theorem 2, for any \(1\leq m\leq k-1\), there is a constant \(C>0\) such that for any \(0<T_1<T_2\) satisfying \(\frac{1}{2}L(\theta(T_1))\geq L(\theta(T_2))\) and \(L(\theta(T_1))\geq L(\theta(t))\) for all \(t\in[T_1,T_2]\), we have \[\frac{\lvert L_{\eta}^+(\theta(T_1))-L_{\eta}^+(\theta(T_2))\rvert}{\lvert L(\theta(T_1))-L(\theta(T_2))\rvert}\leq C\sqrt{T_2-T_1}\eta^{-m}.\label{eq4646SqrtTGeneralLossDifferenceRatio}\qquad{(6)}\]
Proof. Similar to the proof of Theorem 2, we have the upper bound for the numerator \[\lvert L_{\eta}^+(\theta(T_1))-L_{\eta}^+(\theta(T_2))\rvert\leq 2\eta^{-m} C_1\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert L(\theta(t))^{1/2}\mathop{}\!\mathrm{d}{t}\] and lower bound for the denominator \[\lvert L(\theta(T_1))-L(\theta(T_2))\rvert\geq \frac{1}{2}L(\theta(T_1))\] with \(C_1:=\sup_{t\in[T_1,T_2]}\lVert\langle\cdot\rangle^{m}\nabla_\theta\hat{h}(\cdot,\theta(t))\rVert_{L^2(\mathbb{R}^d)}<+\infty\). Therefore, these bounds with the assumption that \(L(\theta(T_1))\geq L(\theta(t))\) for all \(t\in[T_1, T_2]\) leads to \[\begin{align} \frac{\lvert L_{\eta}^+(\theta(T_1))-L_{\eta}^+(\theta(T_2))\rvert}{\lvert L(\theta(T_1))-L(\theta(T_2))\rvert} &\leq \frac{2C_1\eta^{-m}\int_{T_1}^{T_2}\lvert\nabla_\theta \tilde{L}_\rho(\theta(t))\rvert L(\theta(t))^{1/2}\mathop{}\!\mathrm{d}{t}}{\frac{1}{2}L(\theta(T_1))}\nonumber\\ &\leq \frac{4C_1\eta^{-m}\int_{T_1}^{T_2}\lvert\nabla_\theta\tilde{L}_\rho(\theta(t))\rvert\mathop{}\!\mathrm{d}{t}}{\lvert L(\theta(T_1))\rvert^{1/2}}\nonumber\\ &\leq C\sqrt{T_2-T_1}\eta^{-m}, \end{align}\] where the last inequality is due to the same reason as Theorem 2. ◻
Corollary 3. Under the same assumptions in Theorem 2, if the solution \(\theta\) converges to a non-degenerate global minimizer \(\theta^*\), then for any \(1\leq m\leq k-1\), the above upper bound can be improved to the following: there is a constant \(C>0\) such that for any \(T>0\), we have \[\frac{\int_{0}^{T}\left\lvert\frac{\mathrm{d}L_{\eta}^+}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}{\int_{0}^{T}\left\lvert\frac{\mathrm{d}L}{\mathrm{d}t}\right\rvert\mathop{}\!\mathrm{d}{t}}\leq C\eta^{-m}\] and \[\frac{\lvert L_{\eta}^+(\theta(0))-L_{\eta}^+(\theta(T))\rvert}{\lvert L(\theta(0))-L(\theta(T))\rvert} \leq C\eta^{-m}.\]
We skip the proof since this corollary can be obtained directly from Theorem 3.
4.3 F-Principle: Final Stage (Theorem 3)↩︎
In this section, we prove the F-Principle in final stage of the training dynamics.
Proof of Theorem 3 (\(L^2\) loss function). Following 8 , we have \[\begin{align} \frac{\lvert\mathrm{d}L^+_{\rho,\eta}/\mathrm{d}t\rvert}{\lvert\mathrm{d}L_\rho/\mathrm{d}t\rvert} &\leq \frac{\int_{B_\eta^c}\left\lvert\nabla_\theta \hat{h}_\rho(\xi,\theta)\overline{\hat{h}_\rho(\xi,\theta)-\hat{f}_\rho(\xi)}+\mathrm{c.c.}\right\rvert\mathop{}\!\mathrm{d}{\xi}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}\nonumber\\ &\leq \frac{2\lVert\nabla_\theta \hat{h}_\rho(\cdot,\theta)\rVert_{L^2(B_\eta^c)}\lVert\hat{h}_\rho(\cdot,\theta)-\hat{f}_\rho(\cdot)\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta L_\rho(\theta)\rvert},\nonumber\\ &= 2\lVert\nabla_\theta \hat{h}_\rho(\cdot,\theta)\rVert_{L^2(B_\eta^c)}\frac{\lvert L_\rho(\theta)\rvert^{1/2}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}, \end{align}\] where we used \(\lVert\hat{h}_\rho(\cdot,\theta)-\hat{f}_\rho(\cdot)\rVert_{L^2(\mathbb{R}^d)}^2= L_\rho(\theta)\) in the last inequality. Similar to the local-in-time situation, \[\begin{align} \lVert\nabla_\theta \hat{h}_\rho(\cdot,\theta)\rVert_{L^2(B_\eta^c)} &\leq \eta^{-m} \left(\int_{B_\eta^c}\langle\xi\rangle^{2m}\lvert\nabla_\theta \hat{h}_\rho(\xi,\theta)\rvert^2\mathop{}\!\mathrm{d}{\xi}\right)^{1/2}\nonumber\\ &\leq \eta^{-m} \lVert\langle\cdot\rangle^{m} \nabla_\theta\hat{h}_\rho(\cdot,\theta)\rVert_{L^2(\mathbb{R}^d)}. \end{align}\] By Assumption 3, \(\sup_{t\geq 0}\lvert\theta(t)\rvert\leq R\) and \[\sup_{t\in(0,+\infty)}\lVert\langle\cdot\rangle^{m} \nabla_\theta\hat{h}_\rho(\cdot,\theta(t))\rVert_{L^2(\mathbb{R}^d)}<+\infty.\] Now it is sufficient to prove that \[C:=\lim_{t\to+\infty}\frac{\lvert L_\rho(\theta)\rvert^{1/2}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}<+\infty.\] This is true because \[\begin{align} C &= \lim_{\theta\to\theta^*}\frac{\lvert L_\rho(\theta)\rvert^{1/2}}{\lvert\nabla_\theta L_\rho(\theta)\rvert}\nonumber\\ &= \lim_{\theta\to\theta^*}\frac{\lvert(\theta-\theta^*)^T\Lambda(\theta-\theta^*)+o(\lvert\theta-\theta^*\rvert^2)\rvert^{1/2}}{\lvert 2\Lambda(\theta-\theta^*)\rvert+o(\lvert\theta-\theta^*\rvert)}\nonumber\\ &<+\infty, \end{align}\] where we used the assumption that the minimizer is non-degenerate with the Hessian \(\Lambda=\nabla_\theta^2 L_\rho(\theta^*)\). ◻
Now we finish the proof for general loss function.
Proof of Theorem 3 (general loss function). By the proof of Theorem 1, we have \[\begin{align} \frac{\lVert\frac{\mathrm{d}\hat{h}}{\mathrm{d}t}\rVert_{L^2(B_\eta^c)}}{\lVert\frac{\mathrm{d}\hat{h}}{\mathrm{d}t}\rVert_{L^2(\mathbb{R}^d)}} &\leq 2\lVert\sqrt{\rho}\rVert_{L^\infty}\lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)}\frac{\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert} \end{align}\] and \[\begin{align} \lVert\nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(B_\eta^c)} &\leq \eta^{-m} \lVert\langle\cdot\rangle^{m} \nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(\mathbb{R}^d)}. \end{align}\] Since \(\lim_{t\to+\infty}\theta(t)=\theta^*\), we have \[\sup_{t\in(0,+\infty)}\lVert\langle\cdot\rangle^{m} \nabla_\theta \hat{h}(\cdot,\theta)\rVert_{L^2(\mathbb{R}^d)}<+\infty.\] Now it is sufficient to prove that \[\sup_{t\in(0,\infty]}\frac{\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert}<+\infty.\] This is true because \[\begin{align} \lim_{t\to+\infty}\frac{\lVert\ell'(h(\cdot,\theta)-f(\cdot))\sqrt{\rho(\cdot)}\rVert_{L^2(\mathbb{R}^d)}}{\lvert\nabla_\theta \tilde{L}_\rho(\theta)\rvert} &= \lim_{t\to+\infty}\frac{\left(\int_{\mathbb{R}^d}[\ell'(h(x,\theta)-f(x))]^2\rho(x)\mathop{}\!\mathrm{d}{x}\right)^{1/2}}{\lvert\nabla_\theta\tilde{L}_\rho(\theta)\rvert}\nonumber\\ &\leq \lim_{t\to+\infty}\frac{\left(C\int_{\mathbb{R}^d}\ell(h(x,\theta)-f(x))\rho(x)\mathop{}\!\mathrm{d}{x}\right)^{1/2}}{\lvert\nabla_\theta\tilde{L}_\rho(\theta)\rvert}\nonumber\\ &= \lim_{t\to+\infty}\frac{C^{1/2} \lvert\tilde{L}_\rho(\theta)\rvert^{1/2}}{\lvert\nabla_\theta\tilde{L}_\rho(\theta)\rvert}\nonumber\\ &= \lim_{\theta\to\theta^*}\frac{\lvert(\theta-\theta^*)^T\tilde{\Lambda}(\theta-\theta^*)+o(\lvert\theta-\theta^*\rvert^2)\rvert^{1/2}}{\lvert 2\tilde{\Lambda}(\theta-\theta^*)\rvert+o(\lvert\theta-\theta^*\rvert)}\nonumber\\ &<+\infty, \end{align}\] where we used Assumption 5 and the assumption that the minimizer is non-degenerate with the Hessian \(\tilde{\Lambda}=\nabla_\theta^2 \tilde{L}_\rho(\theta^*)\). ◻