In this paper, we develop a regularization theory for neural network approximations of general ill-posed operator equations with noisy data. Within the framework of iterative regularization, we introduce two expanding neural network methods (ENNs) under
different a priori assumptions on the exact solution. Instead of prescribing a fixed architecture, ENNs adaptively select the number of neurons through an a posteriori stopping rule, so that the selected network size serves as a regularization parameter
balancing approximation accuracy and stability with respect to data noise. We prove the regularization properties of the proposed ENNs and establish quantitative relationships between the selected network size and the noise level. Within the framework of
variational regularization, we propose a neural network-based Tikhonov scheme and derive both convergence and convergence-rate results under mild assumptions. The resulting estimates account for the noise level, the network size, and the underlying
smoothness expressed through general variational source conditions, thereby allowing greater flexibility than existing results. Numerical experiments demonstrate the effectiveness and robustness of the proposed algorithms. In particular, they show that,
for highly noisy data, relatively small network architectures can already produce stable reconstructions, whereas excessively large architectures may degrade stability due to overfitting.
Inverse problems arise in various scientific and engineering fields, including medical imaging, geophysics, and computational physics. These problems often involve reconstructing unknown parameters or functions from indirect, noisy, and sometimes
incomplete measurements. Traditional approaches rely on regularization techniques, optimization algorithms, and statistical inference. However, these methods often suffer from computational inefficiency, sensitivity to noise, and the need for handcrafted
priors. With the rapid advancement of deep learning, neural networks have emerged as powerful tools for solving inverse problems. Their ability to learn complex mappings from data has significantly improved reconstruction accuracy and computational
efficiency. In this paper, we study the neural network methods for solving general operator equations \[\label{1}
A(f)=g,\tag{1}\] where \(A \in \mathbb{M}(\mathcal{X}, \mathcal{Y})\), and \(\mathbb{M}(\mathcal{X}, \mathcal{Y})\) denotes the space of all bounded (possibly non-linear)
operators between two reflexive Banach spaces of functions. For simplicity, we denote by \(\|\cdot\|\) the norms for both Banach spaces. The precise setting of ambient spaces for functions \(f\) and \(g\) will be proposed later in Section 2. Suppose that instead of inexact right-hand side \(g\), we are given only
perturbed element \(g^\delta\), obeying the deterministic noise model with noise level \(\delta>0\): \[\label{NoiseLevel}
\|g^{\delta}-g\|_\mathcal{Y} \le \delta.\tag{2}\]
Since the operator equation 1 is generally ill-posed, directly fitting the noisy data \(g^\delta\) may lead to instability. A regularization method is therefore introduced to construct, for the known
forward operator \(A\) and the observed noisy data \(g^\delta\) with noise level \(\delta\), a stable approximation \(f^\delta\) to the exact solution. Following Tikhonov’s concept of regularization [1], such an approximation is called a
regularized solution if it converges to an exact solution \(f^\dagger\) of 1 , in a prescribed topology, as \(\delta\to0\). The computational inverse problem considered
in this work can therefore be formulated as follows: given inexact data \(g^\delta\) with the noise model (2 ) and the forward model (1 ), find a regularized solution \(f^\delta\) of inverse problems 12 . It should be noted that this notion of regularization does not, in general, determine a unique regularized solution. Indeed, if \(f^\delta\) is a regularized solution and \(h(\delta)\in\mathcal{X}\) is any perturbation satisfying \(h(\delta)\to0\) as \(\delta\to0\) in the same topology, then, provided that \(f^\delta+h(\delta)\in\mathcal{X}\), the perturbed family \(f^\delta+h(\delta)\) also converges to \(f^\dagger\). This observation indicates that the definition of regularization leaves room for selecting regularized solutions from a prescribed approximation class. In this work, we investigate whether such a selection can be
made within a neural network class. More precisely, we ask whether stable finite-dimensional regularized solutions can be constructed by neural networks while retaining convergence to the exact solution as the noise level tends to zero.
The approximation theory of neural networks provides an important foundation for this purpose, since it characterizes approximation capabilities for prescribed function classes and quantifies their dependence on network architecture. Classical universal
approximation results show that single-hidden-layer feedforward networks with suitable activation functions can approximate large classes of continuous functions, as established by Cybenko (1989) [2], Hornik (1989, 1991) [3], [4], and Stinchcombe & White (1989) [5]. Barron [6] further obtained quantitative approximation bounds for one-hidden-layer neural
networks, showing that functions with suitable Barron-type regularity can be approximated at the rate \(\mathcal{O}(n^{-1/2})\) in \(L^2\), where \(n\) is
the number of hidden neurons. More recent works have quantified how the approximation error depends on the width and depth of ReLU networks for various function classes; see, for example, [7]–[12].
These approximation results provide important guidance on how the expressive power of neural networks grows with architectural complexity. However, they mainly concern direct approximation problems, where the target function is available. In ill-posed
inverse problems, the unknown solution must be recovered from noisy indirect data through a forward operator. Hence, increasing the network size improves approximation capability but may also amplify the instability caused by data perturbations. This makes
the architecture itself a potential regularization parameter.
Neural network methods have been widely applied to inverse problems. For parameter estimation problems, particularly those governed by partial differential equations (PDEs), representative methods include the deep Ritz method [13], the deep Galerkin method [14], physics-informed
neural networks (PINNs) [15], weak adversarial networks [16], and other neural network-based approaches for parameter identification in PDE models [17]–[22].
For image reconstruction problems, convolutional neural networks and U-Net type architectures have been successfully used in applications such as electrical impedance tomography, computed tomography, and magnetic resonance imaging [23]–[33]. These works demonstrate the practical effectiveness of neural networks in inverse problems, but their main emphasis is often on model design, data-driven
reconstruction performance, or application-specific parameter identification.
Closer to the present work are neural network methods that are explicitly connected to regularization theory. The NETT approach defines a data-driven regularizer through a trained neural network and incorporates it into a generalized Tikhonov
functional, for which well-posedness, convergence, and error estimates have been established [34]. Another related direction is the deep image prior,
in which the architecture of an untrained convolutional generator network itself serves as an implicit image prior [35]. Dittmer et
al. further investigated deep image prior methods for ill-posed inverse problems and interpreted them as a form of regularization induced by the network architecture [36]. These approaches demonstrate that neural networks can contribute to regularization either through learned regularizers or through architecture-induced priors. In contrast, the
present paper employs neural networks directly to represent regularized solutions, rather than to define an external regularizer or to impose a fixed generator prior. More precisely, the approximation in this work is obtained through an unsupervised
learning approach: it is learned directly from the given noisy observation \(g^\delta\), without relying on external training samples or labeled exact solutions. Moreover, we do not prescribe a fixed architecture in
advance. Instead, we investigate whether suitably constrained and adaptively selected classes of shallow neural networks can serve as admissible regularization spaces for ill-posed inverse problems. In this framework, the network width is treated as an
architecture-dependent regularization parameter and is selected from the noisy data. This perspective naturally leads to the following questions:
Does there exist a neural network approximation that provides a regularized solution to the ill-posed problem?
Is a larger neural network always preferable for solving ill-posed inverse problems with noisy data, or is there a stability limitation on the size or complexity of the neural network? What is the relation between noise level and size of neural
network?
If such a limitation exists, how can one construct a neural network with sufficiently small architectural complexity that remains effective in solving the inverse problem with noisy data?
What is an appropriate size of a neural network for a variational regularization method based on neural networks that achieves optimal convergence rates with respect to the noise level for general ill-posed operator problems (1
)?
The questions (Q1)–(Q3) are addressed by the expanding neural network methods (ENNs) developed in Section 3, while (Q4) is studied in Section 4 through a neural network-based Tikhonov
regularization scheme (Tikhonov-NN) under variational source conditions.
In this work, within the framework of iterative regularization theory [37], [38], and inspired by the method of extending compacts [39]–[41], we propose two expanding neural network methods for constructing regularized solutions using shallow neural networks. In these methods, the number of neurons is adaptively selected by an
a posteriori stopping rule, and the selected network size serves as a regularization parameter that balances approximation accuracy and stability in the presence of noisy data. Within the framework of variational regularization theory, in a spirit similar
to PINNs for approximating solution functions of PDEs, we study neural network approximations for inverse problems in a general abstract setting and derive unified convergence results under mild assumptions.
Our analysis of these unified convergence results is carried out for two-layer neural networks. The natural functional framework for such networks is the so-called Barron space. Li et al. [42] established approximation results for two-layer ReLU-type networks in Barron spaces. In particular, for functions in \(\mathcal{B}_1\), such networks with bounded
Barron-type representation cost achieve an \(H^1\) approximation rate of order \(\mathcal{O}(n^{-1/2})\). This result relates the network width, the Barron representation cost, and the
approximation error, and thus provides the theoretical foundation for using bounded two-layer neural network classes as admissible solution spaces in our regularization analysis.
The remainder of the paper is structured as follows: Section 2 provides preliminary results regarding the properties of Barron spaces, forming the foundation for our analysis. Section 3
addresses iterative regularization, where we introduce the two proposed expanding neural network methods. In Section 4, we examine Tikhonov regularization and present convergence analyses, including convergence rates under
variational source conditions. Numerical experiments demonstrating the effectiveness of our methods are presented in Section 5, followed by concluding remarks in Section 6.
Functions in a Barron space take the following form \[\label{continuef}
f(\mathbf{x})=\int_{P} a \sigma\left(\boldsymbol{b}^T \mathbf{x}+c\right) \rho(\mathrm{d} a, \mathrm{d} \boldsymbol{b}, \mathrm{d} c), \quad \mathbf{x} \in \Omega\subset \mathbb{R}^d,\tag{3}\] where \(P=\mathbb{R}\times \mathbb{R}^d\times \mathbb{R}\), \(\rho\) is a probability distribution on \((P,\Sigma_P)\) with \(\Sigma_P\)
being a Borel \(\sigma\)-algebra on \(P\), \(\sigma\) is activation function, and \(\Omega\) is a bounded domain. We focus
on the ReLU activation \(\sigma(z):=\max(0,z):=(z)_{+}\). Then given the probability distribution \(\rho\), we define \[\|f\|_{\mathcal{B}_{p,\rho}}=\left(\mathbb{E}_\rho\left[|a|^p\left(\|\boldsymbol{b}\|_1+|c|\right)^p\right]\right)^{1 / p}=\left(\int_{P}|a|^p(\|\boldsymbol{b}\|_{1}+|c|)^{p})\rho(\mathrm{d} a,\mathrm{d}\boldsymbol{b},\mathrm{d}
c)\right)^{\frac{1}{p}}, \quad 1 \leq p \leq+\infty.\] Then, the Barron norm of a function of the form (3 ) is defined by \[\label{Bpnorm}
\|f\|_{\mathcal{B}_{p}}=\inf_{\rho} \|f\|_{\mathcal{B}_{p,\rho}}.\tag{4}\] The Barron spaces \(\mathcal{B}_p\) are defined as the set of continuous functions that can be represented by \((\ref{continuef})\) with a finite Barron norm: \[\label{Barron}
\mathcal{B}_p := \{ f:~ f \text{~satisfies~} \eqref{continuef} \text{~and~} \|f\|_{\mathcal{B}_{p}}<\infty \}.\tag{5}\] By definition, the Barron space \(\mathcal{B}_p\) consists of continuous functions.
Thus, after setting \(K:=\overline{\Omega}\), we regard \(\mathcal{B}_p\) as a subset of \(C(K)\). Then the following properties hold for Barron spaces.
Proposition 1 ([43]). For any \(1\le p\le \infty\), there hold \(\mathcal{B}_1=\mathcal{B}_{p}\) (as sets) and \(\|f\|_{\mathcal{B}_1}=\|f\|_{\mathcal{B}_{p}}\).
Proposition 2 ([42]). The set \(\mathcal{B}_1\) equipped with the functional \(f \mapsto\|f\|_{\mathcal{B}_1}\) is a normed space. Moreover, the Barron space \(\mathcal{B}_1\) is a subspace of standard Sobolev space \(H^1(\Omega)\), and
there exists a constant \(C(\Omega,d)\) such that for all \(f\in \mathcal{B}_1\) there holds \(\|f\|_{H^1(\Omega)}\le
C(\Omega,d)\|f\|_{\mathcal{B}_1}\).
We construct approximations using two-layer neural networks, which are empirical analogues of 3 and have the form \[\label{repfn}
f_n(\mathbf{x})
=
\frac{1}{n}
\sum_{j=1}^n
a_j
\left(\boldsymbol{b}_j^T\mathbf{x}+c_j\right)_+,
\qquad
(a_j,\boldsymbol{b}_j,c_j)\in
\mathbb{R}\times\mathbb{R}^d\times\mathbb{R} .\tag{6}\] For \(r>0\), define the admissible parameter set \(M_r
:=
\left\{
(a,\boldsymbol{b},c)\in\mathbb{R}\times\mathbb{R}^d\times\mathbb{R}:
|a|\le r,\;
\|\boldsymbol{b}\|_1+|c|=1
\right\}\). Then \(M_r\) is compact in \(\mathbb{R}\times\mathbb{R}^d\times\mathbb{R}\). We define the class of two-layer networks with width \(n\)
and parameter radius \(r\) by \[\label{2}
X_{n,r}
:=
\left\{
f_n:
f_n(\mathbf{x})
=
\frac{1}{n}
\sum_{j=1}^n
a_j
\left(\boldsymbol{b}_j^T\mathbf{x}+c_j\right)_+,
\quad
(a_j,\boldsymbol{b}_j,c_j)\in M_r
\right\}.\tag{7}\] For every \(f_n\in X_{n,r}\), define the empirical probability measure \(\rho_n:=\frac{1}{n}\sum_{j=1}^n
\delta_{(a_j,\boldsymbol{b}_j,c_j)}\) where \(\delta_z\) denotes the Dirac point mass at \(z\). Then \(f_n\) admits the integral representation \(f_n(\mathbf{x})=\int_P
a(\boldsymbol{b}^T\mathbf{x}+c)_+
\,\rho_n(da,d\boldsymbol{b},dc)\) for \(\mathbf{x}\in\Omega\). In applications, the radius may depend on the width; in this case we write \(r=r(n)\) and work with the class \(X_{n,r(n)}\), where \(\{r(n)\}_{n\ge1}\) is chosen to be nondecreasing. For the non-truncated (unbounded) case, we define the admissible approximation class as the union of all finite-radius
classes: \[\label{eq:Xninfty95union}
X_{n,\infty}
:=\bigcup_{r<\infty}X_{n,r}.\tag{8}\] It is immediate that \(\{X_{n,r}\}_{r<\infty}\) is nested: if \(0<r_1\le r_2<\infty\), then \(X_{n,r_1}\subset X_{n,r_2}\subset X_{n,\infty}.\)
We shall use the following compactness results in the convergence analysis. The first lemma concerns the compactness of the finite-dimensional admissible network class with fixed width and fixed parameter radius. The second one gives \(L^2(\Omega)\)-compactness from uniform boundedness in \(\mathcal{B}_1\) or \(H^1(\Omega)\). The third one is tailored to sequences of neural networks whose
widths may vary, provided that their Barron representation costs remain uniformly bounded. The proofs of these compactness results are collected in Appendix 7.
Lemma 1. For any fixed \(n\in \mathbb{N}\) and \(r>0\), \(X_{n,r}\) is sequentially compact in \(C(K)\) and \(L^2(\Omega)\).
Lemma 2. Any bounded sequence in Barron space \(\mathcal{B}_1\) or Sobolev space \(H^1(\Omega)\) has a subsequence that converges in \(L^2(\Omega)\).
Proof. It follows since \(\mathcal{B}_1\) continuously embeds into \(H^1(\Omega)\), and \(H^1(\Omega)\) compactly embeds into \(L^2(\Omega)\). ◻
Lemma 3. Let \(K=\overline{\Omega}\) be compact. Let \[f_m(\mathbf{x})
=
\frac{1}{n_m}\sum_{j=1}^{n_m}
a_j^{(m)}
\left((\boldsymbol{b}_j^{(m)})^T\mathbf{x}+c_j^{(m)}\right)_+,\] where \(\|\boldsymbol{b}_j^{(m)}\|_1+|c_j^{(m)}|=1\). Assume that \(q_m:=
\frac{1}{n_m}\sum_{j=1}^{n_m}|a_j^{(m)}|
\le Q\) for some constant \(Q>0\) independent of \(m\). Then \(\{f_m\}\) is relatively compact in \(C(K)\).
Moreover, if a subsequence \(f_{m_k}\) converges to \(f\) in \(C(K)\), then \(f\in\mathcal{B}_1\) and \[\|f\|_{\mathcal{B}_1}
\le
\liminf_{k\to\infty} q_{m_k}.\]
The next result shows that functions defined in the Barron space \(\mathcal{B}_1\) can be approximated by two-layer networks with good approximation properties ([42] and [43]).
Proposition 3 (Approximation theorem). [42] Let \(f \in \mathcal{B}_1\), and denote \(\rho=\rho(\epsilon)\) as its representing probability which satisfies \(c_{\rho}(f) \leq(1+\epsilon)\|f\|_{\mathcal{B}_1}\) for some small \(\epsilon>0\),
where \[\label{cRho}
c_{\rho}(f)=\|f\|_{\mathcal{B}_{1,\rho}}.\qquad{(1)}\] Then, for any sample size \(n \in \mathbb{N}\), there exists a probability \(\rho_n\), a two-layer network \(f_n\in X_{n,c_{\rho}(f)}\) with \(\left\|f_n\right\|_{\mathcal{B}_1} \leq c_{\rho_n}\left(f_n\right) \leq c_{\rho}(f)\), and a constant \(C(\Omega, d)\)
depending only on the domain \(\Omega\) and dimensionality \(d\) such that \[\left\|f-f_n\right\|_{H^1(\Omega)}\leq \frac{C(\Omega,
d)}{\sqrt{n}}\|f\|_{\mathcal{B}_1}.\]
The proof of Proposition 3 is also recalled in Appendix 7.
In this section, based on the method of expanding compacts [39], [40], we propose a new iterative regularization method, i.e., expanding neural network method (ENNs). It converges to the unique exact solution \(f^{\dagger}\) to \(A(f) = g\), if \(A\) is a continuous injective operator. We consider two cases, depending on whether certain a priori information on the exact solution is available.
3.1 Case 1: given an energy bound of \(c_{\rho}(f^{\dagger})\)↩︎
Let \(b:\mathbb{N}^+\to \mathbb{N}^+\) be a monotonically increasing function such that there exist constants \(C_b, k_0\ge 1\) such that \(b(k+1)\le C_b\,
b(k)\) holds for all \(k\ge k_0\). The first iterative regularization scheme is given in the following algorithm. This algorithm requires the a priori information \(c_\rho(f^\dagger)\) of the exact solution, where \(\rho\) denotes a representing probability measure of \(f^\dagger\) and \(c_\rho(f^\dagger)=\|f^\dagger\|_{\mathcal{B}_{1,\rho}}\).
Figure 1: ENN1: an expanding neural network method for solving (1 ) with knowledge of \(c_{\rho}(f^{\dagger})\)
The update rule \(n \gets b(k)\) in Algorithm 1 provides a flexible strategy for increasing the network complexity. Typical choices include \(b(k) = k +
1\) (linear growth) or \(b(k) = k + m\) for some fixed constant \(m\) (fixed-step growth). Below \(b^{-1}\) denotes the (generalized) inverse of \(b\) defined by \(b^{-1}(t):=\min\{k\in\mathbb{N}^+: b(k)>t\}\). The approximate solution \(f_{n,k}^{\delta}\) is obtained by solving a minimization problem
over a bounded compact set \(X_{n,r(n)}\), which guarantees the existence of a regularized solution. The set \(X_{n,r(n)}\) expands with increasing \(n\), so
the algorithm yields wider two layer networks until it is terminated by Morozov’s discrepancy principle.
Now, we give the convergence properties of Algorithm 1 in Theorem 4. If the upper bound of \(c_{\rho}(f^{\dagger})\) for \(f^\dagger\) is known and \(r(n)\) satisfies mild conditions, then the sequence generated by Algorithm 1 terminates after a finite number of steps and converges to \(f^\dagger\). Moreover, we derive asymptotic estimates on the stopping index and the number of neurons, which are directly
linked to the degree of Hölder continuity of \(A\).
Definition 1. The operator \(A\) is said to be locally Hölder continuous at \(f^\dagger\) of order \(\theta\in(0,1]\) with respect to
the \(H^1\)-norm if there exist \(L_A>0\) and \(\gamma>0\) such that \[\|A(f)-A(f^\dagger)\|_{\mathcal{Y}} \le L_A \,
\|f-f^\dagger\|_{H^1(\Omega)}^{\theta}, \quad\forall f \text{ with } \|f-f^\dagger\|_{H^1(\Omega)}<\gamma.\] The case \(\theta=1\) reduces to local Lipschitz continuity.
Theorem 4. Let \(A:L^2(\Omega)\to\mathcal{Y}\) be continuous and injective. Let \(\{r(n)\}_{n\ge1}\) be a positive nondecreasing sequence such that \[r(n)\nearrow B
\quad\text{as } n\to\infty,
\qquad
B>c_\rho(f^\dagger).\] Then the following statements hold.
(Well-posedness of Algorithm 1) For any \(\delta >0\), the iteration process in Algorithm 1 terminates after a finite number \(k(\delta)\) steps with finitely many \(n(\delta)\) neurons, i.e. \(k(\delta)<+\infty,n(\delta)<+\infty\). Moreover, if \(A\) is locally Hölder continuous at \(f^\dagger\) of order \(\theta\), then there exists \(C>0\), independent of \(\delta\), such that \[k(\delta) \le b^{-1}\!\big(C\,\delta^{-2/\theta}\big),
\qquad
n(\delta) = \mathcal{O}\big(\delta^{-2/\theta}\big)\footnote{ Throughout the paper we write f(\delta)=\mathcal{O}(g(\delta)) as \delta\to0 if there exists a constant C>0, independent of \delta, such that |f(\delta)| \le C\,g(\delta) for all sufficiently
small \delta>0.},
\quad \text{as } \delta\to0.\]
(Regularizing property of Algorithm 1) The approximate solutions \(f_{n(\delta),k(\delta)}^{\delta}\) converge to \(f^{\dagger}\) in \(C(K)\) as \(\delta \to 0\).
Proof. By Lemma 1, \(X_{n,r(n)}\) is sequentially compact in \(L^2(\Omega)\).
Since \(L^2(\Omega)\) is a metric space, \(X_{n,r(n)}\) is compact in \(L^2(\Omega)\). Moreover, since \(A:L^2(\Omega)\to\mathcal{Y}\) is continuous, the functional \(J(f)=\|A(f)-g^{\delta}\|_{\mathcal{Y}}\) is continuous on \(X_{n,r(n)}\). Then by Weierstrass
theorem, \(J\) attains its minimum on \(X_{n,r(n)}\). (The minimizer may be nonunique for a non-convex functional \(J\).)
Since \(r(n)\nearrow B\) and \(B>c_\rho(f^\dagger)\), there exists \(N_1\) such that \(r(n)>c_\rho(f^\dagger)\) for
all \(n\ge N_1\). Since \(A\) is continuous at \(f^\dagger\) with respect to the \(L^2(\Omega)\)-norm, for each \(\delta>0\) there exists \(\varepsilon_1(\delta)>0\) such that \[\|A(f)-A(f^\dagger)\|_{\mathcal{Y}}\le (\tau-1)\delta
\quad\text{whenever}\quad
\|f-f^\dagger\|_{L^2(\Omega)}\le \varepsilon_1(\delta).\] Let \(N=N(\delta)=\max\{N_1,(\frac{C(\Omega,d)\|f^{\dagger}\|_{\mathcal{B}_1}}{\varepsilon_1(\delta)})^2,b(k_0)\}\). For every \(n>N\), Proposition 3 gives \(f_n\in X_{n,c_\rho(f^\dagger)}\subset X_{n,r(n)}\) such that
\(\|f^\dagger-f_n\|_{H^1(\Omega)}<\varepsilon_1(\delta)\). Hence also \(\|f^\dagger-f_n\|_{L^2(\Omega)}<\varepsilon_1(\delta)\).
Let \(k^{*}:=\min\{k\in \mathbb{N}^{+}|b(k)>N\}\) and \(\overline{N}:=b(k^*)\). Then \(\overline{N}>N\). Since \(N\ge
b(k_0)\) and \(b\) is nondecreasing, we have \(k^*>k_0\). Hence \(k^*-1\ge k_0\), and by the minimality of \(k^*\), \(b(k^*-1)\le N<b(k^*)=\overline{N}\). Let \(f_{\overline{N}}\in X_{\overline{N},c_\rho(f^\dagger)}
\subset X_{\overline{N},r(\overline{N})}\) be the approximating network constructed above with \(n=\overline{N}\). Now we show that \(n(\delta)\le b(k^*),k(\delta)\le k^*\). Indeed,
since \(f_{\overline{N},k^*}^{\delta}\) is a minimizer over \(X_{\overline{N},r(\overline{N})}\) and \(f_{\overline{N}}\in
X_{\overline{N},r(\overline{N})}\), we have \[\begin{align}
J_{\overline{N},k^*}^{\delta} &=\left\|A(f_{\overline{N},k^*}^{\delta})-g^{\delta}\right\|_{\mathcal{Y}} \leq \left\|A(f_{\overline{N}})-g^{\delta}\right\|_{\mathcal{Y}} \nonumber \\
\label{JNlestaudel}
&\leq \left\|A(f_{\overline{N}})-A(f^\dagger)\right\|_{\mathcal{Y}} + \left\|A(f^\dagger)-g^{\delta}\right\|_{\mathcal{Y}} \leq(\tau-1)\delta + \left\|g-g^{\delta}\right\|_{\mathcal{Y}} \le \tau\delta.
\end{align}\tag{9}\] Therefore, the discrepancy principle is satisfied at the pair \((\overline{N},k^*)\). By the definitions of \(n(\delta)\) and \(k(\delta)\) in Algorithm 1, we obtain \(n(\delta)\le \overline{N}=b(k^*),k(\delta)\le k^*\). Moreover, since \(k^*-1\ge
k_0\) and \(b(k^*-1)\le N(\delta)<b(k^*)\), the growth condition \(b(k^*)\le C_b b(k^*-1)\) yields \(n(\delta)\le C_b N(\delta)\). Since \(N(\delta)=\mathcal{O}\left((1/\varepsilon_1(\delta))^2\right)\), we conclude that \(n(\delta)=\mathcal{O}\left((1/\varepsilon_1(\delta))^2\right)\). Moreover, since there exists a constant \(C>0\) independent of \(\delta\) such that \(N(\delta)\le C\left(1/{\varepsilon_1(\delta)}\right)^2\) for all sufficiently small \(\delta\), and since \(b^{-1}\) is nondecreasing, we obtain \(k(\delta)\le k^*
= b^{-1}(N(\delta))
\le
b^{-1}\!\left(C\left(1/{\varepsilon_1(\delta)}\right)^2\right)\).
Thus Algorithm 1 terminates after at most \(b^{-1}\!\left(C\left(1/{\varepsilon_1(\delta)}\right)^2\right)\) iterations, with at most \(n(\delta)=\mathcal{O}\left(\left(1/{\varepsilon_1(\delta)}\right)^2\right)\) neurons. Moreover, if \(A\) is locally Hölder continuous of order \(\theta\) at
\(f^\dagger\) with respect to the \(H^1\)-norm, then, for sufficiently small \(\delta\), we may take \(\varepsilon_1(\delta)=
\left(\frac{(\tau-1)\delta}{L_A}\right)^{1/\theta}
<\gamma\). Therefore, \(n(\delta)=\mathcal{O}(\delta^{-2/\theta}),k(\delta)\le b^{-1}\!\left(C\delta^{-2/\theta}\right)\).
Finally, we prove the convergence of \(f_{n(\delta),k(\delta)}^\delta\) to \(f^\dagger\) in \(C(K)\) as \(\delta\to0\) by
contradiction. Assume that the assertion is false. Then there exist \(\varepsilon>0\) and a sequence \(\delta_k\searrow0\) such that \[\label{contr95InEq1} \left\|
f_{n(\delta_k),k(\delta_k)}^{\delta_k}-f^\dagger
\right\|_{C(K)}
\ge \varepsilon .\tag{10}\] For each \(k\), \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}\in X_{n(\delta_k),r(n(\delta_k))}\), denote its outer coefficients by \(\{a_j^{(k)}\}_{j=1}^{n(\delta_k)}\). Then, by the definition of \(X_{n,r}\), we have \(q_k:=
\frac{1}{n(\delta_k)}
\sum_{j=1}^{n(\delta_k)}
|a_j^{(k)}|
\le r(n(\delta_k))
\le B\). By Lemma 3, there exists a subsequence, denoted by \(\{f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}}\}\), and some \(f^*\in C(K)\cap\mathcal{B}_1\) such that \(f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}}\to
f^*\) in \(C(K)\). In particular, the convergence also holds in \(L^2(\Omega)\). By the discrepancy principle and the noise assumption, \[\|A(f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}})-g\|_{\mathcal{Y}}
\le
\|A(f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}})-g^{\delta_{k_i}}\|_{\mathcal{Y}}
+
\|g^{\delta_{k_i}}-g\|_{\mathcal{Y}}
\le
(\tau+1)\delta_{k_i}\to0.\] Since \(f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}}\to f^*\) in \(L^2(\Omega)\) and \(A:L^2(\Omega)\to\mathcal{Y}\)
is continuous, we obtain \(\|A(f^*)-g\|_{\mathcal{Y}}=0\). Hence \(A(f^*)=g\). Since \(A\) is injective, we get \(f^*=f^\dagger\) in \(L^2(\Omega)\). Since both \(f^*\) and \(f^\dagger\) are continuous on \(K\), this equality holds pointwise on \(K\). Therefore, \(f^*=f^\dagger\) in \(C(K)\), and hence \(f_{n(\delta_{k_i}),k(\delta_{k_i})}^{\delta_{k_i}}\to f^\dagger\) in \(C(K)\), which contradicts 10 . Therefore, \(f_{n(\delta),k(\delta)}^\delta \to f^\dagger\) in \(C(K)\) as \(\delta\to0\). ◻
Remark 5. For typical choices of \(b=b(k)\) we obtain explicit rates. For instance, if \(b(k) \asymp k^{p}\)5 for some \(p>0\), then \(k(\delta) = \mathcal{O}\big(\delta^{-2/(p\theta)}\big)\); if \(b(k) \asymp e^{q k}\) for some
\(q>0\), then \(k(\delta) = \mathcal{O}\big(\log(1/\delta)\big)\).
3.2 Case 2: without knowledge of \(c_{\rho}(f^{\dagger})\)↩︎
Now, we consider the inverse problem without any prior knowledge of the quantity \(c_{\rho}(f^{\dagger})\). To compensate for this, our analysis requires the forward operator \(A\) to
satisfy the \(\theta\)-degree Hölder continuity as given in Definition 1, and the introduction of a penalty term \(\mathcal{R}\) in the corresponding optimization problem. This additional term \(\mathcal{R}\) ensures the uniform boundedness of the minimizer throughout the iterations. To this end, we present
the following algorithm, which closely resembles Algorithm 1.
Figure 2: ENN2: a modified expanding neural network method for solving equation (1 )
The corresponding convergence properties of Algorithm 2, including its well-posedness and regularization behavior, are established in Theorem 6.
The result demonstrates that this modified algorithm can produce a sequence of regularized solutions converging to the exact solution in the appropriate function space, without requiring additional conditions on \(r(n)\).
Theorem 6. Suppose \(g^{\delta}\in \mathcal{Y}\) for any \(\delta>0\), and the forward operator \(A\) be a continuous injective
operator that is locally Hölder continuous at \(f^{\dagger}\) of order \(\theta \in (0,1]\) (as in Definition 1) with constants \(L_A > 0\) and \(\gamma_1 > 0\). Let \(r:\mathbb{N}\to(0,\infty)\) be a nondecreasing sequence
satisfying \(r(n)\to+\infty\) as \(n\to\infty\). Assume one of the following holds for the regularizer \(\mathcal{R}\):
\(\mathcal{R}=\mathcal{R}_H\) is weakly lower semicontinuous in \(H^1(\Omega)\). It is also locally Hölder continuous at \(f^\dagger\) of order
\(\theta\), with constants \(L_{\mathcal{R}_H}>0\) and \(\gamma_2>0\). Moreover, there exists \(C_{\mathcal{R}_H}>0\) such that \(C_{\mathcal{R}_H}\|f\|_{H^1(\Omega)}
\le
\mathcal{R}_H(f)
<+\infty\).
\(\mathcal{R}=\mathcal{R}_B\), where for a parametrized network \(f_n(\mathbf{x})
=
\frac{1}{n}\sum_{j=1}^n
a_j(\boldsymbol{b}_j^T\mathbf{x}+c_j)_+\) with \(\|\boldsymbol{b}_j\|_1+|c_j|=1\), we set \(\mathcal{R}_B(f_n):=c_{\rho_n}(f_n)=
\frac{1}{n}\sum_{j=1}^n |a_j|\). Here \(\rho_n\) is the empirical measure associated with the chosen parametrization.
Then, the following hold:
(Well-posedness of Algorithm 2) For any \(\delta >0\) the iteration process in Algorithm 2 terminates after finitely many steps \(k(\delta)\) with finite neurons \(n(\delta)\), i.e. \(k(\delta)<+\infty\) and \(n(\delta)<+\infty\). Moreover, there
exists a constant \(C>0\), independent of \(\delta\), such that \[\label{desire95estimate95kn} k(\delta) \le
b^{-1}\!\big(C\,\delta^{-2/\theta}\big),
\qquad
n(\delta) = \mathcal{O}\big(\delta^{-2/\theta}\big),
\quad \text{as } \delta\to0.\qquad{(2)}\]
(Regularization property of Algorithm 2) The approximate solutions generated by Algorithm 2 satisfy the following convergence properties as \(\delta\to0\): \[f_{n(\delta),k(\delta)}^\delta \to f^\dagger
\quad\text{in }L^2(\Omega),
\qquad \text{if } \mathcal{R}=\mathcal{R}_H,\] and \[f_{n(\delta),k(\delta)}^\delta \to f^\dagger
\quad\text{in }C(K),
\qquad \text{if } \mathcal{R}=\mathcal{R}_B.\]
Proof. Fix \(n\in\mathbb{N}\) and \(\delta>0\). Let \(\mathcal{L}_{n}^{\delta}(f)
:=\|A(f)-g^{\delta}\|_{\mathcal{Y}}+\beta_n \mathcal{R}(f)\) and \(\beta_n = c_0 n^{-\frac{\theta}{2}}\). For \(r(n)\in(0,\infty)\), we first establish the existence of minimizers for
[OPT-ENN2] for both regularizers, \(\mathcal{R}=\mathcal{R}_H\) and \(\mathcal{R}=\mathcal{R}_B\).
Define \(m_n^\delta := \inf_{f\in X_{n,r(n)}} \mathcal{L}_n^\delta(f).\) Choose a minimizing sequence \(\{f_{n,k}\}_{k\in\mathbb{N}}\subset X_{n,r(n)}\) such that \(\mathcal{L}_n^\delta(f_{n,k})\to m_n^\delta\) as \(k\to\infty\). Pick any \(f_0\in X_{n,r(n)}\) with \(\mathcal{L}_n^\delta(f_0)<\infty\) (e.g. \(f_0=0\)), and without loss of generality assume \[\mathcal{L}_n^\delta(f_{n,k})\le \mathcal{L}_n^\delta(f_0)+1, \qquad
\text{for all }k\in\mathbb{N}.\] This directly implies \(\beta_n \mathcal{R}(f_{n,k}) \le \mathcal{L}_n^\delta(f_{n,k})\le \mathcal{L}_n^\delta(f_0)+1\), and hence \(\mathcal{R}(f_{n,k}) \le
\frac{\mathcal{L}_n^\delta(f_0)+1}{\beta_n}
= \frac{\mathcal{L}_n^\delta(f_0)+1}{c_0}\, n^{\theta/2}
=: C_{n,\delta} <\infty\). We now distinguish two choices of \(\mathcal{R}\).
Case A: \(\mathcal{R}=\mathcal{R}_H\). Since \(\mathcal{R}_H(f)\ge C_{\mathcal{R}_H}\|f\|_{H^1(\Omega)}\), the bound \(\mathcal{R}_H(f_{n,k})\le
C_{n,\delta}\) implies that \(\{f_{n,k}\}\) is bounded in \(H^1(\Omega)\). Hence, there exists a subsequence (not relabeled) and some \(f_n^*\in
H^1(\Omega)\) such that \[f_{n,k}\rightharpoonup f_n^* \;\text{in }H^1(\Omega),
\qquad
f_{n,k}\to f_n^* \;\text{in }L^2(\Omega).\] Since \(X_{n,r(n)}\) is closed in \(L^2(\Omega)\) (noting that for \(r(n)<\infty\), closedness follows
from compactness), we have \(f_n^*\in X_{n,r(n)}\). By the continuity of \(f\mapsto \|A(f)-g^\delta\|_{\mathcal{Y}}\) with respect to \(L^2(\Omega)\) and the
weak lower semicontinuity of \(\mathcal{R}_H\), \[\mathcal{L}_n^\delta(f_n^*)
\le \liminf_{k\to\infty} \mathcal{L}_n^\delta(f_{n,k})
= m_n^\delta,\] which shows that \(f_n^*\) is a minimizer of \(\mathcal{L}_n^\delta\) on \(X_{n,r(n)}\) for \(r(n)\in(0,\infty)\).
Case B: \(\mathcal{R}=\mathcal{R}_B\). Write \(f_{n,k}(\mathbf{x})=\frac{1}{n}\sum_{j=1}^n a_j^{(k)}(\boldsymbol{b}_j^{(k)\,T}\mathbf{x}+c_j^{(k)})_+\) with \(\|\boldsymbol{b}_j^{(k)}\|_1+|c_j^{(k)}|=1\). From \(\mathcal{R}_B(f_{n,k})\le C_{n,\delta}\) and the definition of \(\mathcal{R}_B\), we obtain \[|a_j^{(k)}|
\le n\,\mathcal{R}_B(f_{n,k})
\le nC_{n,\delta}
=:R_{n,\delta}
\qquad (j=1,\dots,n).\] Define the radius \(\bar r_{n,\delta}:=\min\{r(n),R_{n,\delta}\}\in(0,\infty).\) Then \(f_{n,k}\in X_{n,\bar r_{n,\delta}}\) for all \(k\). By Lemma 1 and its proof, there exist a subsequence \(\{f_{n,k_\ell}\}\) and some \(f_n^*\in X_{n,\bar r_{n,\delta}}\subset X_{n,r(n)}\) such that \(f_{n,k_\ell}\to f_n^*\) in \(C(K)\) with the associated parameters \(\{(a_j^{(k)},\boldsymbol{b}_j^{(k)},c_j^{(k)})\}\) converging as well. Hence \(\mathcal{R}_B(f_{n,k})\to \mathcal{R}_B(f_n^*)\). Moreover, since \(A:L^2(\Omega)\to\mathcal{Y}\) is continuous, it follows that \(\|A(f_{n,k})-g^\delta\|_{\mathcal{Y}}
\to \|A(f_n^*)-g^\delta\|_{\mathcal{Y}}.\) Therefore, \[\mathcal{L}_n^\delta(f_n^*)
=
\lim_{k\to\infty}\mathcal{L}_n^\delta(f_{n,k})
=
\inf_{f\in X_{n,r(n)}}\mathcal{L}_n^\delta(f).\] Thus \(f_n^*\) is a minimizer of \(\mathcal{L}_n^\delta\) over \(X_{n,r(n)}\).
Now, we show that for any \(\delta >0\), \(k(\delta)<+\infty\) and \(n(\delta)<+\infty\). By Proposition 3, for \(f^\dagger\in \mathcal{B}_1\), and any \(\delta>0\), if \(n>N_1(\delta):=\max\{
(C(\Omega,d)\|f^{\dagger}\|_{\mathcal{B}_1})^2\cdot(\frac{3(L_A+L_{\mathcal{R}_H})}{(\tau-1)\delta})^{\frac{2}{\theta}},(\frac{C(\Omega,d)\|f^{\dagger}\|_{\mathcal{B}_1}}{\min\{\gamma_1,\gamma_2\}})^2\}\), there exists a two-layer network \(f_{n}\in X_{n,c_{\rho}(f^{\dagger})}\) with \(c_{\rho_n}(f_n)\le c_{\rho}(f^{\dagger})\) such that \[\|f^{\dagger}-f_{n}\|_{H^1(\Omega)} <\min\left\{
\left(\frac{\tau -1}{3(L_A+L_{\mathcal{R}_H})} \delta\right)^{\frac{1}{\theta}},\gamma_1,\gamma_2\right\}.\]
For the regularizer \(\mathcal{R}=\mathcal{R}_H\), since the functional \(\mathcal{R}_H(\cdot)\) is locally Hölder continuous at \(f^\dagger\) of the same
order \(\theta\) with constants \(L_{\mathcal{R}_H}>0\) and \(\gamma_2>0\), since it holds \(\|f^{\dagger}-f_n\|_{H^1(\Omega)}\le \gamma_2\) when \(n\ge N_1\), we obtain \[|\mathcal{R}_H(f_{n})-\mathcal{R}_H(f^\dagger)| \le
L_{\mathcal{R}_H}\|f^{\dagger}-f_{n}\|_{H^1(\Omega)}^{\theta} \le L_{\mathcal{R}_H} \frac{\tau -1}{3(L_A+L_{\mathcal{R}_H})} \delta.\] By the definition of \(r(n)\), we have \(r(n)\to+\infty\). Hence we define \[N_2(\delta)
:=
\max\left\{
c_0^{2/\theta},
r^{-1}\bigl(c_\rho(f^\dagger)\bigr),
b(k_0),
\left(
\frac{3c_0\mathcal{R}_H(f^\dagger)}{(\tau-1)\delta}
\right)^{2/\theta},
\left(
\frac{3c_0c_\rho(f^\dagger)}{2(\tau-1)\delta}
\right)^{2/\theta}
\right\},\] where \(r^{-1}\) denotes the generalized inverse, i.e., \(r^{-1}(N):= \inf \{ n \in \mathbb{N}:~ r(n) \ge N \}\).
Denote \(N:=\max\{N_1(\delta),N_2(\delta)\}\). Then, for every integer \(n\ge N\), we have \[r(n)\ge c_{\rho}(f^{\dagger}),\quad c_0
n^{-\frac{\theta}{2}}\mathcal{R}_H(f^{\dagger})\le \frac{\tau -1}{3} \delta,\quad c_0n^{-\frac{\theta}{2}}c_{\rho}(f^{\dagger})\le \frac{2(\tau-1)}{3}\delta,\quad c_0n^{-\theta/2}\le 1.\] Furthermore, we have \(N(\delta)=\mathcal{O}(\delta^{-2/\theta})\) as \(\delta\to 0\). Equivalently, there exists a constant \(C_{\max}>0\), independent of \(\delta\), such that, for all sufficiently small \(\delta\), \(N(\delta)\le C_{\max}\delta^{-2/\theta}\).
These discussions indicate that for any \(\delta>0\) and \(n\ge N\), there exists a two-layer network \(f_n\in X_{n,c_{\rho}(f^{\dagger})}\subset
X_{n,r(n)}\) with \(\mathcal{R}_B(f_n)=c_{\rho_n}(f_n)\le c_{\rho}(f^{\dagger})\) such that \[\|f^{\dagger}-f_n\|_{H^1(\Omega)} < \min\left\{ \left(\frac{\tau
-1}{3(L_A+L_{\mathcal{R}_H})} \delta\right)^{\frac{1}{\theta}},\gamma_1,\gamma_2\right\}, \quad c_0n^{-\frac{\theta}{2}}\mathcal{R}_B(f_n)\le c_0n^{-\frac{\theta}{2}}c_{\rho}(f^{\dagger})\le \frac{2(\tau-1)}{3}\delta,\] and \[c_0n^{-\frac{\theta}{2}}\mathcal{R}_H(f_n)\le c_0 n^{-\frac{\theta}{2}}\left( |\mathcal{R}_H(f^\dagger)| + | \mathcal{R}_H(f_n) - \mathcal{R}_H(f^\dagger)| \right)\le \frac{2(\tau-1)}{3}\delta.\] Let \(k^*:=\min\{k\in \mathbb{N}^{+}|b(k)>N\}\). Then, by definition, \(b(k^*-1)\le N<b(k^*)\) and \(k^*>k_0\). Thus, for \(\overline{N}=b(k^*)>N\), \(\mathcal{R}=\mathcal{R}_H\) or \(\mathcal{R}=\mathcal{R}_B\). By the definition of \(f_{\overline{N},k^*}\) in Algorithm 2, and the locally Hölder-type continuity of \(A\) at \(f^{\dagger}\), we have
\[\begin{align}
J_{\overline{N},k^*}^{\delta} &=\left\|A(f_{\overline{N},k^*}^{\delta})-g^{\delta}\right\|_{\mathcal{Y}} \leq \left\|A(f_{\overline{N}})-g^{\delta}\right\|_{\mathcal{Y}}+ c_0{\overline{N}}^{-\frac{\theta}{2}}\mathcal{R}(f_{\overline{N}}) \notag\\
&\leq L_A \left\|f_{\overline{N}}-f^\dagger\right\|_{H^1(\Omega)}^{\theta} + \left\|A(f^\dagger)-g^{\delta}\right\|_{\mathcal{Y}} + c_0{\overline{N}}^{-\frac{\theta}{2}}\mathcal{R}(f_{\overline{N}}) \notag\\
\label{Jnlessdelta}
&\leq L_A \frac{\tau -1}{3(L_A+L_{\mathcal{R}_H})} \delta +\left\|g-g^{\delta}\right\|_{\mathcal{Y}} + \frac{2(\tau-1)}{3}\delta \le \tau\delta.
\end{align}\tag{11}\] By the definition of the stopping indices \(n(\delta)\) and \(k(\delta)\) in the algorithm, this implies \(n(\delta)\le
\overline{N}= b(k^*)\) and \(k(\delta)\le k^*\). Combining these with the monotonicity of \(b\) and the bound \(b(k^*) \le C_b b(k^*-1) \le C_b N\),
and recalling that \(N = \mathcal{O}(\delta^{-2/\theta})\), we finally obtain the desired estimate ?? .
Next, we prove the regularizing property of the algorithm. Let \(\delta_k\searrow0\). We first consider the case where \(\{n(\delta_k)\}\) is bounded. Then, up to a subsequence, there
exists \(N_0\in\mathbb{N}\) such that \(n(\delta_k)\le N_0\) for all \(k\). Hence \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}
\in\bigcup_{n=1}^{N_0}X_{n,r(n)}\). Since each \(X_{n,r(n)}\) is compact in \(C(K)\) by Lemma 1, the finite union above is compact in \(C(K)\), and hence also in \(L^2(\Omega)\). Therefore, after passing to a subsequence, there exists \(f^*\in C(K)\) such that \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}\to f^*\) in \(C(K)\), and consequently also in \(L^2(\Omega)\).
By the discrepancy principle and the noise assumption, \[\|A(f_{n(\delta_k),k(\delta_k)}^{\delta_k})-g\|_{\mathcal{Y}} \le \|A(f_{n(\delta_k),k(\delta_k)}^{\delta_k})-g^{\delta_k}\|_{\mathcal{Y}} +
\|g^{\delta_k}-g\|_{\mathcal{Y}} \le (\tau+1)\delta_k\to0.\] Since \(A:L^2(\Omega)\to\mathcal{Y}\) is continuous, we obtain \(A(f^*)=g\). The injectivity of \(A\) gives \(f^*=f^\dagger\). Thus the desired convergence follows in this case.
It remains to consider the case where \(\{n(\delta_k)\}\) is unbounded. Passing to a subsequence if necessary, we may assume that \(n(\delta_k)\to\infty\). In this case, by Proposition 3, for each \(n(\delta_k)\), there exists a two-layer network \(f_{n(\delta_k)}\in
X_{n(\delta_k),c_\rho(f^\dagger)}\) such that \[\label{appresult95ndelta} \left\|f^{\dagger}-f_{n(\delta_k)}\right\|_{H^1(\Omega)}\leq \frac{C(\Omega,
d)}{\sqrt{n(\delta_k)}}\|f^{\dagger}\|_{\mathcal{B}_1},\qquad c_{\rho_{n(\delta_k)}}(f_{n(\delta_k)})\le c_\rho(f^\dagger).\tag{12}\] Since \(n_k\to\infty\) and \(r(n)\to\infty\), for all sufficiently large \(k\), \(f_{n(\delta_k)}\in X_{n(\delta_k),c_\rho(f^\dagger)} \subset X_{n(\delta_k),r(n(\delta_k))}\), and the local
Hölder continuity assumptions on \(A\) and \(\mathcal{R}_H\) are applicable to \(f_{n(\delta_k)}\).
We first consider the case \(\mathcal{R} = \mathcal{R}_H\). By the minimizing property of \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}\), the coercivity \(\mathcal{R}_H(f)\ge C_{\mathcal{R}_H}\|f\|_{H^1(\Omega)}\), 12 , and the local Hölder continuity of \(A\) and \(\mathcal{R}_H\), we have \[\begin{align}
&c_0 C_{\mathcal{R}_H}\, (n(\delta_k))^{-\frac{\theta}{2}} \, \| f_{n(\delta_k),k(\delta_k)}^{\delta_k} \|_{H^1(\Omega)} \le \| A(f_{n(\delta_k)}) - g^{\delta_k} \|_{\mathcal{Y}} + c_0 \, (n(\delta_k))^{-\frac{\theta}{2}} \,
\mathcal{R}_H(f_{n(\delta_k)}) \\
&\quad \le \| A(f_{n(\delta_k)}) - A(f^\dagger) \|_{\mathcal{Y}} + \| A(f^\dagger) - g^{\delta_k} \|_{\mathcal{Y}}
+ c_0 (n(\delta_k))^{-\frac{\theta}{2}} \bigl( | \mathcal{R}_H(f^\dagger) | + | \mathcal{R}_H(f_{n(\delta_k)}) - \mathcal{R}_H(f^\dagger) | \bigr) \\
&\quad \le \Bigl( L_A ( C(\Omega, d) \| f^{\dagger} \|_{\mathcal{B}_1} )^{\theta} + c_0 | \mathcal{R}_H(f^\dagger) | + c_0 L_{\mathcal{R}_H} ( C(\Omega, d) \| f^{\dagger} \|_{\mathcal{B}_1} )^{\theta} \Bigr) (n(\delta_k))^{-\frac{\theta}{2}} +
\delta_k.
\end{align}\] Since \(n(\delta_k) \le C_b N(\delta_k)\), and \(N(\delta_k)\le C_{\max}\delta_k^{-2/\theta}\) for all sufficiently small \(\delta_k\),
it follows that \(\delta_k (n(\delta_k))^{\theta/2} \le (C_bC_{\max})^{\theta/2}\). Hence \(\{f_{n(\delta_k),k(\delta_k)}^{\delta_k}\}\) is uniformly bounded in \(H^1(\Omega)\) for all sufficiently small \(\delta_k\). Therefore, by Lemma 2, there exists a subsequence,
still denoted by \(\{f_{n(\delta_k),k(\delta_k)}^{\delta_k}\}\), and some \(f^*\in L^2(\Omega)\) such that \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}\to f^*\)
in \(L^2(\Omega)\). The discrepancy principle, the noise assumption, and the continuity of \(A\) imply \(A(f^*)=g\). By injectivity, \(f^*=f^\dagger\). Hence \(f_{n(\delta),k(\delta)}^\delta \to f^\dagger\) in \(L^2(\Omega)\) as \(\delta\to 0\).
For \(\mathcal{R}=\mathcal{R}_B\), using 12 , the minimizing property, the local Hölder continuity of \(A\), \(\mathcal{R}_B(f_{n(\delta_k)})=c_{\rho_n}(f_{n(\delta_k)})\le c_\rho(f^\dagger)\), and the bound \(\delta_k (n(\delta_k))^{\theta/2} \le (C_bC_{\max})^{\theta/2}\), we obtain \[\begin{align}
\mathcal{R}_B(f_{n(\delta_k),k(\delta_k)}^{\delta_k})
&\le
\frac{1}{c_0}n(\delta_k)^{\frac{\theta}{2}}
\|A(f_{n(\delta_k)})-g^{\delta_k}\|_{\mathcal{Y}}
+\mathcal{R}_B(f_{n(\delta_k)}) \\
&\le
\frac{1}{c_0}
\left[
L_A(C(\Omega,d)\|f^\dagger\|_{\mathcal{B}_1})^\theta
+
(C_bC_{\max})^{\frac{\theta}{2}}
\right]
+c_\rho(f^\dagger).
\end{align}\] Hence \(\mathcal{R}_B(f_{n(\delta_k),k(\delta_k)}^{\delta_k})\) are uniformly bounded for all sufficiently small \(\delta_k\). By the definition of \(\mathcal{R}_B\), \(q_k:=n(\delta_k)^{-1}\sum_{j=1}^{n(\delta_k)}|a_j^{(k)}|\) are also uniformly bounded. By Lemma 3, there exists a subsequence, still denoted by \(\{f_{n(\delta_k),k(\delta_k)}^{\delta_k}\}\), and some \(f^*\in C(K)\cap\mathcal{B}_1\) such
that \(f_{n(\delta_k),k(\delta_k)}^{\delta_k}\to f^*\) in \(C(K)\). As above, the limit must be \(f^\dagger\). Since both \(f^*\) and \(f^\dagger\) are continuous on \(K\), the equality also holds in \(C(K)\). Therefore, \(f_{n(\delta),k(\delta)}^\delta
\to f^\dagger\) in \(C(K)\) as \(\delta\to 0\). ◻
Theorems 1 and 2 indicate that the proposed algorithms provide a constructive strategy for building neural networks that serve as regularization solutions, offering explicit answers to the foundational questions (Q1-Q3)
from the introduction:
Answer to Q1: The proposed algorithms construct neural network approximations that yield stable regularized solutions for ill-posed inverse problems. The discrepancy principle provides an a posteriori stopping rule based on
the noisy data, and the convergence results show that the selected approximations converge to the exact solution as the noise level tends to zero. In this sense, the selected neural network acts as a regularized solution.
Answer to Q2: The theorems quantify how the sufficient network width depends on the noise level. In particular, when the Barron-space approximation error has order \(\mathcal{O}(n^{-1/2})\) and the
forward operator is locally Hölder continuous with exponent \(\theta\), the sufficient width can be chosen as \(n(\delta)=\mathcal{O}(\delta^{-2/\theta})\). This shows that the network size
should be coupled with the noise level rather than enlarged independently. The numerical results in Section 5 further illustrate that, for noisy data, the reconstruction error need not decrease monotonically as the number
of neurons increases.
Answer to Q3: The expanding neural network methodology increases the number of neurons gradually and stops at the first architecture, along the prescribed expansion path, that satisfies the discrepancy principle. Thus the algorithm
automatically selects a sufficiently small and stable network size, balancing approximation accuracy, stability, and computational complexity.
4 A neural network-based Tikhonov regularization scheme↩︎
This section addresses a method different from the expanding neural network scheme discussed earlier. In the previous framework, the number of neurons \(n\) serves as an implicit regularization parameter, making the
quality of the solution highly sensitive to the choice of stopping criterion. By contrast, the variational regularization framework developed here incorporates an explicit parameter \(\alpha\), providing more direct and
stable control over the solution. While by the general principles of variational regularization [37], our theory is established
independently within the neural network setting. We propose a neural network-based Tikhonov regularization scheme (Tikhonov-NN) to address the ill-posed problem 1 . Within this framework, we establish convergence of the regularized
solutions and, under standard variational source conditions, derive explicit convergence rates that characterize the efficiency of the method. The core idea of our scheme is to seek two layer neural networks as the regularized solution that are obtained as
the minimizer of the functional \[\label{Tikfunc}
f_{n,\alpha}^{\delta}:=\arg\min_{f\in X_{n,\infty}}\mathcal{T}_\alpha(f,g^{\delta}),\quad \mathcal{T}_\alpha(f,g^{\delta}):=\mathcal{D}(A(f),g^{\delta})+\alpha \mathcal{R}_n(f),\tag{13}\] where \(X_{n,\infty}\) is defined in 8 , \(\mathcal{D}:(\mathcal{Y},\mathcal{Y}) \to \mathbb{R}_+\) is an appropriate similarity measure in the data space enforcing
data consistency, \(\alpha>0\) is the regularization parameter and \(\mathcal{R}_n(f):X_{n,\infty}\to\mathbb{R}_+\) is the regularization stabilizer term, which is designed here by
\[\label{penalty}
\mathcal{R}_n(f):= \varphi(c_{\rho_n}(f)), \quad c_{\rho_n}(f)\equiv \frac{1}{n} \sum_{i=1}^n\left|a_i\right|(\left\|\boldsymbol{b_i}\right\|_1+\left|c_i\right|)=\frac{1}{n} \sum_{i=1}^n\left|a_i\right|,\tag{14}\] where \(\varphi: \mathbb{R}_+\to \mathbb{R}_+\) denotes the feature selection function and \(\rho_n\) is the empirical measure associated with the chosen parametrization. To this end, we summarize the
main assumptions underlying Tikhonov regularization 13 . These assumptions are standard within the regularization theory of abstract inverse problems formulated in normed spaces; see, for instance, [44].
Assumption 1.
Data consistency term \(\mathcal{D}:\mathcal{Y}\times \mathcal{Y} \to \mathbb{R}_+\) and forward operator \(A\) satisfy the following six conditions:
For some \(\tau_D \geq 1\) we have \(\forall g_0, g_1, g_2 \in \mathcal{Y}: \mathcal{D}\left(g_0, g_1\right) \leq \tau_D \mathcal{D}\left(g_0, g_2\right)+\tau_D \mathcal{D}\left(g_2,
g_1\right)\);
For every fixed \(h\in\mathcal{Y}\), \(g_k\to g\) implies \(\mathcal{D}(h,g_k)\to\mathcal{D}(h,g)\);
The functional \((f,g)\mapsto \mathcal{D}(A(f),g)\) is sequentially lower semi-continuous with respect to \(f\) and \(g\);
For any \(f'\in \mathcal{B}_1\) and exact solution \(f^*\in \mathcal{B}_1\) of \(A(f)=g\), it holds that \[\label{conditionA14} \mathcal{D}(A(f'), g)\le K_A\|f'-f^*\|_{H^1(\Omega)}^s\qquad{(3)}\] for some constants \(s,K_A\ge 0\).
\(\varphi:[0,\infty)\to[0,\infty)\) is continuous, nondecreasing, and satisfies \(\varphi(t)\to\infty\) as \(t\to\infty\).
Note that the squared norm \(\| \cdot - \cdot \|_{L^2(\Omega)}^2\) satisfies Condition (A1) when \(A\) is an \(L_A\)-Lipschitz continuous operator.
Specifically, conditions (A1.1)-(A1.3) in Assumption 1 are straightforward to verify, while (A1.4) and (A1.5) hold with \(K_A = L_A^2\) and \(s = 2\), as guaranteed by the Lipschitz continuity of \(A\) and Proposition 2.
We are now in a position to demonstrate that the Tikhonov-NN method is a well-posed and convergent scheme, characterized by existence, stability, and convergence. In particular, Theorem 7 establishes that, by employing 13 , one can construct a sequence that converges to a \(\mathcal{R}\)-minimizing solution defined in ?? , as \(n \to \infty\) and \(\delta \to 0\), provided that the regularization parameter is chosen appropriately as in ?? .
Theorem 7 (Well-posedness). Let Assumption 1 be satisfied. The following assertions hold true:
Existence: for each \(n\in\mathbb{N}\), every \(g \in \mathcal{Y}\) and every \(\alpha>0\), there exists a minimizer \(f_n^*\) of \(\mathcal{T}_\alpha(f,g)\) in \(X_{n,\infty}\).
Stability: let \(n\in\mathbb{N}\) and \(\alpha>0\) be fixed. If \(g_k \to g\) as \(k\to \infty\) and \(f_{n,k} \in \arg\min_{f\in X_{n,\infty}}\mathcal{T}_\alpha(f,g_k)\), then there exists a subsequence \(\{f_{n,k_{\ell}}\}_{\ell=1}^\infty\) and some \(f_n^{*}\in
\arg\min_{f\in X_{n,\infty}}\mathcal{T}_\alpha(f,g)\) such that \[f_{n,k_{\ell}}\to f^{*}_n \quad \text{in } C(K), \quad \text{as }\ell\to \infty.\]
Convergence: let \(f \in \mathcal{B}_1\) and \(g:=A(f)\). Let \(\{g_k\}_{k=1}^{\infty}\subset\mathcal{Y}\) satisfy \(\mathcal{D}\left(g, g_k\right) \leq \delta_k\) for some sequence \(\delta_k \searrow 0\) as \(k \to \infty\). For each \(n,k\in\mathbb{N}\), let \(\alpha_{k,n}>0\) be a regularization parameter and choose \(f_{n,k} \in \arg\min_{f\in
X_{n,\infty}}\mathcal{T}_{\alpha_{k,n}}(f,g_k)\). Assume that the parameter choice \(\alpha_{k,n}=\alpha(\delta_k,n)\) satisfies \[\label{choicealpha} \alpha_{k,n}
\to 0, \qquad \alpha_{k,n}^{-1} (\delta_k + n^{-s/2})\to 0 \quad \text{as } k \to \infty,\;n\to\infty.\qquad{(4)}\] Then, for every sequence \(\{(n_\ell,k_\ell)\}_{\ell=1}^{\infty}\) satisfying \(n_\ell\to\infty,k_\ell\to\infty\), there exists a subsequence of \(\{f_{n_\ell,k_\ell}\}_{\ell=1}^{\infty}\), still denoted by \(\{f_{n_\ell,k_\ell}\}_{\ell=1}^{\infty}\) such that \[f_{n_\ell,k_\ell} \to f^{\dagger} \quad\text{in } C(K), \quad\text{as } n,\ell\to\infty,\] where \(f^{\dagger}\) is an \(\mathcal{R}\)-minimizing solution of 1 , i.e. \[\label{deffdagger} f^{\dagger} \in \arg \min \left\{
\mathcal{R}(f):=\varphi(\|f\|_{\mathcal{B}_1}) \;\big|\;f \in \mathcal{B}_1,\;A(f)=g \right\}.\qquad{(5)}\]
Proof. (a) Fix \(n\in\mathbb{N}\), \(g\in\mathcal{Y}\), and \(\alpha>0\). Let \(f_0\in X_{n,\infty}\) with
\(\mathcal{T}_\alpha(f_0,g^\delta)<\infty\) (e.g. \(f_0=0\)). Similar to the proof of Case B in Theorem 6, there
exists a minimizing sequence \(\{f_{n,k}\}_{k\in\mathbb{N}}\subset X_{n,\infty}\) such that \[\alpha \mathcal{R}_n(f_{n,k})\le \mathcal{T}_n^\delta(f_{n,k})\le \mathcal{T}_n^\delta(f_0)+1, \quad
\forall k\in\mathbb{N}.\] This implies \(\mathcal{R}_n(f_{n,k})
\le \frac{\mathcal{T}_\alpha(f_0,g^\delta)+1}{\alpha}
=:C_{n,\delta}\). Recall that \(\|\boldsymbol{b}_j\|_1+|c_j|=1\) and \(\mathcal{R}_n(f_{n,k})=\varphi\left(c_{\rho_{n,k}}(f_{n,k})\right) =
\varphi\left(\frac{1}{n}\sum_{j=1}^{n}|a_j^{(k)}|\right)\). By the monotonicity of \(\varphi\) and the definition of its generalized inverse \(\varphi^{-1}\), we obtain \[|a_j^{(k)}|
\le n\,c_{\rho_{n,k}}(f_{n,k})
\le n\,\varphi^{-1}(C_{n,\delta})
=:R_{n,\delta},
\quad j=1,\dots,n.\] Thus \(f_{n,k}\in X_{n,R_{n,\delta}}\) for all \(k\). Since \(R_{n,\delta}<\infty\), Lemma 1 implies that \(X_{n,R_{n,\delta}}\) is sequentially compact in \(C(K)\) and \(L^2(\Omega)\). This implies that the sequence \(\{f_{n,k}\}\) admits a subsequence, denoted again by \(\{f_{n,k_\ell}\}\), and some \(f_n^*\in X_{n,R_{n,\delta}}\) such that \(f_{n,k_\ell} \to f_n^*\) in \(C(K)\), and hence also in \(L^2(\Omega)\). Moreover, by
the proof of Lemma 1, the corresponding normalized parameters may be chosen to converge. Since \(\varphi\) is continuous and
\(\mathcal{R}_n(f)=\varphi(c_\rho(f))\), we obtain \(\mathcal{R}_n(f_{n,k_\ell})\to \mathcal{R}_n(f_n^*)\). Combined with the lower semi-continuity of the functional \(\mathcal{T}_{\alpha}\) from Assumption 1, we obtain \[\mathcal{T}_{\alpha}(f_n^*,g) \le \liminf_{\ell\to\infty}
\mathcal{T}_{\alpha}(f_{n,k_\ell},g) = \inf_{f\in X_{n,\infty}} \mathcal{T}_{\alpha}(f,g),\] and thus \(f_n^*\) is a minimizer of \(\mathcal{T}_{\alpha}(\cdot,g)\) on \(X_{n,\infty}\).
(b) By the minimizing property of \(f_{n,k}\), we have \(\mathcal{T}_\alpha(f_{n,k},g_k)
\le
\mathcal{T}_\alpha(0,g_k)\). Since \(\alpha>0\) and \(\mathcal{R}_n\ge0\), it follows that \[\alpha\mathcal{R}_n(f_{n,k})
\le
\mathcal{T}_\alpha(f_{n,k},g_k)
\le
\mathcal{T}_\alpha(0,g_k).\] Moreover, by the continuity of \(g\mapsto \mathcal{D}(A(0),g)\) and \(g_k\to g\), the sequence \(\{\mathcal{T}_\alpha(0,g_k)\}\) is bounded. Hence \(\{\mathcal{R}_n(f_{n,k})\}\) is uniformly bounded. As in the proof of part (a), this implies that there exists a radius \(R_{n,\alpha}^1<\infty\) such that \(f_{n,k}\in X_{n,R_{n,\alpha}^1},\) for any \(k\in\mathbb{N}\). By Lemma 1, after passing to a subsequence, there exists \(f_n^*\in X_{n,R_{n,\alpha}^1}\subset X_{n,\infty}\) such that \(f_{n,k_\ell}\to
f_n^*\) in \(C(K)\). Moreover, by the same parameter-convergence argument as in part (a), we have \(\mathcal{R}_n(f_{n,k_\ell})\to \mathcal{R}_n(f_n^*)\). Using the sequential lower
semi-continuity of \((f,g)\mapsto\mathcal{D}(A(f),g)\) and the convergence \(g_{k_\ell}\to g\), we have \[\mathcal{T}_\alpha(f_n^*,g)
\le
\liminf_{\ell\to\infty}
\mathcal{T}_\alpha(f_{n,k_\ell},g_{k_\ell}).\] On the other hand, for every \(f\in X_{n,\infty}\), we have \(\mathcal{T}_\alpha(f_{n,k_\ell},g_{k_\ell})\le\mathcal{T}_\alpha(f,g_{k_\ell})\). Letting \(\ell\to\infty\) and using the continuity with respect to \(g\), we
get \[\mathcal{T}_\alpha(f_n^*,g)
\le
\mathcal{T}_\alpha(f,g),
\qquad
\forall f\in X_{n,\infty}.\] Thus \(f_n^*\in\arg\min_{f\in X_{n,\infty}}\mathcal{T}_\alpha(f,g)\).
(c) Let \(\{(n_\ell,k_\ell)\}_{\ell=1}^{\infty}\) be any sequence such that \(n_\ell\to\infty, k_\ell\to\infty\). For simplicity, set \(f_\ell:=f_{n_\ell,k_\ell},
\alpha_\ell:=\alpha_{k_\ell,n_\ell}\). We show that a subsequence of \(\{f_\ell\}\) converges in \(C(K)\) to an \(\mathcal{R}\)-minimizing
solution.
Let \(\overline{f}\in\mathcal{B}_1\) be an arbitrary solution of \(A(\overline{f})=g\). For any \(\varepsilon>0\), choose a representing probability
measure \(\rho\) such that \(c_\rho(\overline{f})\le (1+\varepsilon)\|\overline{f}\|_{\mathcal{B}_1}\). By Proposition 3, for every \(n_\ell\) there exists \(\overline{f}_{n_\ell}\in X_{n_\ell,c_{\rho}(f_{n_\ell})}\) such that \[c_{\rho_{n_\ell}}(\overline{f}_{n_\ell})
\le c_\rho(\overline{f}),
\qquad
\|\overline{f}-\overline{f}_{n_\ell}\|_{H^1(\Omega)}
\le
\frac{C(\Omega,d)}{\sqrt{n_\ell}}\|\overline{f}\|_{\mathcal{B}_1}.\] Using the minimizing property of \(f_\ell\), we have \[\mathcal{D}(A(f_\ell),g_{k_\ell})
+
\alpha_\ell \mathcal{R}_n(f_\ell)
\le
\mathcal{D}(A(\overline{f}_{n_\ell}),g_{k_\ell})
+
\alpha_\ell \mathcal{R}_n(\overline{f}_{n_\ell}).\] Since \(\mathcal{R}_n(\overline{f}_{n_\ell})
=
\varphi(c_{\rho_{n_\ell}}(\overline{f}_{n_\ell}))
\le
\varphi(c_\rho(\overline{f}))\), and by Assumption 1, \[\mathcal{D}(A(\overline{f}_{n_\ell}),g_{k_\ell})
\le
\tau\mathcal{D}(A(\overline{f}_{n_\ell}),A(\overline{f}))
+
\tau\mathcal{D}(g,g_{k_\ell})
\le
\tau C n_\ell^{-s/2}+\tau\delta_{k_\ell},\] where \(C=K_A\bigl(C(\Omega,d)\|\overline{f}\|_{\mathcal{B}_1}\bigr)^s\), we obtain \(\mathcal{D}(A(f_\ell),g_{k_\ell})\to0\) and
\[\label{ApenCeq1} \mathcal{R}_n(f_\ell)
\le
\tau\alpha_\ell^{-1}
\bigl(Cn_\ell^{-s/2}+\delta_{k_\ell}\bigr)
+
\varphi(c_\rho(\overline{f})).\tag{15}\] According to the parameter choice ?? , the right-hand side is bounded, and hence \(\{\mathcal{R}_n(f_\ell)\}\) is bounded. Since \(\mathcal{R}_n(f_\ell)=\varphi(c_{\rho_{n_\ell}}(f_\ell))\) and \(\varphi(t)\to\infty\) as \(t\to\infty\), the corresponding coefficient averages \(q_\ell:=\frac{1}{n_\ell}\sum_{j=1}^{n_\ell}|a_j^{(\ell)}|=c_{\rho_{n_\ell}}(f_\ell)\) are uniformly bounded. Therefore, by Lemma 3, there exists a subsequence, still denoted by \(\{f_\ell\}\), and some \(f^*\in C(K)\cap\mathcal{B}_1\) such that \(f_\ell\to f^*\) in \(C(K)\). Moreover, \(\|f^*\|_{\mathcal{B}_1}\le\liminf_{\ell\to\infty}c_{\rho_{n_\ell}}(f_\ell)\).
By the sequential lower semi-continuity of \(\mathcal{D}\) and the convergence \(g_{k_\ell}\to g\), we get \[\mathcal{D}(A(f^*),g)
\le
\liminf_{\ell\to\infty}
\mathcal{D}(A(f_\ell),g_{k_\ell})
=0.\] Since \(\mathcal{D}\) is nonnegative and definite, this implies \(A(f^*)=g\). It remains to show that \(f^*\) is \(\mathcal{R}\)-minimizing. From the estimate 15 , we have \(\limsup_{\ell\to\infty}\mathcal{R}_n(f_\ell)\le\varphi(c_\rho(\overline{f}))\). Using Lemma 3 and the monotonicity and continuity of \(\varphi\), we obtain \[\mathcal{R}(f^*)
=
\varphi(\|f^*\|_{\mathcal{B}_1})
\le
\liminf_{\ell\to\infty}
\varphi(c_{\rho_{n_\ell}}(f_\ell))
=
\liminf_{\ell\to\infty}\mathcal{R}_n(f_\ell)
\le
\varphi(c_\rho(\overline{f})).\] Since \(c_\rho(\overline{f})\le(1+\varepsilon)\|\overline{f}\|_{\mathcal{B}_1}\), we have \(\mathcal{R}(f^*)\le\varphi\bigl((1+\varepsilon)\|\overline{f}\|_{\mathcal{B}_1}\bigr)\). Letting \(\varepsilon\to0\) and using the continuity of \(\varphi\), we
conclude that \[\mathcal{R}(f^*)
\le
\varphi(\|\overline{f}\|_{\mathcal{B}_1})
=
\mathcal{R}(\overline{f}).\] Since \(\overline{f}\) was an arbitrary solution of \(A(f)=g\), \(f^*\) is an \(\mathcal{R}\)-minimizing solution. Renaming \(f^*\) as \(f^\dagger\), we obtain the desired subsequential convergence. ◻
We next derive convergence rates results about the Tikhonov-NN method. It is well-known that under general assumptions (specifically, Assumption 1), the convergence rate of
\(f_{n} \to f^{\dagger}\) as \(n\to \infty\) for precise data, and of \(f_{n,k_{\ell}} \to f^{\dagger}\) as \(n,\ell\to
\infty\) for noisy data, can be arbitrarily slow if the exact solution \(f^{\dagger}\) lacks sufficient smoothness (cf. [23]). Consequently, establishing convergence rates typically requires imposing appropriate
smoothness conditions on the exact solution. Commonly used smoothness conditions include range-type source conditions (see, e.g., [45]–[47]), approximate source conditions (see [48]), and
variational source conditions, which take the form of variational inequalities (see [49]). In what follows, we focus specifically on variational
source conditions, as these represent the weakest assumptions under which convergence rate results can still be derived.
Assumption 2 (Variational source condition). Let \(f^\dagger\in\mathcal{B}_1\) be an \(\mathcal{R}\)-minimizing solution of \(A(f)=g\). There exist a radius \(\epsilon>0\), an error measure functional \(\mathcal{E}^{\dagger}: \mathcal{B}_1 \to[0, \infty]\) with \(\mathcal{E}^{\dagger}\left(f^*\right)=0\) and a concave index function6\(\Phi:[0, \infty) \to[0, \infty)\) such
that for all \(f\in\mathcal{B}_1\) with \(\|f-f^\ast\|_{C(K)}<\epsilon\), there holds \[\label{souceinquality}
\mathcal{E}^{\dagger}(f) \leq \mathcal{R}(f)-\mathcal{R}\left(f^*\right)+\Phi(D(A(f), g)).\qquad{(6)}\]
By Assumption 2, we are able to obtain the convergence rates results, as shown in Theorem 8.
Theorem 8 (Convergence rates). Let Assumption 1 hold, and let \(f^\dagger\) be an \(\mathcal{R}\)-minimizing solution in ?? satisfying Assumption 2. Let \(g^\delta\in\mathcal{Y}\)
satisfy \(\mathcal{D}(g,g^\delta)\le \delta\). For \(n\in\mathbb{N}\) and \(\alpha>0\), let \(f_{n,\alpha}^\delta
\in
\arg\min_{f\in X_{n,\infty}}
\mathcal{T}_\alpha(f,g^\delta)\). Assume that, \(\alpha\to0\) and \(\alpha^{-1}(\delta+n^{-s/2})\to0\) as \(\delta\to0\) and \(n\to\infty\). Then, up to a subsequence, \(f_{n,\alpha}^\delta\to f^\dagger\) in \(C(K)\). In addition, the following statements hold:
For all sufficiently small \(\delta,\alpha\) and sufficiently large \(n\), we have \[\begin{align}
\label{errorestimate}
\mathcal{E}^\dagger(f_{n,\alpha}^\delta)
&\le
\tau\alpha^{-1}(\delta+Cn^{-s/2})
+\Phi(\tau\delta)
+\frac{1}{\tau\alpha}\Phi^{-*}(\tau\alpha),
\\
\label{varphiestimate}
\|f_{n,\alpha}^{\delta}\|_{\mathcal{B}_1}
&\le
\varphi^{-1}
\left(
\tau\alpha^{-1}(\delta+Cn^{-s/2})
+\varphi(\|f^\dagger\|_{\mathcal{B}_1})
\right),
\end{align}\] {#eq: sublabel=eq:errorestimate,eq:varphiestimate} where \(C=K_A(C({\Omega,d})\|f^{\dagger}\|_{\mathcal{B}_1})^s\), and \(\Phi^{-*}\) denotes the Fenchel conjugate of
the inverse function \(\Phi^{-1}\), namely \(\Phi^{-*}(\alpha):=(\Phi^{-1})^*(\alpha)=\sup_{t\ge0}\{\alpha t-\Phi^{-1}(t)\}\).
If \(n(\delta)\asymp \delta^{-2/s},\alpha(\delta)\asymp \delta/\Phi(\tau\delta),\) we have \(\mathcal{E}^{\dagger}(f_{n(\delta),\alpha(\delta)}^{\delta})
\;=\;\mathcal{O}\bigl(\Phi(\tau\delta)\bigr)\). In particular, when \(\Phi(t)=c_0 t^\mu\) with \(0<\mu<1\), the choices \(n(\delta)\asymp
\delta^{-2/s},\alpha(\delta)\asymp \delta^{1-\mu}\) yield the rate \(\mathcal{E}^{\dagger}(f_{n(\delta),\alpha(\delta)}^{\delta})= \mathcal{O}(\delta^\mu)\).
Proof. Choose a representing probability measure \(\rho\) of \(f^\dagger\) such that \(c_{\rho}(f^{\dagger})\le
(1+\epsilon)\|f^{\dagger}\|_{\mathcal{B}_1}\). According to Proposition 3, the definition of \(\mathcal{B}_1\)
norm, and monotonicity of \(\varphi\), there exists a two-layer network \(f^{\dagger}_n\) that satisfies \[\label{preineq}
\varphi(\|f^{\dagger}_n\|_{\mathcal{B}_1})\le \varphi(c_{\rho_n}(f^{\dagger}_n))\le \varphi(c_{\rho}(f^{\dagger})),\quad \|f^{\dagger}-f^{\dagger}_n\|_{H^1(\Omega)}\leq \frac{C(\Omega, d)}{\sqrt{n}}\|f^{\dagger}\|_{\mathcal{B}_1}.\tag{16}\]
Therefore, by the minimization property of \(f_{n,\alpha}^{\delta}\), we obtain the estimate \[\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\alpha \varphi(c_{\rho_n}(f_{n,\alpha}^{\delta}))\le
\mathcal{D}(A(f^{\dagger}_n),g^{\delta})+\alpha \varphi(c_{\rho}(f^{\dagger}))\] Using \(\varphi(\|f^{\dagger}_n\|_{\mathcal{B}_1})\le \varphi(c_{\rho_n}(f^{\dagger}_n))\), since \(\epsilon\to 0\) is arbitrary and \(\varphi\) is continuous, we have \[\label{prorate1}
\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\alpha \varphi(\|f_{n,\alpha}^{\delta}\|_{\mathcal{B}_1})\le \mathcal{D}(A(f^{\dagger}_n),g^{\delta})+\alpha\varphi(\|f^{\dagger}\|_{\mathcal{B}_1}).\tag{17}\] Hence, it follows from 16 , \(\mathcal{D}(g,g^{\delta})\le \delta\), and the conditions of \(\mathcal{D}\) in Assumption 1 that \[\|f_{n,\alpha}^{\delta}\|_{\mathcal{B}_1}\le
\varphi^{-1}\left(\tau\alpha^{-1}(\delta+K_A(C({\Omega,d})\|f^{\dagger}\|_{\mathcal{B}_1})^sn^{-\frac{s}{2}})+\varphi(\|f^{\dagger}\|_{\mathcal{B}_1})\right),\] this gives the estimate ?? . Moreover, by noting that \(\mathcal{R}(f)=\varphi(\|f\|_{\mathcal{B}_1})\), 17 also yields \[\label{R-diff}
\alpha\bigl(\mathcal{R}(f_{n,\alpha}^{\delta})-\mathcal{R}(f^{\dagger})\bigr) \le \mathcal{D}(A(f^{\dagger}_n),g^{\delta}) -\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta}),\tag{18}\]
According to Theorem 7(c), we have \(f_{n,\alpha}^{\delta}\to f^{\dagger}\) in \(C(K)\) as \(\delta\to0\) and \(n\to\infty\), hence for every \(\varepsilon>0\), \(\|f_{n,\alpha}^{\delta}-f^{\dagger}\|_{C(K)}<\varepsilon\) holds for all sufficiently small \(\delta\) and sufficiently large \(n\). By Assumption ?? and 18 , we obtain \[\begin{align} \alpha \mathcal{E}^{\dagger}(f_{n,\alpha}^{\delta}) &\le \mathcal{D}(A(f^{\dagger}_n),g^{\delta})-\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\alpha
\Phi(\mathcal{D}(A(f_{n,\alpha}^{\delta}),g))\notag\\ &\le
\tau(\mathcal{D}(A(f^{\dagger}_n),A(f^{\dagger}))+\mathcal{D}(g,g^{\delta}))-\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\alpha\Phi(\tau(\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\mathcal{D}(g,g^{\delta}))\notag \\ &\le
\tau(\mathcal{D}(A(f^{\dagger}_n),A(f^{\dagger}))+\delta)-\mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta})+\alpha \Phi(\tau \delta)+\alpha\Phi(\tau \mathcal{D}(A(f_{n,\alpha}^{\delta}),g^{\delta}))\notag\\ &\le \tau(\delta+Cn^{-\frac{s}{2}})+\alpha
\Phi(\tau \delta)+\tau^{-1}\Phi^{-*}(\tau\alpha),
\end{align}\] where the last inequality follows from Young’s inequality applied to \(\Phi^{-1}\) and its Fenchel conjugate \(\Phi^{-*}:=(\Phi^{-1})^*\). More precisely, for \(t\ge0\), it holds that \(\alpha\Phi(\tau t)\le t+\tau^{-1}\Phi^{-*}(\tau\alpha)\). Balancing \(\delta\) and \(n^{-s/2}\) yields
\(n\asymp\delta^{-2/s}\) and a constant \(C_1>0\) with \(\delta+Cn^{-s/2}\le C_1\delta\). Then, \[\mathcal{E}^{\dagger}(f_{n,\alpha}^{\delta})
\;\le\;\Phi(\tau\delta)\;+\;
\Bigl(\tau C_1\delta\,\alpha^{-1}
+(\tau\alpha)^{-1}\,\Phi^{-*}(\tau\alpha)\Bigr).\] An a priori scale that balances the two terms in \(H_\delta:=\Bigl(\tau C_1\delta\,\alpha^{-1}+(\tau\alpha)^{-1}\,\Phi^{-*}(\tau\alpha)\Bigr)\) is \(\alpha(\delta)\;\asymp\;\frac{\delta}{\Phi(\tau\delta)}\). Therefore, for the choice \(n(\delta)\asymp \delta^{-2/s}\) and \(\alpha(\delta)\asymp
\delta/\Phi(\tau\delta)\) we obtain \(\mathcal{E}^{\dagger}(f_{n(\delta),\alpha(\delta)}^{\delta}) =\mathcal{O}\bigl(\Phi(\tau\delta)\bigr)\). In the special case \(\Phi(t)=c_0
t^\mu\) with \(0<\mu<1\), this reduces to \(\alpha(\delta)\asymp \delta^{1-\mu}\) and the rate \(\mathcal{E}^{\dagger}(f_{n(\delta),\alpha(\delta)}^{\delta}) =\mathcal{O}(\delta^\mu)\). ◻
For the specific data consistency term \(\mathcal{D}(A(f),g)=\|A(f)-g\|_{L^2(\Omega)}^2\), we present the following corollary. It summarizes the corresponding convergence properties established in Theorem 7 and the convergence rates derived in Theorem 8.
Corollary 1. Suppose that the forward operator \(A\) of (1 ) is \(L_A\)-Lipschitz continuous, and that \(f^{\dagger}\) in ?? satisfies a certain variational source condition ?? with \(\mathcal{E}^{\dagger}(f)=\|f-f^{\dagger}\|_{C(K)}^2\) and \(\mathcal{R}(f)=\|f\|_{\mathcal{B}_1}^2\), i.e., there exists an index function \(\Phi\) such that for all \(f\in\mathcal{B}_1\) satisfying \(\|f-f^{\dagger}\|_{C(K)}<\varepsilon\) for some \(\varepsilon>0\), one has \[\|f-f^{\dagger}\|_{C(K)}^2 \leq
\|f\|_{\mathcal{B}_1}^2-\|f^{\dagger}\|_{\mathcal{B}_1}^2+\Phi\big(\|A(f)-A(f^{\dagger})\|_{L^2(\Omega)}^{2}\big).\] Let \(g^{\delta}\in \mathcal{Y}\) satisfy \(\|g-g^{\delta}\|^2_{\mathcal{Y}}\le \delta\) for some \(\delta>0\). For \(n\in\mathbb{N}\) and \(\alpha>0\), let \(f_{n, \alpha}^\delta(\mathbf{x})\in \arg\min_{f\in X_{n,\infty}}\mathcal{T}_\alpha(f,g^{\delta})\). Then the following hold:
(Convergence) If the parameter choice \(\alpha=\alpha(\delta,n)\) satisfy \[\alpha \to 0,\quad \alpha^{-1}(\delta+n^{-1})\to 0,\quad \text{as}~n\to \infty,\quad \delta\to 0,\]
then there exists a convergent subsequence, still denoted by \(\{f_{n,\alpha}^{\delta}\}\), such that \(f_{n,\alpha}^{\delta}\to f^{\dagger}\) in \(C(K)\).
(Convergence rate) If the parameters are chosen such that \(n(\delta) \asymp \delta^{-1}\) and \(\alpha(\delta) \asymp \delta / \Phi(\tau\delta)\), then it follows that \(\| f_{n(\delta),\alpha(\delta)}^{\delta} - f^{\dagger} \|_{C(K)} = \mathcal{O}(\sqrt{\Phi(\tau\delta)})\). In particular, if \(\Phi(t)=c_0 t^\mu\) with \(c_0>0\) and \(0<\mu<1\), then the choices \(n(\delta)\asymp \delta^{-1}\) and \(\alpha(\delta) \asymp \delta^{1-\mu}\)
yields the rate \(\|f_{n(\delta),\alpha(\delta)}^{\delta}-f^{\dagger}\|_{C(K)}=\mathcal{O}(\delta^{\mu/2})\).
Proof. These conclusions follow naturally from a combination of Theorem 7 and Theorem 8 by letting the data consistency term \(\mathcal{D}(A(f),g)=\|A(f)-g\|_{L^2(\Omega)}^2\) with \(s=2,K_A=L_A^2\). ◻
The convergence rate results established in Theorem 8, along with Corollary 1,
imply an optimal choice for the number of neurons, thereby providing explicit answers to the foundational question (Q4) posed in the introduction.
Answer to Q4: Under a general variational source condition, the proposed Tikhonov-NN scheme achieves the convergence rate \(\Phi(\tau\delta)\). This rate has the same form as the classical rate obtained
under variational source conditions [50]. Moreover, Theorem 8 quantifies the network width needed to realize this rate together with a suitable choice of the regularization parameter, namely \(n(\delta) \asymp \delta^{-2/s}\), as established in
Theorem 8.
In this section, we present three numerical examples to verify the theoretical results discussed in the previous sections. The numerical experiments follow the procedure outlined below, and all relevant data and source code are publicly available on
GitHub at https://github.com/z1998w/DNNip.
The numerical procedure consists of the following four steps.
Step 1: Generation of the exact solution.
As a reference solution we generate the exact function \(f^{\dagger}\) of the form (cf. [42])
\[\label{exactf}
f^{\dagger}(t)=\sum_{j=1}^T \phi_j K_d\left(t, \ell^j\right), \quad t \in[0,1]^d,\tag{19}\] where \(K_d\) is the polynomial kernel defined by \(K_d(t, \ell)=\langle t, \ell
\rangle_{\mathbb{R}^d}+1\), for \(t \in[0,1]^d\). Unless otherwise specified, the nodes \(\ell^j=\left(\ell_1^j, \ell_2^j, \cdots, \ell_d^j\right)\) and coefficients \(\phi_j\) (for \(j=1, \cdots, T\)) were independently and uniformly drawn from the cubes \([-1,1]^d\) and \([-1,1]\)
respectively. As discussed in Remark 9, the exact solutions constructed in this way belong to the Barron space \(\mathcal{B}_1\). For each example
below, appropriate modifications were made to this construction to suit specific purposes.
Step 2: Generation of noisy data.
We compute the exact data \(g=A(f^\dagger)\) and generate the noisy data by \[g^\delta=g+\delta\xi,\] where \(\delta>0\) denotes the noise level and
\(\xi\) is uniformly distributed in \([-1,1]\).
Step 3: Computation of regularized solutions.
We approximate \(f^\dagger\) by two-layer neural networks of the form \(f(\mathbf{x})
=
\frac{1}{n}\sum_{j=1}^n
a_j(\boldsymbol{b}_j^T\mathbf{x}+c_j)_+\). In the numerical experiments, we compare three reconstruction strategies. For ENN1 (Algorithm 1), we choose an increasing sequence \(r(n)\) bounded by a constant \(B\ge c_{\rho_n}(f^{\dagger})=\|\sum_{j=1}^T \phi_jl^j\|_{1}+|\sum_{j=1}^T\phi_j|\). For each fixed width \(n\), we solve \[\min_{\substack{(a_j,\boldsymbol{b}_j,c_j) \in M_{r(n)},\\ j=1,\dots,n}}
\;\left\|\,A\!\left(\frac{1}{n} \sum_{j=1}^n a_j \left(\boldsymbol{b}_j^T \mathbf{x}+c_j\right)_{+}\right) - g^\delta \right\|_{\mathcal{Y}}\] where \(M_r := \{ (a_{j},\boldsymbol{b}_j,c_j)\in \mathbb{R}\times \mathbb{R}^d
\times \mathbb{R},\;|a_j|\le r(n),\;\|\boldsymbol{b}_j\|_1+|c_j| =1 \}\). For ENN2 (Algorithm 2), we take \(\mathcal{R}(f)=\mathcal{R}_H(f)=C_{\mathcal{R}_H}\|f\|_{H^1(\Omega)}\)
that is a \(C_{\mathcal{R}_H}\)-Lipschitz continuous majorant functional of the \(H^1\) norm, then we solve such minimization problem \[\min_{\substack{(a_j,\boldsymbol{b}_j,c_j) \in M_{r(n)},\\ j=1,\dots,n}}
\;\left\|\,A\!\left(\frac{1}{n}\sum_{j=1}^n a_j\, (\boldsymbol{b}_j^T \mathbf{x} + c_j)_{+}\right) - g^\delta \right\|_{\mathcal{Y}}
\;+\; \beta_n a \left\|\frac{1}{n}\sum_{j=1}^n a_j\, (\boldsymbol{b}_j^T \mathbf{x} + c_j)_{+}\right\|_{H^1(\Omega)}
,\] where we choose \(\beta_n=c_0n^{-\frac{\theta}{2}}\), and \(c_0=C_{\mathcal{R}_H}=0.1\). Furthermore, we also consider Tikhonov-NN (the neural network-based Tikhonov
regularization method) as discussed in Section 4. We solve \[\min_{\substack{(a_j,\boldsymbol{b}_j,c_j) \in M_{\infty},\\ j=1,\dots,n}}
\;\left\|\,A\!\left(\frac{1}{n}\sum_{j=1}^n a_j\, (\boldsymbol{b}_j^T \mathbf{x} + c_j)_{+}\right) - g^\delta \right\|^2_{\mathcal{Y}}
\;+\; \alpha \left( \frac{1}{n}\sum_{j=1}^n |a_j|(\left\|\boldsymbol{b_j}\right\|_1+\left|c_j\right|) \right)^{2}
,\] here the regularization parameter \(\alpha\) is a small positive number chosen by \(\alpha=(\delta+n^{-1})^{0.8}\).
Step 4: Error evaluation.
To quantify the accuracy of the regularized solutions \(f_n^\delta\), we compute the relative \(L^2\)-error with respect to the exact solution \(f^{\dagger}\), defined by \[\label{eq:rel95L295error} \text{Relative } L^2 \text{ Error} := \frac{\|f_n^\delta -
f^\dagger\|_{L^2([0,1]^d)}}{\|f^\dagger\|_{L^2([0,1]^d)}} \approx \frac{\sqrt{\sum_{i=1}^N (f_n^\delta(t_i) - f^\dagger(t_i))^2}}{\sqrt{\sum_{i=1}^N (f^\dagger(t_i))^2}}.\tag{20}\]
Remark 9. The exact solution \(f^\dagger\) generated by 19 belongs to the Barron space \(\mathcal{B}_1\). Indeed, it can be written as \(f^{\dagger}(t)=u^T t+v\) with \(u:=\sum_{j=1}^T \phi_j \ell^j\in \mathbb{R}^d\) and \(v:=\sum_{j=1}^T \phi_j\). Let \(n=d+1\)
and choose the uniform empirical measure \(\rho_n\) supported on the \(d+1\) atoms \((a_k,\boldsymbol{b}_k,c_k)=(n u_k,e_k,0)\), \(k=1,\ldots,d\), and \((a_0,\boldsymbol{b}_0,c_0)=(n v,\boldsymbol{0},1)\). Then, for \(t\in[0,1]^d\), since \(t_k\ge0\), we have
\[\frac{1}{n}\sum_{k=1}^d n u_k(e_k^Tt)_+
+
\frac{1}{n} n v(\boldsymbol{0}^Tt+1)_+
=
\sum_{k=1}^d u_k t_k+v
=
f^\dagger(t).\] Thus \(f^\dagger\) admits a finite Barron representation. Moreover, the corresponding representation cost is \[c_{\rho_n}(f^{\dagger})=\|f^{\dagger}\|_{\mathcal{B}_{1,\rho_n}}
=\mathbb{E}_{\rho_n}\left[
|a|\left(\|\boldsymbol{b}\|_1+|c|\right)
\right]
=\sum_{k=1}^d |u_k|+|v|
=\Big\|\sum_{j=1}^T \phi_j \ell^j\Big\|_1+\Big|\sum_{j=1}^T \phi_j\Big|.\] Consequently, \(\|f^\dagger\|_{\mathcal{B}_1}\le c_{\rho_n}(f^\dagger)<\infty\), and hence \(f^\dagger\in\mathcal{B}_1\).
We now detail the numerical implementation common to all experiments. All continuous problems are discretized: the solution \(f\) is represented on an \(N\)-point solution grid and the
noisy data \(g^\delta\) on an \(M\)-point data grid. The continuous norms in the objective functionals are approximated by standard numerical quadrature on these grids. The resulting
non-convex optimization problems are solved with the Adam optimizer with a fixed learning rate of \(10^{-3}\). To balance expressive capacity and training efficiency, we adopt a progressive width schedule: the number of
neurons \(n\) starts at \(50\) and is increased in steps of \(20\) up to a prescribed maximum \(n_{\max}\). This expansion
is accelerated by a warm-start strategy, where the parameters of existing neurons are retained and only the parameters of newly added neurons are randomly initialized. The specific discretization parameters and training iterations are chosen as follows.
For Examples 1-2 (1D problems) we set \(N=101\) (\(M=51\)
for Example 1, \(M=101\) for Example 2), \(n_{\max}=1000\), and train for 600 (ENN1 and ENN2) or 1200 (Tikhonov-NN) iterations per increment of \(n\). For Example 3 (2D
problem) we set \(N=31^2=961\), \(M=124\), \(n_{\max}=500\), and train for 1500 (ENN1 and ENN2) or 2500 (Tikhonov-NN) iterations per increment. The
robustness of the observed trends is assessed by repeating all experiments for several random seeds. In the figures below, the horizontal axis (number of neurons \(n\)) is plotted on a base-10 logarithmic scale in order to
emphasize the initial steep reduction of the error.
Example 1. 1D Fredholm integral equation of the first kind: \[A(f)=\int_0^1 K(t,s)f(s)\, ds=g(t),\quad t\in [0,1],s\in[0,1],\] where \[K(t, s)= \begin{cases}s(1-t), &
0\le s \le t\le 1, \\ t(1-s), & 0\le t<s\le 1.\end{cases}\] The operator \(A\) is linear, compact, and self-adjoint on \(L^2[0,1]\). Its eigenvalues are \(\lambda_n = 1/(n^2\pi^2)\) with corresponding eigenfunctions \(\sin(n\pi t)\), which implies that \(A\) is injective. Moreover, as \(\|u\|_{L^2(0,1)}\le \|u\|_{H^1(0,1)}\) and \(\|A\|_{\mathcal{L}(L^2(0,1),L^2(0,1))}=1/\pi^2\), we have \[\|A(f)-A(f^\dagger)\|_{L^2(0,1)}
\le
\frac{1}{\pi^2}\|f-f^\dagger\|_{L^2(0,1)}
\le
\frac{1}{\pi^2}\|f-f^\dagger\|_{H^1(0,1)}.\] Hence \(A\) is locally Hölder continuous at \(f^\dagger\) with respect to the \(H^1(0,1)\)-norm, with
exponent \(\theta=1\) and constant \(L_A=1/\pi^2\). Therefore, the forward operator satisfies the continuity, injectivity and local Hölder continuity assumptions of Theorems 1–2. In this
example, the exact solution \(f^{\dagger}\) is constructed by 19 with \(T=10\) and \(d=1\).
Figure 3: 1D linear integral equation: Relative \(L^2\) error of \(f_n^\delta\) obtained by ENN1 (top row), ENN2 (middle row), and Tikhonov-NN (bottom row).
Each column corresponds to a different random seed: 111, 666, 3333 (from left to right). The marker on each curve highlights the first network architecture that satisfied the stopping criterion for each noise level, with \(\tau=1.0001\). The corresponding value of \(n\) (the stopping number) is indicated in the legend.
Example 2. 1D auto-convolution Volterra integral equations (AVIEs) of the first kind: \[\label{IntegralEq} A(f)=\int_{0}^{t} f(t-s)f(s) \,ds = g(t), \quad t\in
[0,1].\qquad{(7)}\] Here, \(A\) is a nonlinear continuous operator. However, on a finite interval, the auto-convolution operator is not injective on the whole nonnegative cone. Following [51], [52], injectivity holds after restricting the admissible set to \[D_0^+
:=
\left\{
f\in L^2(0,1): f(t)\ge0\;\text{a.e. on }[0,1],
\text{ and } f \text{ does not vanish a.e. on any interval }[0,\varepsilon],
\varepsilon>0
\right\}.\] In particular, this condition is satisfied for nonnegative continuous function \(f\in C[0,1]\) with \(f(0)>0\). Furthermore, for any \(f,f^{\dagger}\in H^1(\Omega)\) with \(\|f-f^{\dagger}\|_{H^1(\Omega)}\le \gamma\), the following local Lipschitz estimate holds according to the Young’s inequality: \[\|A(f)-A(f^{\dagger})\|_{L^2(0,1)}\; \le\; \bigl(\|f\|_{L^1(0,1)}+\|f^{\dagger}\|_{L^1(0,1)}\bigr)\,\|f-f^{\dagger}\|_{L^2(0,1)}.\] By Sobolev embedding, \(\|u\|_{L^2(\Omega)} \le
\|u\|_{H^1(\Omega)}\) and \(\|u\|_{L^1(\Omega)} \le C_\Omega \|u\|_{H^1(\Omega)}\) for some constant \(C_\Omega > 0\). Together with the bound \(\|f\|_{H^1(\Omega)} \le \|f^\dagger\|_{H^1(\Omega)} + \gamma\), we obtain \[\|A(f) - A(f^\dagger)\|_{L^2(0,1)} \le C_\Omega \left( 2\|f^\dagger\|_{H^1(\Omega)} + \gamma \right) \|f -
f^\dagger\|_{H^1(\Omega)}.\] Thus, after restricting the admissible set to the positive cone, Example 2 satisfies the injectivity and local Hölder continuity assumptions required in
Theorems 1–2 with \(\theta=1\). For this example, to ensure that the exact solution belongs to \(D_0^+\), we define \(f^\dagger\) according to 19 , where \(\ell^j\in[0,1]^d\) and \(\phi_j\ge0\) are chosen such that \(\sum_{j=1}^T \phi_j>0\), and we set \(T=5,d=1\).
Figure 4: Auto-convolution equation: Relative \(L^2\) error of \(f_n^{\delta}\) obtained by ENN1 (top row), ENN2 (middle row), and Tikhonov-NN (bottom row).
Each column corresponds to a different random seed: 678, 765, 987 (from left to right). The marker on each curve highlights the first network architecture that satisfied the stopping criterion for each noise level, with \(\tau=1.0001\). The corresponding value of \(n\) (the stopping number) is indicated in the legend. In some figures, the blue circles (\(\delta=0.0001\)) and green
squares (\(\delta=0.001\)) coincide.
Example 3. The 2D equation of electrical impedance tomography (EIT) [26] is considered on the unit square domain
\(\Omega=[0,1]^2\): \[\begin{align}
\label{EIT95equation}
\nabla \cdot\!\big(f(x,y)\,\nabla u(x,y)\big)=0, \quad (x,y)\in\Omega,
\end{align}\qquad{(8)}\] where \(f(x,y)\) denotes the electrical conductivity, while \(u(x,y)\) means the electrical potential. A voltage distribution, denoted by \(h(x,y)\), is applied on the boundary \(\partial\Omega\), corresponding to the Dirichlet boundary condition \(u(x,y)|_{\partial\Omega} = h(x,y)\). This applied
voltage induces a boundary current flux \(g(x,y)\), which is then measured as the Neumann boundary data: \[g(x,y)=f(x,y)\,\frac{\partial u}{\partial \nu}(x,y),\quad
(x,y)\in\partial\Omega,\] where \(\nu\) is the unit outward normal vector to \(\partial\Omega\). The Dirichlet-to-Neumann (DN) map is the operator that maps any applied voltage to the
resulting measured current: \[\Lambda_f:\;u(x,y)\big|_{\partial\Omega}\longmapsto \big(f(x,y)\,\frac{\partial u}{\partial \nu}(x,y)\big)\big|_{\partial\Omega}.\]
The forward problem in EIT is to compute the boundary current corresponding to a given conductivity \(f\) and a prescribed boundary voltage \(h\). The associated inverse problem,
known as Calderon’s problem, seeks to recover \(f\) from the complete DN map. In the theoretical formulation, we take \(A(f)=\Lambda_f\). We consider the admissible class \[\mathcal{D}_M
:=
\left\{
f\in W^{1,\infty}(\Omega):
\alpha\le f(\boldsymbol{x})\le \beta
\;\text{for a.e. } \boldsymbol{x}\in\Omega,\;
\|f\|_{W^{1,\infty}(\Omega)}\le M
\right\},
\qquad 0<\alpha<\beta<\infty .\] This class guarantees the uniform ellipticity of ?? and the well-posedness of the DN map. Since \(\mathcal{D}_M\subset L^\infty(\Omega)\) is uniformly elliptic, the
two-dimensional Calderon uniqueness theorem implies that the complete-data operator \(A:f\mapsto\Lambda_f\) is injective on \(\mathcal{D}_M\)[53]. Moreover, by Alessandrini’s identity and standard energy estimates, for \(f,f^\dagger\in\mathcal{D}_M\), \[\|\Lambda_f-\Lambda_{f^\dagger}\|_{\mathcal{L}(H^{1/2}(\partial\Omega),
H^{-1/2}(\partial\Omega))}
\le C\|f-f^\dagger\|_{L^\infty(\Omega)}.\] Since \(\Omega\subset\mathbb{R}^2\), the Gagliardo–Nirenberg inequality gives \[\|f-f^\dagger\|_{L^\infty(\Omega)}
\le C_\Omega
\|f-f^\dagger\|_{L^2(\Omega)}^{1/2}
\|f-f^\dagger\|_{W^{1,\infty}(\Omega)}^{1/2}
\le C_{\Omega,M}\|f-f^\dagger\|_{H^1(\Omega)}^{1/2}.\] Consequently, \[\|\Lambda_f-\Lambda_{f^\dagger}\|_{\mathcal{L}(H^{1/2}(\partial\Omega),
H^{-1/2}(\partial\Omega))}
\le C\|f-f^\dagger\|_{H^1(\Omega)}^{1/2},
\qquad f\in\mathcal{D}_M.\] Thus \(A\) is locally Hölder continuous with respect to the \(H^1(\Omega)\) metric on \(\mathcal{D}_M\), with Hölder
exponent \(1/2\).
In the numerical implementation, the complete DN map is replaced by a finite number of boundary measurements. For example, for one prescribed voltage pattern \(h_1\), we use \(A_1(f)=\Lambda_f h_1\), where \[h_1(x,y)=
\begin{cases}
\sin(\pi x), & y=0,\\
0, & \text{otherwise on } \partial\Omega .
\end{cases}\] More generally, for voltage patterns \(h_1,\ldots,h_m\), the numerical measurement operator is \(A_m(f)=\bigl(\Lambda_f h_1,\ldots,\Lambda_f h_m\bigr)\).
The preceding theoretical discussion concerns the complete-data forward map \(A(f)=\Lambda_f\), for which injectivity is available on the uniformly elliptic admissible class \(\mathcal{D}_M\). The operator \(A_m\) is used only for the numerical realization, as a practical finite-dimensional measurement model for the EIT experiment.
In the present example, the true conductivity is \(f^\dagger(x,y)=x+y+p_0\) with \(p_0>0\) on \(\Omega=[0,1]^2\), so \(f^\dagger\) is uniformly elliptic and essentially bounded: \(0<p_0\le f^\dagger\le p_0+2\). Here, \(p_0\) is set to 1.
Figure 5: EIT: Relative \(L^2\) error of \(f_n^{\delta}\) obtained by ENN1 (top row), ENN2 (middle row), and Tikhonov-NN (bottom row). Each column corresponds
to a different random seed: 20, 30, 40 (from left to right). The marker on each curve highlights the first network architecture that satisfied the stopping criterion for each noise level, with \(\tau=1.0001\). The
corresponding value of \(n\) (the stopping number) is indicated in the legend. In some figures, the blue circles (\(\delta=0.0001\)) and green squares (\(\delta=0.001\)) coincide.
The numerical results for Examples 1-3 are reported in Figures 3, 4, and 5, respectively. These plots show how the relative \(L^2\) error of the neural network
approximations produced by the three methods evolves with the number of neurons under different random seeds.
To begin with, we draw attention to a phenomenon observed in Figure 3: in the first column, the solutions generated by ENN1 and ENN2 display a pronounced instability, namely, the relative error increases
with the number of neurons \(n\) rather than decreasing. This effect reflects the intrinsic ill-posedness of the underlying inverse problem: as the network capacity grows, the approximation becomes increasingly sensitive to
noise and numerical perturbations, which manifests as instability and error amplification. By contrast, Tikhonov-NN effectively suppresses this phenomenon. The improvement arises from the presence of the regularization term, which imposes an additional
constraint that balances the fidelity to the data with the stability of the solution, thereby damping spurious oscillations and preventing uncontrolled error growth. This underscores the stabilizing effect of Tikhonov regularization, which provides a
stronger balance between accuracy and stability than the alternative regularization strategies.
Beyond this specific behavior, more general trends can be identified across the results. From the second and third columns of Figure 3, and throughout Figures 4 and 5, it can be observed that, in general, the error of the neural network approximation decreases as the noise level \(\delta\) decreases and as the number of neurons \(n\) increases. However, for larger noise levels (e.g., \(\delta=0.1,0.2\)), the approximation error first
decreases and then increases with growing \(n\), leading to a typical semi-convergence, or U-shaped, behavior. This observation suggests that, under high noise conditions, a simpler and smaller network architecture can
already achieve satisfactory performance, while blindly increasing the network size does not necessarily improve the fitting accuracy and may instead lead to overfitting or degraded performance.
In addition, the points marked for the stopping index \(n\) of ENN1 and ENN2 indicate that both algorithms successfully terminate at finite iteration steps under all three random seed settings once the stopping criterion
is met. In high-noise cases, the discrepancy threshold is larger, and the algorithm may stop before the true reconstruction error reaches its minimum. This is consistent with the classical semi-convergence phenomenon: the discrepancy principle is designed
to ensure stability rather than to minimize the unknown true error. Moreover, the specific values of the stopping \(n\) marked in the figures reflect that \(n=n(\delta)\) generally increases
as \(\delta\) decreases, and are qualitatively consistent with the theoretical estimate \(\mathcal{O}(1/\delta^2)\) provided in Theorems 4 and 6.
Regarding the Tikhonov-NN method, we examine the last row of the three figures. It can be observed that the error generally decreases initially and then stabilizes as the number of neurons increases when the noise level is small. In some cases, the
error continues to decrease without stabilizing, which may be attributed to the limitation of the maximum number of neurons (\(n_{\max}\)) considered in the experiments. Moreover, for relatively large noise levels (e.g.,
\(\delta=0.1,0.2\)), the error is mainly dominated by the noise. Under such circumstances, as observed in some of the presented figures, increasing the number of neurons does not significantly reduce the approximation
error, since the noise-induced instability persists regardless of model capacity. Overall, in the non-blow-up cases, ENN1 and ENN2 as well as the Tikhonov-NN method are all capable of constructing neural network approximations with good accuracy. In
comparison, ENN1 and ENN2 tend to achieve approximations with smaller errors using relatively fewer neurons than the Tikhonov-NN approach, whereas the Tikhonov-NN method provides greater robustness and stability.
In this work, we have developed a regularization framework for solving ill-posed inverse problems by shallow neural network approximations. In contrast to a fixed-architecture formulation, we considered expanding two-layer network classes with
increasing width and controlled representation cost, so that the network size can be coupled with the noise level. For iterative regularization, we develop ENNs and prove its regularization properties under different assumptions. As a by-product, we show
that for data with high noise levels, a small network architecture is sufficient to obtain a good approximate solution. For variational regularization, we analyze neural network approximation within the framework of Tikhonov regularization. Under
variational source conditions, we establish convergence rate results. In summary, this paper demonstrates that neural networks, when integrated with a slightly modified conventional regularization framework, can serve as effective regularization methods
with theoretical guarantees. However, several challenges remain.
Our theoretical analysis focuses on shallow (two-layer) neural networks in the classical Barron-space setting. Within this framework, we establish a regularization theory for neural network approximations of inverse problem solutions: for a given
noise level \(\delta\), we show that a corresponding network width \(n=n(\delta)\) can be selected so that the resulting neural network approximation converges to the exact solution as \(\delta\to0\). Under variational source conditions, this convergence is further quantified by rates of order \(\Phi(\tau\delta)\). A current limitation is that the depth is fixed at \(l=2\), hence the approximation error scaling as \(\mathcal{O}(1/\sqrt{n})\) (Proposition 3),
does not improve with increasing depth. Extending the analysis to networks with greater depth \(l\) is an important next step, with the goal of establishing convergence properties and rates when both \(n=n(\delta)\) and \(l=l(\delta)\) are adaptively chosen. However, a principal obstacle to extending our method to deep networks is that existing universal approximation theorems for DNNs rarely
provide explicit, uniform bounds on network parameters (e.g., layerwise max-norm radii). Such bounds are needed so the algorithm searches within a radius-constrained network class where the feasible set \(X_{n,r}\) is
compact, ensuring existence of minimizers and enabling convergence to the exact solution. This motivates the incorporation of parameter-bounded deep neural network approximation results into the regularization framework, with the aim of establishing
convergence properties for depth-adaptive neural network regularization methods.
Beyond the ridge Barron models considered here, recent studies on graph convolutional neural networks (GCNNs) (e.g., [54]) have
introduced shallow models of the form \(f_M(\mathbf{x},\Theta) = \frac{1}{M} \sum_{m=1}^M \mathbf{a}_m^{\!\top} \, \sigma(\mathbf{b}_m *\mathbf{x} + c_m), \quad \mathbf{x} \in \Omega\). Analogous to these neural networks,
the corresponding function class is referred to as the graph Barron space\(\mathcal{B}_G\). It has been shown to form a reproducing kernel Banach space, and in addition to satisfying a Lipschitz property, it
enjoys a fundamental approximation theorem guaranteeing that any function in \(\mathcal{B}_G\) can be well-approximated by GCNN outputs. Incorporating this graph-convolutional extension into our framework represents a
promising direction for future research.
One particularly important open problem is the development of efficient optimization algorithms to solve the non-convex optimization problems [OPT-ENN1], [OPT-ENN2], and 13 . Another key issue is the numerical implementation of the regularization stabilizer \(\mathcal{R}_n(f)\) in Tikhonov regularization 13 . The current definition of \(\mathcal{R}_n(f)\) is derived from theoretical analysis, but it is nonlinear and
difficult to implement using conventional optimization algorithms. Finding a suitable surrogate that meets both theoretical requirements and practical feasibility remains an intriguing and open challenge.
This work was funded by the Shenzhen Sci-Tech Fund (No. RCJC20231211090030059), National Key Research and Development Program of China (No. 2025YFE0113400) and National Natural Science Foundation of China (No. W2421102).
Proof. It suffices to verify that, for every sequence \(\{f_n(a^k,\boldsymbol{b}^k,c^k;\mathbf{x})\}_{k=1}^{\infty}
\subset X_{n,r(n)}\), there exists a convergent subsequence whose limit belongs to \(X_{n,r(n)}\). For each \(k\), the parameters of \(f_n(a^k,\boldsymbol{b}^k,c^k;\mathbf{x})\) can be written as \((a^k,\boldsymbol{b}^k,c^k)
:=
\bigl((a_j^k,\boldsymbol{b}_j^k,c_j^k)\bigr)_{j=1}^n
\in (M_r)^n\). Since \(M_r\) is compact, \((M_r)^n\) is compact. Hence there exists a subsequence \(\{(a^{k_\ell},\boldsymbol{b}^{k_\ell},c^{k_\ell})\}_{\ell=1}^{\infty}\) and a limit \((\tilde{a},\tilde{\boldsymbol{b}},\tilde{c})
:=
\bigl((\tilde{a}_j,\tilde{\boldsymbol{b}}_j,\tilde{c}_j)\bigr)_{j=1}^n
\in (M_r)^n\) such that \((a^{k_\ell},\boldsymbol{b}^{k_\ell},c^{k_\ell})
\to
(\tilde{a},\tilde{\boldsymbol{b}},\tilde{c})
\quad\text{as } \ell\to\infty
\quad\text{in }(\mathbb{R}\times\mathbb{R}^d\times\mathbb{R})^n\) .
Since \(K\) is bounded, there is \(R_K>0\) with \(K\subset\{x:\|x\|_\infty\le R_K\}\). Let \(C_R:=\max\{R_K,1\}\).
Since \(t\mapsto t_+\) is \(1\)-Lipschitz and \(\|\boldsymbol{b}\|_1+|c|=1\), for every \(x\in K\), we have
\[\begin{align}
&\bigl|f_n(a^{k_{\ell}},\boldsymbol{b}^{k_{\ell}},c^{k_{\ell}};\mathbf{x})-f_n(\tilde{a},\tilde{\boldsymbol{b}},\tilde{c};\mathbf{x})\bigr|\notag\\
&\le \frac{1}{n}\sum_{j=1}^n \bigl|a_j^{(k_{\ell})}-\tilde{a}_j\bigr|\, \bigl| \bigl((\boldsymbol{b}_j^{(k_{\ell})})^T \mathbf{x} + c_j^{(k_{\ell})}\bigr)_+ \bigr| \;+\; \frac{1}{n}\sum_{j=1}^n \bigl|\tilde{a}_j\bigr|\, \bigl|
\bigl((\boldsymbol{b}_j^{(k_{\ell})})^T \mathbf{x} + c_j^{(k_{\ell})}\bigr)_+ - \bigl(\tilde{\boldsymbol{b}}_j^T \mathbf{x} + \tilde{c}_j\bigr)_+ \bigr| \notag\\
&\le \frac{1}{n}\sum_{j=1}^n \bigl|a_j^{(k_{\ell})}-\tilde{a}_j\bigr|\, \bigl( \bigl|(\boldsymbol{b}_j^{(k_{\ell})})^T \mathbf{x}\bigr| + \bigl|c_j^{(k_{\ell})}\bigr| \bigr) \;+\; \frac{1}{n}\sum_{j=1}^n \bigl|\tilde{a}_j\bigr|\, \bigl(
\bigl|(\boldsymbol{b}_j^{(k_{\ell})}-\tilde{\boldsymbol{b}}_j)^T \mathbf{x}\bigr| + \bigl|c_j^{(k_{\ell})}-\tilde{c}_j\bigr| \bigr) \notag\\ \label{eqlemma1}
&\le C_R\Biggl[ \frac{1}{n}\sum_{j=1}^n \bigl|a_j^{(k_{\ell})}-\tilde{a}_j\bigr| + \frac{1}{n}\sum_{j=1}^n \bigl|\tilde{a}_j\bigr| \bigl(\|\boldsymbol{b}_j^{(k_{\ell})}-\tilde{\boldsymbol{b}}_j\|_1 + \bigl|c_j^{(k_{\ell})}-\tilde{c}_j\bigr|\bigr)
\Biggr],
\end{align}\tag{21}\] Hence \(||f_n(a^{k_{\ell}},\boldsymbol{b}^{k_{\ell}},c^{k_{\ell}};\mathbf{x})-f_n(\tilde{a},\tilde{\boldsymbol{b}},\tilde{c};\mathbf{x})||_{C(K)}\to 0\), which means \(f_n\left(a^{k_{\ell}},\boldsymbol{b}^{k_{\ell}},c^{k_{\ell}};\mathbf{x}\right)\to f_n(\tilde{a},\tilde{\boldsymbol{b}},\tilde{c};\mathbf{x})\) in \(C(K)\), proving sequential compactness.
Moreover, we note that \(f_n(\tilde{a},\tilde{\boldsymbol{b}},\tilde{c};\mathbf{x})\in\mathcal{B}_1\) for the reason that \(\frac{1}{n}\sum_{j=1}^{n}|\tilde{a}_j|(\|\boldsymbol{\tilde{b}}_j\|_1+|\tilde{c}_j|)=\frac{1}{n}\sum_{j=1}^{n}|\tilde{a}_j|\le r(n)<+\infty\). Since \(\Omega\) is bounded, the map \(C(K)\to L^2(\Omega)\) is continuous with \(\|f\|_{L^2(\Omega)}\le |\Omega|^{1/2}\|f\|_{C(K)}\). Hence \(X_{n,r(n)}\) is also sequentially compact in \(L^2(\Omega)\). ◻
Proof. Let \(R_K:=\sup_{\mathbf{x}\in K}\|\mathbf{x}\|_\infty<\infty\). Since \(\|\boldsymbol{b}_j^{(m)}\|_1+|c_j^{(m)}|=1\), we have \(|(\boldsymbol{b}_j^{(m)})^T\mathbf{x}+c_j^{(m)}|\le R_K+1\) for \(\mathbf{x}\in K\). Hence \[|f_m(\mathbf{x})|
\le q_m(R_K+1)
\le Q(R_K+1).\] Moreover, since \(t\mapsto t_+\) is \(1\)-Lipschitz, \[|f_m(\mathbf{x})-f_m(\mathbf{y})|
\le q_m\|\mathbf{x}-\mathbf{y}\|_\infty
\le Q\|\mathbf{x}-\mathbf{y}\|_\infty .\] Thus \(\{f_m\}\) is uniformly bounded and equicontinuous on \(K\). By the Arzelà–Ascoli theorem, it is relatively compact in \(C(K)\).
Now suppose that \(f_{m_k}\to f\) in \(C(K)\). Passing to a further subsequence if necessary, assume that \(q_{m_k}\to
L:=\liminf_{k\to\infty}q_{m_k}\). If \(L=0\), then \(\|f_{m_k}\|_{C(K)}\le q_{m_k}(R_K+1)\to0\), so \(f=0\), and the conclusion is immediate.
Assume \(L>0\). For all sufficiently large \(k\), \(q_{m_k}>0\). Define \(\alpha_j^{(m_k)}:=|a_j^{(m_k)}|/(n_{m_k}q_{m_k})\) and the probability measure \[\rho_k
:=
\sum_{j=1}^{n_{m_k}}
\alpha_j^{(m_k)}
\delta_{\left(
\operatorname{sign}(a_j^{(m_k)})q_{m_k},
\boldsymbol{b}_j^{(m_k)},c_j^{(m_k)}
\right)} .\] Then \(f_{m_k}(\mathbf{x})
=
\int a(\boldsymbol{b}^T\mathbf{x}+c)_+
\,d\rho_k(a,\boldsymbol{b},c)\). For every \(\varepsilon>0\), \(q_{m_k}\le L+\varepsilon\) for all large \(k\). Hence the supports of \(\rho_k\) are eventually contained in the compact set \(K_{L+\varepsilon}
=
\{(a,\boldsymbol{b},c): |a|\le L+\varepsilon,\;
\|\boldsymbol{b}\|_1+|c|\le1\}\). Passing to a further subsequence, \(\rho_k\rightharpoonup\rho^*\) weakly as probability measures on \(K_{L+\varepsilon}\). Since \((a,\boldsymbol{b},c)\mapsto a(\boldsymbol{b}^T\mathbf{x}+c)_+\) is continuous and bounded on this compact set, we obtain \[f(\mathbf{x})
=
\int a(\boldsymbol{b}^T\mathbf{x}+c)_+
\,d\rho^*(a,\boldsymbol{b},c).\] Thus \(f\in\mathcal{B}_1\). Moreover, \(\rho^*\) is supported in \(K_{L+\varepsilon}\), and therefore \(\int |a|(\|\boldsymbol{b}\|_1+|c|)\,d\rho^* \le L+\varepsilon\). Taking the infimum over all Barron representations and then letting \(\varepsilon\downarrow0\) gives \(\|f\|_{\mathcal{B}_1}
\le L
=
\liminf_{k\to\infty}q_{m_k}\). The proof is complete. ◻
Proof. The estimates in the proposition follow directly from the proof of [42] by taking \(k=1\) and \(m=1\). It remains only to verify that the approximating network constructed there can be chosen from our admissible class \(X_{n,c_\rho(f)}\). Indeed, the network constructed in the proof of
[42] has the form \[\tilde{f}_n(\mathbf{x})
=
\frac{1}{n}\sum_{j=1}^n
c_\rho(f)\bar a_j
\left(\bar{\boldsymbol{b}}_j^T\mathbf{x}+\bar c_j\right)_+,\] where \((\bar a_j,\bar{\boldsymbol{b}}_j,\bar c_j)\) are the sampled parameters from the empirical measure \(\rho_n\). To
relate this network to our admissible class \(X_{n,c_\rho(f)}\), we normalize each neuron. Set \(s_j:=\|\bar{\boldsymbol{b}}_j\|_1+|\bar c_j|\). For \(s_j>0\), define \(a_j:=c_\rho(f)\bar a_j s_j,
\boldsymbol{b}_j:=\frac{\bar{\boldsymbol{b}}_j}{s_j},
c_j:=\frac{\bar c_j}{s_j}.\) Then, by the positive homogeneity of the ReLU activation, \[c_\rho(f)\bar a_j
\left(\bar{\boldsymbol{b}}_j^T\mathbf{x}+\bar c_j\right)_+
=
a_j(\boldsymbol{b}_j^T\mathbf{x}+c_j)_+,\] with \(\|\boldsymbol{b}_j\|_1+|c_j|=1\). Moreover, since the construction in [42] yields \(|\bar a_j|s_j\le 1\), we have \(|a_j|\le c_\rho(f)\). If \(s_j=0\), the corresponding neuron is identically
zero and can be represented by taking \(a_j=0\) and any normalized pair \((\boldsymbol{b}_j,c_j)\) with \(\|\boldsymbol{b}_j\|_1+|c_j|=1\). Hence the same
network can be written as \[f_n(\mathbf{x})
=
\frac{1}{n}\sum_{j=1}^n
a_j(\boldsymbol{b}_j^T\mathbf{x}+c_j)_+,\] with \((a_j,\boldsymbol{b}_j,c_j)\in M_{c_\rho(f)}\) for \(j=1,\ldots,n\). Therefore, \(f_n\in
X_{n,c_\rho(f)}\). ◻
A. N. Tikhonov, A. V. Goncharsky, V. V. Stepanov, and A. G. Yagola, Numerical methods for the solution of ill-posed problems. Dordrecht, Netherlands: Kluwer Academic Publishers,
1995.
[2]
G. Cybenko, “Approximation by superpositions of a sigmoidal function,”Math. Control Signals Syst., vol. 2, no. 4, pp. 303–314, 1989, doi: 10.1007/BF02551274.
[3]
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,”Neural Netw., vol. 2, no. 5, pp. 359–366, 1989, doi: 10.1016/0893-6080(89)90020-8.
[4]
K. Hornik, “Approximation capabilities of multilayer feedforward networks,”Neural Netw., vol. 4, no. 2, pp. 251–257, 1991, doi: 10.1016/0893-6080(91)90009-T.
[5]
M. Stinchcombe and H. White, “Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions,” in International 1989 joint
conference on neural networks, 1989, pp. 613–617.
[6]
A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,”IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930–945, 1993, doi: 10.1109/18.256500.
[7]
M. Telgarsky, “Benefits of depth in neural networks,” in Conference on learning theory, 2016, pp. 1517–1539.
[8]
R. Eldan and O. Shamir, “The power of depth for feedforward neural networks,” in Conference on learning theory, 2016, pp. 907–940.
[9]
J. Lu, Z. Shen, H. Yang, and S. Zhang, “Deep network approximation for smooth functions,”SIAM J. Math. Anal., vol. 53, no. 5, pp. 5465–5506, 2021, doi: 10.1137/20M134695X.
[10]
Z. Shen, H. Yang, and S. Zhang, “Optimal approximation rate of ReLU networks in terms of width and depth,”J. Math. Pures Appl., vol. 157, pp. 101–135,
2022, doi: 10.1016/j.matpur.2021.07.009.
S. Hon and H. Yang, “Simultaneous neural network approximation for smooth functions,”Neural Networks, vol. 154, pp. 152–164, 2022, doi: 10.1016/j.neunet.2022.06.040.
[13]
W. E and B. Yu, “The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems,”Commun. Math. Stat., vol. 6, no. 1,
pp. 1–12, 2018, doi: 10.1007/s40304-018-0127-z.
[14]
J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,”J. Comput. Phys., vol. 375, pp.
1339–1364, 2018, doi: 10.1016/j.jcp.2018.08.029.
[15]
M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial
differential equations,”J. Comput. Phys., vol. 378, pp. 686–707, 2019, doi: 10.1016/j.jcp.2018.10.045.
[16]
Y. Zang, G. Bao, X. Ye, and H. Zhou, “Weak adversarial networks for high-dimensional partial differential equations,”J. Comput. Phys., vol. 411, p. 109409, 2020,
doi: 10.1016/j.jcp.2020.109409.
[17]
A. Al-Aradi, A. Correia, G. Jardim, D. de Freitas Naiff, and Y. Saporito, “Extensions of the deep galerkin method,”Appl. Math. Comput., vol. 430, p. 127287, 2022,
doi: 10.1016/j.amc.2022.127287.
[18]
J. N. Fuhg, A. Karmarkar, T. Kadeethum, H. Yoon, and N. Bouklas, “Deep convolutional ritz method: Parametric PDE surrogates without labeled data,”Appl.
Math. Mech., vol. 44, no. 7, pp. 1151–1174, 2023, doi: 10.1007/s10483-023-2992-6.
[19]
E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes, “A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics,”Comput.
Methods Appl. Mech. Eng., vol. 379, p. 113741, 2021, doi: 10.1016/j.cma.2021.113741.
[20]
T. Kadeethum et al., “A framework for data-driven solution and parameter estimation of PDEs using conditional generative adversarial networks,”Nat. Comput. Sci., vol. 1, no. 12, pp. 819–829, 2021, doi: 10.1038/s43588-021-00171-3.
[21]
M. Raissi, A. Yazdani, and G. E. Karniadakis, “Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations,”Science, vol. 367, no. 6481,
pp. 1026–1030, 2020, doi: 10.1126/science.aaw4741.
[22]
Y. Wang et al., “Kolmogorov–Arnold-Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on
Kolmogorov–Arnold Networks,”Comput. Methods Appl. Mech. Eng., vol. 433, p. 117518, 2025, doi: 10.1016/j.cma.2024.117518.
[23]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998, doi: 10.1109/5.726791.
[24]
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted
intervention – MICCAI 2015, 2015, pp. 234–241, doi: 10.1007/978-3-319-24574-4_28.
[25]
G. Bao, X. Ye, Y. Zang, and H. Zhou, “Numerical solution of inverse problems by weak adversarial networks,”Inverse Probl., vol. 36, no. 11, p. 115003, 2020, doi:
10.1088/1361-6420/abb447.
[26]
S. Cen, B. Jin, K. Shin, and Z. Zhou, “Electrical impedance tomography with deep calderón method,”J. Comput. Phys., vol. 493, p. 112427, 2023, doi: 10.1016/j.jcp.2023.112427.
[27]
Q. Wang et al., “Error-constraint deep learning scheme for electrical impedance tomography (EIT),”IEEE Trans. Instrum. Meas., vol. 71, pp.
1–11, 2022, doi: 10.1109/TIM.2021.3135327.
[28]
J. Adler and O. Öktem, “Solving ill-posed inverse problems using iterative deep neural networks,”Inverse Probl., vol. 33, no. 12, p. 124007, 2017, doi: 10.1088/1361-6420/aa9581.
[29]
R. Barbano, Ž. Kereta, A. Hauptmann, S. R. Arridge, and B. Jin, “Unsupervised knowledge-transfer for learned image reconstruction,”Inverse Probl., vol. 38, no. 10,
p. 104004, 2022, doi: 10.1088/1361-6420/ac8a91.
[30]
K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convolutional neural network for inverse problems in imaging,”IEEE Trans. Image Process., vol. 26, no. 9,
pp. 4509–4522, 2017, doi: 10.1109/TIP.2017.2713099.
[31]
Z.-X. Cui, Q. Zhu, J. Cheng, B. Zhang, and D. Liang, “Deep unfolding as iterative regularization for imaging inverse problems,”Inverse Probl., vol. 40, no. 2, p.
025011, 2024.
[32]
K. Hammernik et al., “Learning a variational network for reconstruction of accelerated MRI data,”Magn. Reson. Med., vol. 79, no. 6, pp.
3055–3071, 2018, doi: 10.1002/mrm.26977.
[33]
J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert, “A deep cascade of convolutional neural networks for dynamic MR image reconstruction,”IEEE Trans. Med. Imaging, vol. 37, no. 2, pp. 491–503, 2018, doi: 10.1109/TMI.2017.2760978.
[34]
H. Li, J. Schwab, S. Antholzer, and M. Haltmeier, “NETT: Solving inverse problems with deep neural networks,”Inverse Probl., vol. 36, no. 6, p.
065005, 2020, doi: 10.1088/1361-6420/ab6d57.
[35]
D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.
9446–9454.
[36]
S. Dittmer, T. Kluth, P. Maass, and D. Otero Baguer, “Regularization by architecture: A deep prior approach for inverse problems,”J. Math. Imaging Vis., vol. 62,
no. 3, pp. 456–470, 2020, doi: 10.1007/s10851-019-00923-x.
A. G. Yagola, A. S. Leonov, and V. N. Titarenko, “Data errors and an error estimation for ill-posed problems,”Inverse Probl. Eng., vol. 10, no. 2, pp. 117–129,
2002, doi: 10.1080/10682760290031195.
[40]
K. Dorofeev and Y. Yagola, “The method of extending compacts and a posteriori error estimates for nonlinear ill-posed problems,”J. Inverse Ill-Posed Probl., vol.
12, pp. 627–636, 2004, doi: 10.1515/1569394042545120.
[41]
Y. Zhang, R. Gong, M. Gulliksson, and X. Cheng, “A coupled complex boundary expanding compacts method for inverse source problems,”J. Inverse Ill-Posed Probl.,
vol. 27, no. 1, pp. 67–86, 2019, doi: 10.1515/jiip-2017-0002.
[42]
Y. Li, S. Lu, P. Mathé, and S. V. Pereverzev, “Two-layer networks with the ReLU\(^k\) activation function:
Barron spaces and derivative approximation,”Numer. Math., vol. 156, no. 1, pp. 319–344, 2023, doi: 10.1007/s00211-023-01384-6.
[43]
W. E, C. Ma, and L. Wu, “Barron spaces and the compositional function spaces for neural network models,”arXiv preprint arXiv:1906.08039, 2019.
[44]
A. N. Tikhonov, A. S. Leonov, and A. G. Yagola, Nonlinear ill-posed problems. London, U.K.: Chapman; Hall, 1998.
[45]
P. Mathé, “Saturation of regularization methods for linear ill-posed problems in Hilbert spaces,”SIAM J. Numer. Anal., vol. 42, no. 3, pp. 968–973,
2004, doi: 10.1137/S0036142903420947.
[46]
Y. Zhang and B. Hofmann, “On the second-order asymptotical regularization of linear ill-posed inverse problems,”Appl. Anal., vol. 99, no. 6, pp. 1000–1025, 2020,
doi: 10.1080/00036811.2018.1517412.
[47]
L. Yuan and Y. Zhang, “A scaling fractional asymptotical regularization method for linear inverse problems,”Adv. Comput. Math., vol. 51, no. 1, p. 8, 2025, doi: 10.1007/s10444-025-10222-2.
[48]
T. Hein and B. Hofmann, “Approximate source conditions for nonlinear ill-posed problems – chances and limitations,”Inverse Probl., vol. 25, p. 035003, 2009, doi:
10.1088/0266-5611/25/3/035003.
[49]
B. Hofmann, B. Kaltenbacher, C. Pöschl, and O. Scherzer, “A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth
operators,”Inverse Probl., vol. 23, pp. 987–1010, 2007, doi: 10.1088/0266-5611/23/3/009.
[50]
J. Flemming, “Variational source conditions yield convergence rates,” in Variational source conditions,
quadratic inverse problems, sparsity promoting regularization: New results in modern theory of inverse problems and an application in laser optics, Cham, Switzerland: Springer International Publishing, 2018, pp. 15–20.
[51]
Y. Deng, B. Hofmann, and F. Werner, “Deautoconvolution in the two-dimensional case,”Electron. Trans. Numer. Anal., vol. 59, pp. 24–42, 2023, doi: 10.1553/etna_vol59s24.
K. Astala and L. Päivärinta, “Calderón’s inverse conductivity problem in the plane,”Ann. Math., vol. 163, no. 1, pp. 265–299, 2006.
[54]
S.-Y. Chung and Q. Sun, “Barron space for graph convolutional neural networks,”J. Fourier Anal. Appl., vol. 32, no. 3, p. 52, 2026, doi: 10.1007/s00041-026-10260-4.
\(^1\)School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100081, China.↩︎
\(^2\)Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong.↩︎
\(^3\)MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University, Shenzhen 518172, China.↩︎
\(^*\)Author to whom any correspondence should be addressed. Email: ye.zhang@smbu.edu.cn.↩︎
We write \(f\asymp g\) if there exist constants \(0<c\le C<\infty\), independent of the argument, such that \(cg(x)\le f(x)\le
Cg(x)\) for all \(x\) in the domain under consideration..↩︎
\(\Phi\) is called an index function if it is continuous and strictly increasing with \(\Phi(0)=0\).↩︎
