1 Introduction↩︎

By convention, an operator typically refers to a possibly non-linear mapping \(G:D\subseteq \mathcal{X}\to \mathcal{Y}\) between infinite-dimensional normed spaces \(\mathcal{X}\) and \(\mathcal{Y}\). Over the past five years, using neural networks for approximating and learning such operators, particularly between function spaces, has received increasing attention. For example, operator learning has been investigated in the field of partial differential equations (PDEs) when learning parameter-to-state maps, that map the parameter function of a PDE to the corresponding solution [1]–[8]. For some review on deep learning for PDEs, covering also operator learning and its application in parameter identification problems, we refer to [9].

For approximating operators \(G\), lots of neural network architectures have been developed. One of the historical starting points has been set by T. Chen and H. Chen [10] in 1995, whose approach has been rediscovered by Lu et al. [1] and generalized to the well-known deep operator networks (DeepONets). DeepONets fall under the category of encoder-decoder architectures, i.e., they consist of three building blocks \[\begin{align} G_\theta \mathrel{\vcenter{:}}= D \circ \varphi \circ E, \end{align}\] where the encoder \(E:\mathcal{X}\to \mathbb{R}^m\) extracts finite information about the input, \(\varphi:\mathbb{R}^m\to \mathbb{R}^n\) is a neural network and the decoder \(D:\mathbb{R}^n\to \mathcal{Y}\) maps into the output space of \(G\). This is illustrated in 1. The trainable parameters \(\theta\) of \(G_\theta\) belong to \(\varphi\), but can also parameterize the encoder and decoder. For example, in case of DeepONets and suitable function spaces \(\mathcal{X}\) and \(\mathcal{Y}\), the encoder evaluates the input function at finitely many sampling points, whereas the decoder outputs a linear combination of the so-called trunk networks, see Section 5.1.1 for more details. An example in which both the encoder and decoder are represented by neural networks, are the BasisONets presented in [11]. Further encoder-decoder approaches are, e.g., Principle Component Analysis Networks (PCANets, [3]), Multi Input Operator Networks (MIONets, [12]) which uses encoders corresponding to Schauder bases in Banach spaces, Deep-H-ONets [13] using encoders and decoders based on orthonormal bases, or the approach in [14] based on Riesz bases. Usually, the considered encoders and decoders are linear, but there are a few exceptions [15], [16].

As alternatives to encoder-decoder approaches we mention here, e.g., Neural Operators (NOs, [5]), Fourier Neural Operators (FNOs, [4]), Wavelet Neural Operators (WNOs, [8]), Representation Invariant Neural Operators (ReNOs, [7], [17]), Injective Integral Neural Operators [18], Causal Neural Operators [19] or Optimal Transport Neural Operators [20]. Some overview on operator learning methods can be found in [21].

1.1 Related work on operator approximation theory↩︎

On the theoretical side, a fundamental question is which classes of operators \(G:\mathcal{X}\to \mathcal{Y}\) can be approximated by neural networks and under which topology. When both \(\mathcal{X}\) and \(\mathcal{Y}\) are finite dimensional spaces (\(\mathbb{R}^d\) or \(\mathcal{X}\) being a subset thereof) this question falls within the well-established field of function approximation, which is not the primary focus of our study. Nevertheless, extensive results exist on the approximation properties of neural networks in various function spaces, including spaces of continuously differentiable functions, Lebesgue spaces, and Sobolev spaces, see for example [22]–[27]. Regarding the field of operator approximation, so when \(\mathcal{X}\) or \(\mathcal{Y}\) are infinite-dimensional, we recap below the two most commonly studied types of approximation, where the first is the focus of our study.

Uniform convergence on compacta. Let \(G:\mathcal{X}\to \mathcal{Y}\) be a continuous operator and \(K\subseteq \mathcal{X}\) be compact. The goal is to construct a sequence of approximants \(G_n:\mathcal{X}\to \mathcal{Y}\), such as encoder-decoder architectures as described above, that converges uniformly to \(G\) on \(K\). That is, \[\begin{align} \sup_{f\in K} \big \Vert G(f) - G_n(f) \big \Vert \xrightarrow{n\to\infty} 0. \end{align}\] Universal approximation theorems have been derived for a variety of neural network architectures and choices of the spaces \(\mathcal{X}\) and \(\mathcal{Y}\). Such theorems state that every continuous operator \(G\) can be approximated uniformly on compact subsets of \(\mathcal{X}\) by approximants \(G_n\) from the corresponding architecture class, where \(G_n\) usually depends on the respective compact set. Universal approximation results have been derived for

• DeepONets when \(\mathcal{X}\) and \(\mathcal{Y}\) are spaces \(\mathcal{C}(\Omega, \mathbb{R})\) of continuous real-valued functions on compact domains \(\Omega\subset \mathbb{R}^d\) [1], [10];

• MIONets when \(\mathcal{X}\) is a Banach space having Schauder bases and \(\mathcal{Y}=\mathcal{C}(\Omega, \mathbb{R})\) [12];

• Riesz-basis encoder-decoder networks when \(\mathcal{X}\) and \(\mathcal{Y}\) are separable Hilbert spaces [14];

• FNOs when \(\mathcal{X}\) and \(\mathcal{Y}\) are Sobolev spaces \(W^{s,2}(\Omega, \mathbb{R})\) with smoothness \(s\geq 0\) [28];

• NOs when \(\mathcal{X}\) and \(\mathcal{Y}\) are Sobolev spaces \(W^{s,p}(\Omega, \mathbb{R})\) (with \(1\leq p< \infty\) and \(s\geq 0\)) or spaces \(\mathcal{C}^k(\Omega, \mathbb{R})\) of continuously differentiable functions [29], [30];

• Injective Integral Neural Operators when \(\mathcal{X}\) and \(\mathcal{Y}\) are Lebesgue spaces \(L^2(\Omega, \mathbb{R})\) [18];

• Neural operators based on nonlinear projection operators when \(\mathcal{X}\) and \(\mathcal{Y}\) are Banach spaces [16];

• Transformers when \(\mathcal{X}=\mathcal{P}(\Omega)\times \Omega\) and \(\mathcal{Y}=\mathbb{R}^k\), where \(\Omega\subseteq \mathbb{R}^d\) is compact and \(\mathcal{P}(\Omega)\) denotes the set of Borel probability measures equipped with any \(p\)-Wasserstein distance for \(p\geq 1\) [31].

• Neural Filters when \(\mathcal{X}\) and \(\mathcal{Y}\) are Fréchet spaces having Schauder bases [19]. Related results when \(\mathcal{X}\) is a Fréchet space having a Schauder basis and when \(\mathcal{Y}\) is a Banach space can be found in [32].

Although approximation theorems are typically stated for spaces of \(\mathbb{R}\)-valued functions, they often naturally extend to \(\mathbb{R}^a\)-valued functions.

Approximation in Bochner spaces. Let \(\mathcal{X}\) be a measurable space, \(\mathcal{Y}\) a Banach space and \(\mu\) a probability measure on \(\mathcal{X}\). For a given operator \(G\) belonging to the Bochner space \(L^p(\mathcal{X}, \mathcal{Y}; \mu)\), one is looking for a sequence \(G_n\), being representable by neural networks, that converges to \(G\) in the Bochner norm (for suitable \(1\leq p <\infty)\), i.e., \[\begin{align} \big \Vert G - G_n\big \Vert_{L^p(\mathcal{X}, \mathcal{Y}; \mu)} = \left(\int_{\mathcal{X}} \big \Vert G(f) - G_n (f) \big \Vert^p \mathop{} \! \mathrm{d}\mu(f)\right)^{1/p} \xrightarrow{n\to\infty} 0. \end{align}\] Also for convergence in Bochner space, universal approximation theorems have been derived for different architectures, e.g., for DeepONets [33], PCANets [3], Riesz-basis encoder-decoder networks [14], FNOs [28], NOs [29], [30] or Deep-H-Onets [13].

For both types of approximation, there do not only exist universal approximation theorems, but also results on estimating the required size of the neural network to achieve a desired approximation accuracy. Typically, specific input spaces of the operator \(G\) and additional assumptions on \(G\) are required, e.g., that \(G\) arises from specific PDEs, is holomorphic, or Lipschitz/Hölder continuous, see e.g., [14], [19], [28], [33]–[36]. For more information about this important field of research we refer to the review [21], in which also the curse of dimensionality is discussed. Within this work, we restrict our attention to universal approximation results only.

There are also other types of approximating operators. For example, in [36] continuous operators \(G\) between Polish spaces \(\mathcal{X}, \mathcal{Y}\) are approximated by encoder-decoder architectures with respect to the 1-Wasserstein metric. Last but not least, in [37], dual spaces \(\mathcal{X}\) and \(\mathcal{Y}\) of separable Banach spaces were considered. If \(\mathcal{Y}\) is equipped with the (metrizable) weak\(^\ast\)-topology, a sequence of generalized ReLU-networks has been found that uniformly approximates a given operator (belonging to some variation norm space) on bounded sets. Further, approximation in Bochner spaces \(L^p(\mathcal{X}, (\mathcal{Y}, d_\ast); \mu)\) with suitable probability measures \(\mu\) was shown.

1.2 Contribution↩︎

To derive universal operator approximation results, we consider the space \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) of continuous mappings between normed or metric spaces \(\mathcal{X}\) and \(\mathcal{Y}\). Throughout this work, we study approximation in the topology on \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) induced by the uniformity of uniform convergence on compact sets; see [38] for a detailed definition. With respect to this topology, there are two distinct notions of approximating any continuous operator \(G\in \mathcal{C}(\mathcal{X}, \mathcal{Y})\) by some class \(S\) of approximators \(G_n:\mathcal{X}\to \mathcal{Y}\). These notions are illustrated in 2. In this work, we focus on \(S\) being diverse classes of encoder-decoder architectures.

\(\displaystyle \begin{tabular}{p{.535\textwidth}} \begin{enumerate}[leftmargin=0.2cm, , label=(\boldsymbol{\Alph*})] \item\) G(, ) \(\label{fig:approximation95types:40A41} \item\) G(, ) \(\label{fig:approximation95types:40B41} \end{enumerate} \end{tabular} \raisebox{-0.1cm}{{\scalebox{1.5}{\Bigg\}}}} \raisebox{-0.08cm}{\begin{aligned} \sup_{f\in K} d_\mathcal{Y}&\big(G (f)\, ,\, G_n (f)\big)\\[-0.1cm] &\xrightarrow{n\to\infty} 0. \end{aligned}}\)

Our main contributions to the approximation theory of encoder-decoder architectures in the topology of uniform convergence on compact sets are as follows:

• Proof of statement .
  In most other studies, statement in 2 is shown for \(S\) being, for example, DeepONets or (F)NOs. That is, for every compact \(K\subseteq\mathcal{X}\) one can find a sequence \((G_n)_{n\in \mathbb{N}}\) in \(S\) that converges uniformly to a given operator \(G\in \mathcal{C}(\mathcal{X}, \mathcal{Y})\) on \(K\). However, can the sequence \(G_n\) be chosen independently of the compact set \(K\)? To the best of our knowledge, there exists only one result by Schwab et al. [14], who have shown this statement for \(S\) being encoder-decoder networks based on Riesz bases in separable Hilbert spaces \(\mathcal{X}\) and \(\mathcal{Y}\). We show in our main 14 that statement - which is the major focus of our study - can indeed be achieved by much more diverse choices of encoder-decoder architectures. For example, our theory also covers classical DeepONets [1], [10], [33], special cases of MIONets [12], BasisONets [11] and Deep-H-ONets [13].
  Some advantage of over is that it could simplify the derivation of approximation theorems in Bochner spaces. This is because implies pointwise convergence, allowing one to invoke the dominated convergence theorem. We leave this application in Bochner approximation for future research.

• Proof that really is stronger than .
  A natural question arises: Is really a stronger statement than for metric spaces \(\mathcal{X}, \mathcal{Y}\)? In other words, is there a set \(S\) of approximators that satisfies but not ? The answer is yes if and only if \(\mathcal{X}\) is not hemicompact. In particular, the answer is yes if the input space \(\mathcal{X}\) is any infinite-dimensional normed space, which is typically the case for operator approximation tasks. In the case of \(\mathcal{Y}=\mathbb{R}\) and any separable, metrizable topological space \(\mathcal{X},\) this question has been answered in [39], which easily extends to any finite dimensional \(\mathcal{Y}\). However, we could not find a result for infinite-dimensional \(\mathcal{Y}\) within the literature. Therefore, we provide a compact proof in 3 for any metric space \(\mathcal{X}\) and normed space \(\mathcal{Y}\). Nevertheless, these topological considerations are not required for understanding the other parts of this study.

• One Theorem for many architectures and spaces.
  Within the literature, approximation results are often shown individually for different operator learning architectures. This means that an approximation theorem for a certain architecture class may not directly apply to a different architecture class, see for example 17. In contrast, we provide theorems that cover many classes of encoder-decoder architectures, as well as diverse normed and metric input and output spaces \(\mathcal{X}\) and \(\mathcal{Y}\), at once: For approximation as in statement we provide our main 14, whereas for the weaker statement we provide 16 and 17. Applicability to well-known architectures is outlined in Section 5.1. Noteworthy, in [36] and [40] the authors proved statement , but not (whenever \(\mathcal{X}\) is not hemicompact), for broad classes of encoder-decoder architectures and diverse metric/topological input and output spaces. Nevertheless, neither their results nor 16 are more general than the other regarding possible choices of encoders and decoders, see 13.

• Identify sufficient properties of input and output spaces.
  Regarding statement , our results significantly extend the existing theory for encoder-decoder architectures regarding possible input and output spaces, as well as possible encoder and decoder constructions. Regarding the theory related to statement , we also make contributions, although the existing literature in this area is more extensive. For that, we introduce in Sections 3 and 4.1 sufficient properties of the normed or metric input and output spaces \(\mathcal{X}\) and \(\mathcal{Y}\), which enable approximation of continuous operators by encoder-decoder architectures as in statements and , respectively. Inspired by the (bounded) approximation property of normed spaces, our conditions require approximation of the identity mappings on the input and output spaces by encoder-decoder pairs rather than by linear finite-rank operators. The corresponding encoders and decoders can then be used directly in the approximation [thm:sequential_density_normed,thm:density_metric,thm:mionet], which allows for a straight-forward application to many classes of architectures. Further, we relax several assumptions often imposed in the operator-learning literature. In particular, our framework accommodates metric spaces, permits nonlinear encoders and decoders, allows encoders to be discontinuous, and does not require decoders to be Lipschitz continuous. We give examples of metric spaces along with natural, constructive choices of encoders and decoders that benefit from these relaxations, see the point below.

• Theory covers metric spaces: \(p\)-Wasserstein and Skorohod spaces.
  To the best of our knowledge, the existing operator learning theory for statement handles operators between separable Hilbert spaces. Our frameworks for both and additionally applies to every normed space possessing the (bounded) approximation property, including, for example, Lebesgue spaces, Sobolev spaces, and spaces of continuously differentiable functions. In addition, also metric spaces can be treated as input or output spaces within our frameworks. For example, we prove that spaces of probability measures \(\mathcal{P}_p(\Omega)\) equipped with the \(p\)-Wasserstein (\(p\geq1\)) distance as well as Fréchet spaces having Schauder bases can be treated as input and output spaces. Moreover, we show that Skorohod spaces of càdlàg functions equipped with the Skorohod topology can be treated as input spaces in our theory. In many spaces, explicit and natural constructions of suitable encoders and decoders are possible, which we outline in Sections 3.1 and 3.2 for normed and metric spaces, respectively. Note that some of our natural encoders for Wasserstein and Skorohod spaces are currently not covered by the existing theory even for statement , which is due to their discontinuity, see also 13. Furthermore, in Section 5.2 we point out some potential applications in optimal transport, for learning operators \(\mathcal{P}_p(\Omega)\to \mathcal{Y}\) with encoder-decoder architectures. In particular, we discuss how 17 provides a possible route towards a first approximation theorem for geodesic operator networks [41].

1.3 Outline↩︎

In 2, we discuss the topological difference between statements and shown in 2. We highlight that is generally a stronger result in most contexts relevant to operator approximation. Nevertheless, this section is not required for understanding the remainder of this work. In Section 3, we introduce sufficient properties of the input and output spaces \(\mathcal{X}\) and \(\mathcal{Y}\) that enable the approximation of continuous operators \(G\in \mathcal{C}(\mathcal{X}, \mathcal{Y})\) by encoder-decoder architectures as in statement . We also discuss several examples of normed and metric spaces satisfying these properties and present explicit choices of encoders and decoders that can be used within our approximation theorems. In Section 4, we establish our main universal approximation results: 14, which yields approximation as in statement , and 16 for approximation as in . In Section 5.1, we discuss the applicability of our theoretical framework to existing encoder-decoder architectures and show how our results unify and extend several universal approximation theorems from the literature. Finally, in Section 5.2, we discuss some applications in optimal transport. This is motivated by the fact that our theoretical framework allows the consideration of \(p\)-Wasserstein spaces as input spaces.

1.4 Notation↩︎

Given a metric \(d\) on a set \(\mathcal{X}\), we write \((\mathcal{X}, d)\) for the corresponding metric space. We sometimes omit \(d\) in the notation. Given a metric space \((\mathcal{X}, d)\), the open ball of radius \(r>0\) around some \(x\in \mathcal{X}\) is denoted by \(B_r(x;d)\). If there is no ambiguity in the choice of \(d\), we write \(B_r(x).\) The closed ball is written as \(\overline{B}_r(x)\). Whenever the symbol \(\mathbb{K}\) is used, it is meant that the according definitions or results are valid for both fields \(\mathbb{K}=\mathbb{R}\) or \(\mathbb{K}=\mathbb{C}.\) All vector spaces considered within this study are meant to be over the field(s) \(\mathbb{K}\). For metric spaces \(\mathcal{X}\) and \(\mathcal{Y}\), we denote by \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) the space of continuous mappings \(\mathcal{X}\to \mathcal{Y}\), and by \(\mathcal{C}_b(\mathcal{X}, \mathcal{Y})\) its subset of bounded continuous mappings. Whenever \(\mathcal{Y}\) is a vector space, both spaces are regarded as vector spaces with pointwise addition and scalar multiplication. Finally, we denote the Euclidean norm on \(\mathbb{K}^n\) as \(\vert\cdot \vert\). Nevertheless, due to the equivalence of norms, also different choices of norms are possible throughout our study. The expressions \(\vert \cdot \vert_1\) and \(\vert\cdot\vert_\infty\) are used for the \(\ell^1\)- and \(\ell^\infty\)-norm, respectively.

2 Perspective from the compact-open topology↩︎

In this section, we will have a closer look on the topological difference between the two types of operator approximation given in 2 by a set \(S\) of approximants. For simplicity, we assume that \(S\subseteq \mathcal{C}(\mathcal{X}, \mathcal{Y})\). In the following two sections, we recall what the statements and in 2 mean in terms of the so called compact-open topology. Finally, in Section 2.3 we show that is indeed a stronger statement in general, at least for settings relevant for operator approximation tasks, in which \(\mathcal{X}\) and \(\mathcal{Y}\) are typically infinite-dimensional normed (function) spaces. Regarding the definition of the compact-open topology, we follow [38].

It is to be mentioned that the compact-open topology coincides with the topology induced by the uniformity of uniform convergence on compacta, e.g., when \(\mathcal{X}\) and \(\mathcal{Y}\) are metric spaces [38]. Nevertheless, we choose the perspective from the compact-open topology, as its definition is straightforward and since the studies we reference use this perspective, too.

2.1 Density↩︎

In this section, we recall in 1 that statement in 2 is equivalent to \(S\subseteq \mathcal{C}(\mathcal{X}, \mathcal{Y})\) being dense with respect to the compact-open topology. We start with basic definitions of adherent points and dense sets in topological spaces.

In other words, \(S\subseteq \mathcal{Z}\) is dense if its closure is the whole space \(\mathcal{Z}\). The following Lemma characterizes adherent points in the compact-open topology. It seems to be a well-known fact, but we could not find an explicit statement within the literature. Therefore, for the interested reader, we provide a proof in the appendix (see 7.1).

An immediate consequence of the preceding lemma is the following characterization of dense sets.

2.2 Sequential density↩︎

We have seen that with a dense set \(S\subseteq \mathcal{C}(\mathcal{X}, \mathcal{Y})\) one can approximate every \(G\in \mathcal{C}(\mathcal{X}, \mathcal{Y})\) on any compact \(K\subseteq \mathcal{X}\) by a sequence, which depends on \(K\). If we require these sequences to be independent on \(K\), so if statement is supposed to hold, we will recall in 2 below that this is equivalent to sequential density of \(S\) in \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) with respect to the compact-open topology.

In other words, a set \(S\) is sequentially dense if its sequential closure is the whole space. The following characterization of convergent sequences in the compact-open topology can be found, for example, in [42], which immediately allows for a characterization of sequentially dense sets in 2.

2.3 Density versus sequential density in the compact-open topology↩︎

In the previous sections, we characterized the different types of operator approximation given in 2 in the context of the compact-open topology. According to 1, statement in 2 means that \(S\) is dense with respect to the compact-open topology, whereas is equivalent to sequential density of \(S\) due to 2. The question whether it is an improvement to prove and not only hence corresponds to the question whether dense subsets of \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) are always sequentially dense w.r.t. the compact-open topology. In fact, this question is closely related to the Fréchet-Urysohn property. For the definition, we follow [38], in which the term Fréchet space is used.

Clearly, if a space is a Fréchet-Urysohn space, then dense sets are particularly also sequentially dense. However, the converse may not be true in general, see for example 20. On the other hand, for the compact-open topology on \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\), the converse holds for many choices of \(\mathcal{X}\) and \(\mathcal{Y}\), as revealed by the following theorem.

For \(\mathcal{Y}=\mathbb{R}\) and separable, metrizable \(\mathcal{X}\), this result has been derived in [39], which easily extends to any finite dimensional \(\mathcal{Y}\). In 3, we also treat infinite-dimensional \(\mathcal{Y}\). Further, for the case that \(\mathcal{X}\) is a separable metric space and for certain metrizable spaces \(\mathcal{Y},\) equivalence between and has been shown in [43].

In the remainder of this section, we will prove the implication " \(\to\) " with [lem:denseFU_k_sequence,lem:select_k_sequence,lem:ksequence_hemicompact]. Implication " \(\to\) " is handled in 6. For the proofs we follow the ideas presented in [44] and [39]. Note that " \(\to\) " is evident. To start, let us recall the definitions of hemicompact and locally compact spaces, for which we follow [45] and [38], respectively.

Note that every hemicompact metric space \(\mathcal{X}\) must be separable, as it can be written as a countable union of compact metric spaces, where the latter are separable (see 19). For non-separable \(\mathcal{X}\), 3 hence implies that there is always a dense set \(S\subset \mathcal{C}(\mathcal{X}, \mathcal{Y})\) which is not sequentially dense. This shows that and above are not equivalent in general. An explicit construction of such a set is provided in 20, for which we are grateful to Hendrik Vogt from the University of Bremen for his valuable contribution to this example. Further, if \(\mathcal{X}\) is an infinite-dimensional normed space, it follows that it cannot be locally compact, as the closed balls \(\overline{B}_r(x)\) for \(x\in \mathcal{X}\) and \(r>0\) are never compact. A combination of 3 as well as the [lem:denseFU_k_sequence,lem:ksequence_hemicompact] leads to the fact that every hemicompact metric space is locally compact (alternatively, see also [46]). Therefore, 3 also reveals that for any infinite-dimensional normed space \(\mathcal{X}\), there must be dense sets in \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) which are not sequentially dense.

In order to prove 3, we need to settle down some more terminology and introduce the notion of a so-called open \(k\)-cover and a \(k\)-sequence, for which we follow [44] and [39].

Let us remark that a compact space \(\mathcal{X}\) cannot have a \(k\)-cover, simply because \(\mathcal{X}\neq U_i\) for every \(U_i\) in the \(k\)-cover. Further, every \(k\)-cover must be an infinite set. Otherwise, one could choose \(x_i\in \mathcal{X}\setminus U_i\) for all \(i\in I\) (finitely many), so the compact set \(\{x_i: i\in I\}\) would not be contained in any \(U_i\). Further, the union over all elements of an open \(k\)-cover is the whole space (so a \(k\)-cover really is a cover of the space).

Clearly, every \(k\)-sequence of open sets is also an open \(k\)-cover. We say that an open \(k\)-cover \(\mathfrak{U} = \{U_i: i\in I\}\) contains a \(k\)-sequence if there is a sequence \((U_{i_n})_{n\in \mathbb{N}}\) in \(\mathfrak{U}\) which is a \(k\)-sequence. The following Lemma draws a connection between 3 and the property that every open \(k\)-cover of \(\mathcal{X}\) contains a \(k\)-sequence. Note that within the literature, the latter characteristic of the space \(\mathcal{X}\) is also sometimes referred to as \(\mathcal{X}\) being a \(\gamma_k\)-set [39] or \(\mathcal{X}\) fulfilling the \(\gamma_\mathfrak{I}\text{-property}\) for the ideal \(\mathfrak{I}\) consisting of all compact subsets of \(\mathcal{X}\) [43]. The idea for the proof is inspired by [39] in which the case \(\mathcal{Y}=\mathbb{R}\) was considered.

The next lemma is similar to [44]. However, we prove a weaker statement which will simplify the proof, but still be sufficient for our purpose of proving implication "\(\rightarrow\)" in 3.

Finally, we are able to show hemicompactness of \(\mathcal{X}\) in the next lemma, where the idea of the proof is inspired by [44].

Together with 3, we have thus shown implication "\(\rightarrow\)" in 3. Namely, we have shown that a metric space \(\mathcal{X}\) must be hemicompact if every dense subset of \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\) is sequentially dense. For completing the proof of 3, it only remains to show the following.

It is to be mentioned that the previous result also follows from [46], in which it has been shown that \(\mathcal{C}(\mathcal{X}, \mathcal{Y})\), equipped with the compact-open topology, is metrizable if \(\mathcal{X}\) is hemicompact. Note that every metrizable space is a Fréchet-Urysohn space. Moreover, in [45] it was shown that for \(\mathcal{X}\) being any topological space, \(\mathcal{C}(\mathcal{X},\mathbb{R})\) is metrizable if and only if \(\mathcal{X}\) is hemicompact.

3 Approximation properties↩︎

During this section, we present sufficient properties for metric spaces \(\mathcal{X}\) and \(\mathcal{Y}\), so that every continuous operator \(G\in \mathcal{C}(\mathcal{X}, \mathcal{Y})\) can be approximated by the specific encoder-decoder architectures constructed in Section 4. Let us start by defining a general encoder-decoder architecture.

A similar definition has been given, for example for normed spaces in [21], but therein, the encoder and decoder were supposed to be linear and continuous, which we both do not require here. Recall that one of our main goals is to approximate any operator \(G\) by a sequence \(G_n\) of encoder-decoder architectures uniformly on every compact \(K\subseteq \mathcal{X}\), see statement in 2. For that, one has to ensure that the encoding does not loose too much information from the input space \(\mathcal{X}\), whereas the decoding can provide enough information in the output space \(\mathcal{Y}\). We introduce the following encoder-decoder approximation properties (EDAP) of metric spaces which will be sufficient for deriving our universal approximation 14. If approximation as in statement is desired, we provide similar properties in Section 4.1.

If the space \(\mathcal{X}\) is clear from the context, we simply write \(E_n\mathrel{\vcenter{:}}= E_n^\mathcal{X}\) and \(D_n\mathrel{\vcenter{:}}= D_n^\mathcal{X}\) for the encoders and decoders, respectively.

Below we state two simple observations. First, both EDAPs are preserved when switching to any equivalent metric. Further, we provide a simple criterion for a subset \(M\subset \mathcal{X}\) to inherit the input-EDAP from \(\mathcal{X}\).

The definitions of the input- and output-EDAP are inspired by and resemble the approximation property (AP) of locally convex topological vector spaces, which has been intensively studied in [47], see also [48]. For 14 of the AP in normed spaces, we follow [49], which also contains a comprehensive survey of various types of approximation properties and their interrelations. In contrast to the EDAPs, the standard AP requires the approximating mappings \(T_n\) to be linear and continuous, and allows them to depend on the compact set \(K\). A key feature of the EDAP framework is that it is formulated entirely in terms of the metric structure and does not require compatibility with any underlying linear structure. This allows one to treat spaces such as \(p\)-Wasserstein spaces (\(p \geq 1\)), as well as Skorohod spaces. Note that Skorohod spaces are vector spaces, but their topology is not compatible with the linear structure, in the sense that the addition of vectors is not continuous. That these metric spaces indeed have the input- or output-EDAP is outlined in Section 3.2. There, we also present natural examples of discontinuous encoders for the input-EDAP and non-Lipschitz decoders suitable for the output-EDAP.

Below we state [29], which is a universal approximation theorem, in which the AP has been used for constructing suitable encoder-decoder architectures. We provide a similar universal approximation result for more diverse encoder-decoder architectures in 4.1, and a more relaxed version of the AP.

In other words, the AP enabled universal operator approximation as in statement of 2, where the approximating sequence \(G_{K, n}\) depends on the compact set \(K\). However, since our goal is to get rid of this dependence on \(K\) in statement , the standard AP is not a sufficient property, which motivated our definition of the input- and output-EDAP with \(K\)-independent mappings \(T_n\). Important examples of spaces possessing both EDAPs are separable normed spaces that have the bounded approximation property (BAP). Since we have already observed that any space with either the input-EDAP or the output-EDAP is necessarily separable, we restrict the definition of the BAP below to separable normed spaces. For separable Banach spaces, an overview of the relationships among different approximation properties is shown in 3.

In the following, we discuss different examples of spaces having the input- and output-EDAP, and describe various explicit choices for suitable encoders and decoders. First, normed spaces are considered, followed by metric spaces.

3.1 Examples of normed spaces↩︎

Before discussing examples of normed spaces having the EDAPs, we start with a helpful lemma which simplifies proving that a Banach space \(\mathcal{X}\) has the BAP, so in particular also both the input- and output-EDAP (if \(\mathcal{X}\) is separable). It follows from the Banach-Steinhaus theorem, also known as the uniform boundedness principle. For the latter, we refer to [52].

An overview of the constructed encoders and decoders for the provided examples below can be found in ¿tbl:tab:encoder95decoder95overview?.

3.1.1 Banach spaces with Schauder bases↩︎

As a first straightforward example of spaces that fulfill the BAP, hence both the input- and output-EDAP, we discuss Banach spaces with Schauder bases in what follows. We restrict the attention to infinite-dimensional spaces and for a definition, we follow [53]. However, finite-dimensional spaces can be treated analogously.

Note that the uniqueness of the coefficients already implies that the coefficient functionals \(c_i:\mathcal{X}\to \mathbb{K}\) are linear and countinuous, since \(\mathcal{X}\) is a Banach space. Therefore, and due to 9, the following well-known observation is immediate.

Lots of separable Banach spaces possess Schauder bases and hence also have the BAP, input- and output-EDAP. For example, Schauder bases exist in every separable Hilbert space, in several spaces of continuously differentiable functions and Lebesgue spaces (except \(L^\infty\)) [54], and in some Sobolev spaces [55]. It is to be noted that not every separable Banach space has a Schauder basis, nor fulfills the BAP. A first counterexample has been found by P. Enflo in 1973 [56].

3.1.2 Frames in Hilbert spaces↩︎

Frames are a generalization of Schauder bases, as they relax the uniqueness constraint on the basis coefficients \(c_i\). For the sake of a simpler notation, we consider in what follows the classical case of an infinite-dimensional, separable Hilbert space \(\mathcal{X}\) with inner product \(\langle \cdot, \cdot\rangle.\) However, note that finite-dimensional spaces can be handled in a similar way [58]. Furthermore, frames can be generalized to non-separable Hilbert spaces [59] as well as to a Banach space setting [58].

For a given a frame \((f_i)_{i\in \mathbb{N}}\) in \(\mathcal{X}\), it is well-known that there exists a canonical dual frame of \((f_i)_{i\in \mathbb{N}}\) defined by \(\big(S^{-1}(f_i)\big)_{i\in \mathbb{N}} \subset \mathcal{X}\) via the inverse of the frame operator \(S: \mathcal{X}\to\mathcal{X}\) given by \(S(f) \mathrel{\vcenter{:}}= \sum_{i=1}^\infty\langle f, f_i\rangle f_i,\) so that every \(f\in \mathcal{X}\) can be expressed as \[f = \sum_{i=1}^\infty \big\langle f,\, S^{-1} (f_i)\big\rangle f_i,\] see [58]. This motivates the definition of, more generally, dual frames.

A frame has a dual frame different from the canonical dual frame if and only if it is not a Riesz basis, see, e.g., [58]. In the following, we state the main result of this section, which will give rise to the construction of encoders and decoders based on frames.

3.1.3 Sampling in spaces of continuous functions↩︎

Given some compact metric space \(\Omega\) and \(a\in \mathbb{N}\), we consider the space \(\mathcal{X}=\mathcal{C}(\Omega, \mathbb{K}^a)\) equipped with the supremum norm throughout this section. If \(\Omega\) is uncountable, it is due to Milutin’s theorem that \(\mathcal{C}(\Omega, \mathbb{K}^a)\) is isomorphic to \(\mathcal{C}([0,1], \mathbb{K}^a)\), see for example [54]. Since the latter has a Schauder basis, according to [53], also \(\mathcal{C}(\Omega, \mathbb{K}^a)\) must possess a Schauder basis, meaning that it has the BAP according to 3.1.1. However, the corresponding choice of \(T_n(f) = \sum_{i\leq n} c_i(f) b_i\) is not the only possible choice for \(T_n\) in 15 of the BAP, which we want to outline in what follows. The constructions within this section are inspired from [10], in which the case \(\mathcal{C}(\Omega, \mathbb{R})\) and \(\Omega\) being a compact subset of a Banach space was considered.

Note that the spaces in 1 have the BAP, but are not complete. When \(\mathcal{C}^k\) is equipped with the \(\mathcal{C}^k\)-norm, instead of the supremum norm, one can also show that it has the BAP with suitable sampling encoders. We give an impression of how to construct suitable \(T_n\) in Appendix 8.1.

Finally, we provide in Appendix 8.2 a comparison of the above sampling encoders and the basis encoders discussed in Section 3.1.1. In 21 we show that, in general, Schauder basis encoders cannot coincide with sampling encoders for \(n\) large enough.

3.1.4 Substitution by dense sets↩︎

Throughout this section, let \(\mathcal{X}\) be a separable normed space having the BAP, with suitable mappings \(T_n\) from 15. As \(T_n\) maps into a finite dimensional space, we can choose a basis \((b_i^{(n)})_{i\in I_n}\) for \(T_n(\mathcal{X})\), with \(I_n\) being a finite index set. Therefore, we can write \[\begin{align} \label{eq:BAP95range95basis95representation} T_n f = \sum_{i\in I_n} c^{(n)}_i(T_n f) b_i^{(n)}, \end{align}\tag{4}\] where \(c_i^{(n)}:T_n(\mathcal{X})\to \mathbb{K}\) are linear and continuous functionals. For some dense set \(S\subseteq \mathcal{X}\), we will see that one can also replace the basis \(b_i^{(n)}\) of \(T_n(\mathcal{X})\) by some vectors from \(S\), without loosing the ability to approximate the identity uniformly on compact sets. The decoders below will be useful in Section 5.1 when discussing classical DeepONets or MIONets.

If \(\mathcal{X}\) is a Hilbert space, it is possible to construct an encoder that also uses \(v_i^{(n)}\). This will be of interest in 5.1.3 when discussing BasisONets [11].

3.2 Examples of metric spaces↩︎

In this section, we discuss several well-known metric spaces that are neither normed spaces nor even topological vector spaces, yet possess the input- or output-EDAP. These examples demonstrate the advantage of formulating the EDAPs in the general setting of metric spaces. Classical approximation properties such as the AP and BAP are based on approximation by finite-rank linear operators and are therefore naturally associated with Banach spaces or more generally locally convex topological vector spaces [47]–[49]. Furthermore, we give examples of discontinuous, though natural, encoders that are nevertheless suitable for the input-EDAP. In addition, we provide examples of uniformly continuous but non-Lipschitz decoders that are suitable for the output-EDAP. Note that some examples were already given in 6.

3.2.1 Wasserstein spaces↩︎

Throughout this section, \((\Omega, d)\) denotes a complete, separable metric space. For \(p\geq 1\), denote by \(\mathcal{P}_p(\Omega;d)\) the set of Borel probability measures on \(\Omega\) with finite moments of order \(p\). So \(\mu\in \mathcal{P}_p(\Omega;d)\) if for some \(x_0\in \Omega\) (hence for any \(x_0\)) it is \[\begin{align} \int_\Omega d(x, x_0)^p \mathop{} \! \mathrm{d}\mu(x) < \infty. \end{align}\] Note that if \(d\) is a bounded metric, \(\mathcal{P}_p(\Omega;d)\) coincides with the set \(\mathcal{P}(\Omega)\) of all probability measures. The main goal is to show in 10 that if \(\Omega\) is proper, \(\mathcal{P}_p(\Omega;d)\) has the input-EDAP when being equipped with the \(p\)-Wasserstein distance. In 4, we show that the set of compactly supported measures also has the input-EDAP. If \(\Omega\) is compact, also the output-EDAP is fulfilled, see 11. Further, we verify both EDAPs for the set of absolutely continuous measures in \(\mathcal{P}_p(\mathbb{R}^m; \vert \cdot \vert)\) in 5. For non-compact \(\Omega\), it remains an open question whether \(\mathcal{P}_p(\Omega; d)\) also has the output-EDAP.

For example, \(\Omega=\mathbb{R}^m\) or any of its closed subsets are proper, equipped with any norm. Note that every proper metric space is complete and separable. We consider the \(p\)-Wasserstein metric, also called Kantorovich-Rubinstein distance, where we follow [61].

In order to show that \((\mathcal{P}_p(\Omega; d), W_p)\) has the input-EDAP, we construct suitable encoders and decoders in 22. We begin with the following elementary result.

Note that \(E_{a,r}\) is not continuous with respect to the \(p\)-Wasserstein metric \(W_p\), simply because \(W_p\)-convergence does not necessarily imply strong convergence of probability measures. Furthermore, note that \(A_{a,r}\subset \mathbb{R}^{k(a,r)}\) is closed and bounded, hence compact.

The following two results are extracted from the proof in [61]. For \(R>0\), denote by \(W^{(R)}_p(\mu, \nu)\) the \(p\)-Wasserstein distance between \(\mu, \nu\in \mathcal{P}_p(\Omega;d_R)\) w.r.t. to the bounded metric \(d_R(x,y)\mathrel{\vcenter{:}}= \min\{d(x,y), R\}\) on \(\Omega\).

2 can be used to show Hölder continuity of the decoders \(D_{a,r}\). Note that the assumption of \((\Omega,d)\) being proper in the next result is only relevant for the specific selection of points \(x_i^{(a,r)}.\)

Furthermore, we need a characterization of compact sets in \((\mathcal{P}_p(\Omega), W_p)\). To do so, we follow [62] and adjust the result to proper metric spaces.

Below, we observe the input-EDAP for two common subsets of \(\mathcal{P}_p(\Omega;d)\). The first result is an immediate consequence of 8 and 10 as the decoders in 10 map to compactly supported measures.

Note that in the next result, one could also consider absolutely continuous measures in \(\mathcal{P}_p(\Omega;\vert \cdot \vert)\) for certain closed subsets \(\Omega\subset \mathbb{R}^m\), which we avoid for better readability.

Below, we show that for any compact metric space \((\Omega, d)\), the \(p\)-Wasserstein spaces also have the output-EDAP. Note that for compact \(\Omega\), all \(\mathcal{P}_p(\Omega; d)\) coincide and \(\mathcal{P}(\Omega)\mathrel{\vcenter{:}}= \mathcal{P}_p(\Omega;d)\) is compact with respect to every \(p\)-Wasserstein metric, see [65].

3.2.2 Skorohod spaces \(\mathcal{D}([0, R])\)↩︎

Within this section, we consider Skorohod spaces, which are the set of càdlàg functions equipped with the so-called Skorohod topology. We show in 13 that càdlàg functions on \([0,R]\), equipped with any metric inducing the Skorohod topology, satisfy the input-EDAP. In 9, we verify this for càdlàg functions on \([0, \infty)\). It remains an open question whether these spaces also fulfill the output-EDAP. For definitions and basic properties, we mainly follow [64].

For simplicity, we restrict ourself to real-valued càdlàg functions. However, their definition as well as the considerations below might also be extended to càdlàg functions with values in other normed (or metric) spaces than \(\mathbb{R}\). Note that for \(R\in (0,\infty)\), \(\mathcal{D}([0,R])\) can be equipped with the supremum norm, but becomes a non-separable space. Instead, one can define the following metric \(d_R\), which induces the so-called Skorohod topology, and turns \(\mathcal{D}([0,R])\) into a separable metric space [64].

Of course, \(\big(\mathcal{D}([0,R]), d_R\big)\) is an \(\mathbb{R}\)-vector space with pointwise addition of càdlàg functions. However, it is not a topological vector space, as pointwise addition of càdlàg functions is not a topological group [64]. Equipped with \(d_R\), the space \(\mathcal{D}([0, R])\) is not complete, but there is a topologically equivalent metric \(d^\circ_R\), which turns it into a complete separable space [64]. Due to Lemma 7, the input-EDAP is preserved for topologically equivalent metrics. Thus, when showing below that \((\mathcal{D}([0, R]), d_R)\) has the input-EDAP, this automatically implies the input-EDAP of \((\mathcal{D}([0, R]), d_R^\circ)\), where the same encoders and decoders can be chosen. The consideration of the following mapping \(A_\sigma^{(R)}\) will be helpful, as it has a canonical decomposition into an encoder and decoder.

Sampling, and hence \(E_\sigma\) and \(A_\sigma^{(R)}\), is not continuous w.r.t. the metric \(d_R\) [64]. However, we show in what follows that for suitable \(\sigma\), the encoder and decoder fulfill the requirements in Definition 12 of the input-EDAP. Although it would be sufficient to prove continuity of the decoders, we show that they are even Lipschitz continuous.

We need the following result, which can be found in [64].

According to [64], relatively compact subsets of \((\mathcal{D}([0, R]), d_R)\) can be characterized as follows:

An immediate consequence of 15 and 12 is that if \(\sigma\) is fine enough within \([0,R]\), it follows that \(A_\sigma^{(R)}=D_\sigma \circ E_\sigma\) uniformly approximates the identity mapping on compact sets.

4 New universal operator approximation theorem↩︎

We recall that one of the main goals of this work is to approximate any continuous operator \(G:\mathcal{X}\to\mathcal{Y}\) between suitable separable metric spaces \(\mathcal{X}\) and \(\mathcal{Y}\) by a sequence of encoder-decoder architectures \(G_n=D_n^\mathcal{Y}\circ \varphi_n \circ E_n^\mathcal{X}:\mathcal{X}\to \mathcal{Y}\) that converges uniformly to \(G\) on every compact set \(K\subseteq \mathcal{X}\), see statement of 2. In this section, we prove in 14 that this is possible if encoders and decoders from the input- and output-EDAP are used for the construction of \(G_n\), see [def:input_EDAP,def:output_EDAP]. If only statement is desired, so when \(G_n\) is allowed to depend on \(K\), we further present a corresponding universal approximation theorem at the end of this section. For this weaker type of approximation, the definitions of the EDAPs can be relaxed, which would then also allow for the consideration of non-separable spaces.

Due to the encoding and decoding, the operator approximation task can be reduced to a function approximation task, that is, approximating functions between spaces \(\mathbb{K}^a\) by \(\varphi_n\) used in the encoder-decoder architecture \(G_n=D_n^\mathcal{Y}\circ \varphi_n \circ E_n^\mathcal{X}\). Therefore, the functions \(\varphi_n\) must be chosen from a sufficiently rich set of functions, which leads to the term of universal function approximators. A similar terminology and definition has been considered, for example, within the work of Kratsios et al. [36].

There are different possible choices for universal function approximators. For example, one can consider different types of neural network architectures such as fully connected or convolutional neural networks [22], [25], [66]. Moreover, polynomials build another type of universal function approximators according to the Stone-Weierstraß theorem [42].

The following basic lemma will be useful for showing our operator approximation theorem. Similar versions have been derived and used in several other studies for proving universal operator approximation results, see for example [1], [10], [11], [14], [28]. We state a general version below which is similar to [29], in which it was shown for Banach spaces \(\mathcal{X}\) and \(\mathcal{Y}\) and some continuous sequence of operators \(G_n\), using a different proof.

Some consequence of the previous lemma is the following result, which shows that uniform convergence on every compact set for sequences of operators is preserved under concatenation.

With the preceding considerations, we are now prepared to prove our main result: a universal operator approximation theorem for operators between spaces having the EDAP. It shows that operator approximation is possible by a sequence of encoder-decoder architectures that converge uniformly to the given operator on every compact set, which is statement in 2. As discussed in 3, there are diverse choices for suitable encoders and decoders. As we will point out in Section 5.1, the theorem is hence applicable, in particular, to many famous architectures used in the field of operator learning, but it is not restricted to neural networks.

In contrast to most results within the operator learning literature, the choice of the approximating sequence \(G_n\) in 14 is independent of the compact set \(K\subseteq \mathcal{X}\). As pointed out in 3, the previous result is hence a stronger result for most spaces \(\mathcal{X}\) relevant for operator approximation, for example, whenever \(\mathcal{X}\) is an infinite-dimensional normed space. Nevertheless, in [14], an analogous result has been achieved for separable Hilbert spaces \(\mathcal{X}, \mathcal{Y}\) and encoder-decoder architectures based on Riesz-bases, see also 5.1.4 for more information. To the best of our knowledge, [14] is the only other study showing uniform convergence of encoder-decoder architectures \(G_n\) towards \(G\) on every compact set \(K\). For other operator learning architectures, such as FNOs or WNOs, we have not found any result similar to the theorem above. However, we highly suspect that also for those approaches, it would be possible to derive such a result, which we leave as an interesting future task.

Furthermore, to the best of our knowledge, 14 is the first universal approximation theorem for encoder-decoder architectures that shows statement for continuous operators between metric spaces (satisfying suitable encoder-decoder approximation properties). In particular, the underlying spaces need not be topological vector spaces, and the input encoders need not be continuous. This allows the consideration of \(p\)-Wasserstein spaces as input and output spaces; see 3.2.1. Moreover, Skorohod spaces of càdlàg functions can be treated as input spaces; see 3.2.2.

The operators \(G\) in 14 were required to be defined on the whole space \(\mathcal{X}\). However, what if \(G\) is only defined on a subset \(A\subset \mathcal{X}\)? In 6 in the proof of 14, it was crucial that the decoder \(D_n^\mathcal{X}\) maps into the domain of the operator \(G\). Therefore, one cannot use, in general, the same encoders and decoders for \(A\) to obtain a similar result for operators \(G\in \mathcal{C}(A, \mathcal{Y})\). Further, it is not automatically clear whether the subset \(A\) itself has the input-EDAP. We suspect that this may not be true in general, because of the following analogous fact about the \(\lambda\)-BAP: There exists a separable Banach space \(\mathcal{X}\) having the \(\lambda\)-BAP, but there is a subspace \(A\subset \mathcal{X}\), which has not the \(\lambda'\)-BAP for any \(\lambda'>0\), see for example [67]. Of course, if also \(A\) has the input-EDAP, then 14 is applicable with suitable encoders and decoders on \(A\). Nevertheless, for closed \(A\subset \mathcal{X}\), one can indeed use the encoders from the input-EDAP of \(\mathcal{X}\) to approximate operators in \(\mathcal{C}(A, \mathcal{Y})\), see 8 below. This is due to Dugundji’s extension theorem [68].

4.1 Approximation depending on compact sets↩︎

In 14, the input- and output-EDAP of the input and output space has been used for constructing a sequence of encoder-decoder architectures that converges to a given operator uniformly on every compact set (statement in 2). If instead the weaker statement is desired, one can derive an analogous approximation theorem, see 16 below. For this, it is sufficient to consider the following weaker versions of the EDAPs (recall [def:input_EDAP,def:output_EDAP] of the EDAPs).

Evidently, if a metric space \(\mathcal{X}\) has the input- or output-EDAP, it has also the input- or output-CEDAP, respectively. Further, note that the AP (14) implies both the input- and output-CEDAP. Therefore, also non-separable spaces can have the CEDAPs, whereas this is impossible for the EDAPs. The following result generalizes [29], in which Banach spaces \(\mathcal{X}\) and \(\mathcal{Y}\) having both the AP were considered. The theorem below covers several existing universal approximation theorems of encoder-decoder networks, for example, for classical DeepONets [10] and [1], special cases of MIONets [12] and BasisONets [11], see 5.1 for more details on these implications. As outlined in 3.2, the following theorem also handles operators between metric input and output spaces, allowing also discontinuous encoders. For example, it enables the consideration of \(p\)-Wasserstein and Skorohod spaces.

5 Applications↩︎

In 14 and 16 we proved that continuous operators \(G:\mathcal{X}\to \mathcal{Y}\) can be approximated by encoder-decoder architectures if the metric spaces \(\mathcal{X}\) and \(\mathcal{Y}\) have the input- and output-(C)EDAP, respectively. As shown in 3, many spaces can be considered - even Wasserstein or Skorohod spaces - and there are diverse choices for suitable corresponding encoders and decoders.

In Section 5.1 below, we point out that our developed theoretical framework unifies and extends approximation theory for several well-known encoder-decoder architectures used in operator learning, such as classical DeepONets [1], [10], architectures based on frames or Riesz bases [14], BasisONets [11] or special cases of MIONets [12].

In Section 5.2, we illustrate some potential applications within optimal transport. For example, we propose a combination of optimal transport neural operators [20] with an encoder-decoder architecture. Further, we sketch how 14 may help to verify approximation capabilities of geodesic operator networks [41].

5.1 Well-known encoder-decoder architectures↩︎

In this section, we demonstrate that the approximation theory in 4 is modular in the sense that different encoder-decoder architectures, as well as different input and output spaces can be treated within a single unified framework. In contrast, within the operator learning literature, individual approximation results are derived for specific architectures and therefore do not automatically extend to other architectures. By considering more general architectures in 14 and 16, we obtain a more unified theoretical framework. Moreover, most approximation theorems in the literature establish only statement , whereas 14 proves the topologically stronger statement for, in particular, all architectures below. Throughout this section, we consider \(\mathbb{R}\)-vector spaces, in accordance with the standard setting in the operator learning literature. The analogous results for \(\mathbb{C}\)-vector spaces can be obtained in the same way.

5.1.1 Classical DeepONets↩︎

As a first example for approximating continuous operators \(G:\mathcal{X}\to \mathcal{Y}\), we consider classical DeepONets, which have been presented in [1], [33] and are inspired by the shallow architectures constructed within [10]. For the definition of a classical DeepONet, assume that \(\mathcal{X}=\mathcal{X}(\Omega_\mathcal{X}, \mathbb{R})\) and \(\mathcal{Y}= \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) are normed spaces of (for readability) \(\mathbb{R}\)-valued functions with domain spaces \(\Omega_\mathcal{X}\) and \(\Omega_\mathcal{Y}\), where \(\Omega_\mathcal{X}\) is a compact metric space and \(\Omega_\mathcal{Y}\subset \mathbb{R}^{d_\mathcal{Y}}\) is compact.

In parts of the literature, the term DeepONet is used more broadly to include architectures whose encoder is not necessarily based on point evaluations of the input function \(f\). To distinguish these settings, we use the term classical DeepONet for architectures that encode \(f\) through finitely many samples. This terminology allows for a more precise discussion of approximation-theoretic results for different encoder classes.

To obtain universal approximation theorems for classical DeepONets as in statement , one can simply interpret them as encoder-decoder architectures \(\hat{G} = D\circ \varphi \circ E\) and apply 14 as follows:

• Choose sampling encoders on \(\mathcal{X}(\Omega_\mathcal{X}, \mathbb{R})\) from 7 or 1;
  (Also possible: Encoders from 10 which use samples of \(f'\), or sampling encoders from 13 or 24 if \(\mathcal{X}\) is a Skorohod space.)

• Choose \(\varphi\) from a universal function approximator \(\mathcal{F}\) which consists of neural networks, see 26. Examples for \(\mathcal{F}\) can be found, e.g., in [22], [25], [66];

• Choose dense decoders on \(\mathcal{Y}\) from 8, that correspond to a dense set \(S\subset \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) of neural networks, so all \(\psi_i\) of the DeepONet should belong to \(S\). Examples for \(S\) can be found, e.g., in [25], [27] if \(\mathcal{Y}\) is a Lebesgue space, Sobolev space or space of continuously differentiable functions.

In particular, 14 implies and extends [1], in which the weaker statement was shown for classical DeepONets and the special case of \(\mathcal{X}=\mathcal{C}(\Omega_\mathcal{X}, \mathbb{R})\), \(\mathcal{Y}=\mathcal{C}(\Omega_\mathcal{Y}, \mathbb{R})\), and \(\Omega_\mathcal{X}\) being a compact subset of a Banach space.

The same setup for \(\mathcal{X}\) and \(\mathcal{Y}\) has been considered within [10] to derive statement for shallow classical DeepONets. Noteworthy, the constructed decoders for \(\mathcal{Y}=\mathcal{C}(\Omega_\mathcal{Y}, \mathbb{R})\) can be expressed as \[\begin{align} (D c)(x) = \sum_i c_i g(w_i \cdot x + b_i), \end{align}\] where \(g:\mathbb{R}\to \mathbb{R}\) is a suitable activation function and the parameters \(w_i\in \mathbb{R}^{d_\mathcal{Y}}, b_i\in \mathbb{R}\) depend on the compact set \(K\subset \mathcal{Y}\) on which approximation is desired. Note that these decoders are not dense decoders as in 8 since the set \(\{ x\mapsto g(w\cdot x+b): w\in \mathbb{R}^{d_\mathcal{Y}}, b\in \mathbb{R}\}\) is not dense in \(\mathcal{Y}=\mathcal{C}(\Omega_\mathcal{Y}, \mathbb{R})\). Nevertheless, it follows from [10] that there exist encoders \(E\), such that \(D\circ E\) are suitable choices for the output-CEDAP of \(\mathcal{Y}\). Hence, [10] is a special case of 16. Note that in [10], the activation function \(g\) is allowed to be discontinuous. Nevertheless, according to 11, one can just consider the decoder \(D\) as a mapping into the larger space of bounded, possibly discontinuous, functions \(\Omega_\mathcal{Y}\to \mathbb{R}\) and still obtain an approximation result by 16. For that, one needs to assume, however, that \(g\) maps compact sets into bounded sets.

5.1.2 MIONets↩︎

In this section, we discuss multi input operator networks (MIONets), which were introduced in [12]. MIONets are designed to approximate operators of the form \(\mathcal{X}_1\times ... \times \mathcal{X}_N\to \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\), where the \(\mathcal{X}_i\) are Banach spaces having Schauder bases. To simplify the presentation, we restrict our attention to the case \(N=2\). Moreover, we consider only low-rank MIONets, which constitute the default architecture in [12] due to their lower computational cost compared to high-rank variants. Let \(\mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) be again a normed (or metric) space of functions defined on a compact set \(\Omega_\mathcal{Y}\subset \mathbb{R}^{d_\mathcal{Y}}\).

In [12], it has been shown that continuous operators \(G:\mathcal{X}_1\times ... \times \mathcal{X}_N\to \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) can be approximated by MIONets as in statement , in which the choice of the approximating sequence of MIONets \(\hat{G}_n\) depends on the compact set \(K\subset \mathcal{X}_1\times...\times \mathcal{X}_N\). For \(N=1\), 14 yields the stronger approximation property described in statement for low-rank MIONets as follows:

• Choose basis encoders on \(\mathcal{X}_1\) from 5;

• As in Section 5.1.1, choose \(\varphi^{(1)}\) from a universal function approximator \(\mathcal{F}\) which consists of neural networks;

• As in Section 5.1.1, choose dense decoders on \(\mathcal{Y}\) from 8, that correspond to a dense set \(S\subset \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) of neural networks.

For \(N>1\), 14 can likewise be applied to approximate continuous operators \(\mathcal{X}_1\times ... \times \mathcal{X}_N\to \mathcal{Y}\). Indeed, one may view the Cartesian product as a single Banach space \(\mathcal{X}=\mathcal{X}_1\times\cdots\times\mathcal{X}_N\). It is straightforward to verify that \(\mathcal{X}\) inherits a Schauder basis from the spaces \(\mathcal{X}_i\). Applying 14 using Schauder basis encoders on \(\mathcal{X}\) then leads to architectures with a single \(\varphi\in \mathcal{F}\) that receives the Schauder basis coefficients associated with all spaces \(\mathcal{X}_i\). Consequently, the resulting architecture is not a low-rank MIONet.

To handle low-rank MIONets for \(N>1\), one can however combine 14 with the idea in [12] of identifying \(\mathcal{C}(K_1\times K_2, \mathcal{Y})\) with the injective tensor product \(\big(\mathcal{C}(K_1, \mathbb{R})\hat{\otimes}_\varepsilon\mathcal{C}(K_1, \mathbb{R})\big)\hat{\otimes}_\varepsilon \mathcal{Y}\) for compact sets \(K_i\subseteq\mathcal{X}_i\). By that, one obtains an approximation 17 as in statement , which extends [12] in that more general input spaces can be considered, and different encoders than basis encoders can be used, for example, sampling encoders or encoders from Section 3.2.1. We suspect that for \(N>1\), it is not possible, in general, to prove approximation by MIONets as in statement . This is because the construction of isomorphisms between \(\big(\mathcal{C}(K_1, \mathbb{R})\hat{\otimes}_\varepsilon\mathcal{C}(K_1, \mathbb{R})\big)\hat{\otimes}_\varepsilon \mathcal{Y}\) and \(\mathcal{C}(K_1\times K_2, \mathcal{Y})\) depends on the compact sets \(K_1\) and \(K_2\).

5.1.3 BasisONets↩︎

For classical and Schauder basis DeepONets, sampling and basis encoders were considered. In this section, we consider BasisONets, which were introduced in [11] as mappings between Lebesgue spaces \(\mathcal{X}=L^2(\Omega_\mathcal{X}, \mathbb{R})\) and \(\mathcal{Y}= L^2(\Omega_\mathcal{Y}, \mathbb{R})\), where \(\Omega_X\subset \mathbb{R}^{d_\mathcal{X}}\) and \(\Omega_\mathcal{Y}\subset \mathbb{R}^{d_\mathcal{Y}}\) are compact sets. For BasisONets, the encoder computes inner products with a finite selection of neural networks. We consider a more general setup for the function spaces \(\mathcal{X}\) and \(\mathcal{Y}\). Namely, assume that \(\mathcal{X}= \mathcal{X}(\Omega_\mathcal{X}, \mathbb{R})\) is a separable Hilbert space and \(\mathcal{Y}=\mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) is a normed space.

To obtain universal approximation theorems for BasisONets as in statement , one can simply interpret them as encoder-decoder architectures \(\hat{G} = D\circ \varphi \circ E\) and apply 14 as follows:

• Choose dense encoders on \(\mathcal{X}(\Omega_\mathcal{X}, \mathbb{R})\) from 9 corresponding to a dense set \(S_\mathcal{X}\subset \mathcal{X}\) of neural networks. If \(\mathcal{X}=L^2(\Omega_\mathcal{X}, \mathbb{R})\) or \(\mathcal{X}\) being a Sobolev Hilbert space, \(S_\mathcal{X}\) can, for example, be chosen as a set of ReLU-networks [27];

• As in Section 5.1.1, choose \(\varphi\) from a universal function approximator \(\mathcal{F}\) which consists of neural networks;

• As in Section 5.1.1, choose dense decoders on \(\mathcal{Y}\) from 8, that correspond to a dense set \(S_\mathcal{Y}\subset \mathcal{Y}(\Omega_\mathcal{Y}, \mathbb{R})\) of neural networks.

In particular, 14 implies and extends [11], in which the weaker statement was shown for BasisONets and the special case of \(\mathcal{X}=L^2(\Omega_\mathcal{X}, \mathbb{R})\) and \(\mathcal{Y}=L^2(\Omega_\mathcal{Y}, \mathbb{R})\). Note that in [11], BasisONets are defined with \(u_j\) and \(\psi_i\) being so-called neural bases. That is, they approximate finitely many elements of an orthonormal basis. This is analogous to the construction of dense encoders and decoders in the proof of 9.

5.1.4 Frame Architectures↩︎

In this section, we make use of the frame encoders and decoders introduced in Section 3.1.2 to define a class of encoder–decoder architectures based on frames. Special cases of these architectures were studied in [14] and [13], where Riesz bases and orthonormal bases were used in place of frames, respectively. Throughout this section, let \(\mathcal{X}\) and \(\mathcal{Y}\) be infinite-dimensional separable Hilbert spaces. The finite-dimensional case can be treated analogously.

To obtain a universal approximation theorem for frame architectures as in statement , we can interpret them as encoder-decoder architectures \(\hat{G} = D\circ \varphi \circ E\) as follows and apply 14:

• Choose frame encoders on \(\mathcal{X}\) and frame decoders on \(\mathcal{Y}\) from 6;

• As in Section 5.1.1, choose \(\varphi\) from a universal function approximator \(\mathcal{F}\) which consists of neural networks.

In particular, 14 implies [14]. More precisely, both results establish the stronger approximation statement , whereas 14 applies in the more general setting of frames rather than being restricted to Riesz bases. Furthermore, 14 yields an approximation theorem for Deep-H-ONets [13], whose encoders and decoders are induced by orthonormal bases. Since orthonormal bases are a particular class of Riesz bases, this result is likewise covered by [14].

5.2 Applications in optimal transport↩︎

We have seen in Sections 4 and 3.2.1 that encoder-decoder architectures can be used to approximate continuous operators \(\mathcal{X}\to \mathcal{Y}\), where \(\mathcal{X}\) or \(\mathcal{Y}\) are \(p\)-Wasserstein-spaces \((\mathcal{P}_p(\Omega), W_p)\). In this section, we present some applications which may benefit from our developed theory.

5.2.1 Surrogates for (entropic) optimal transport maps↩︎

In this section, we discuss some operators arising in linear optimal transport that fit into our approximation framework. The concept of linear optimal transport has been proposed in [72], and the basic idea is to embed the non-linear space \((\mathcal{P}_p(\Omega), W_p)\) into the Hilbert space \(L^2(\Omega, \mathbb{R}^d; \rho)\) to benefit from the latter’s structure. This is typically done by fixing a reference measure \(\rho\) and then identifying a measure \(\mu\) with the corresponding optimal transport map \(T_{\rho, \mu}\in L^2(\Omega, \mathbb{R}^d; \rho)\) from \(\rho\) to \(\mu\). We observe below that these embedding operators \(G_\rho: \mu \mapsto T_{\rho, \mu}\) can be approximated by encoder-decoder architectures, according to 14.

Therefore, it will be interesting to investigate in future work whether encoder-decoder surrogates for \(G_\rho\) can be beneficial in methods that require solving many optimal transport problems, that is, when \(G_\rho\) often needs to be evaluated at different \(\mu\). Since repeatedly solving optimal transport problems can become computationally demanding, the encoder-decoder surrogates may reduce the overall computational cost. For example, one could investigate replacing the classical solver for estimating the optimal transport maps within optimal transport neural operators (OTNOs) [20]. OTNOs are a recent operator learning approach for learning PDE solutions on varying domains by using optimal transport maps to move between complex geometries \((\mu)\) and a fixed reference domain \((\rho)\). On the reference domain, a Fourier Neural Operator is then applied.

Let us state the following version of Brenier’s theorem [73].

Therefore, the optimal transport map \(T_{\rho, \mu} \mathrel{\vcenter{:}}= \nabla \varphi_{\rho, \mu}\) is uniquely determined up to a \(\rho\)-null set. \(T_{\rho, \mu}\) is also called Brenier map [61], [74], [75] or Monge map [76] and is the unique solution of the Monge transport problem. Note that the assumption on \(\rho\) being absolutely continuous with respect to the Lebesgue measure can be relaxed, see e.g. [65]. It has been noted in [74], [75] that \(G_\rho:\mu\mapsto T_{\rho, \mu}\) is continuous, which we state in the theorem below. A detailed proof can be found in [77].

Note that for every \(\mu, \nu\in \mathcal{P}_2(\mathbb{R}^d)\), it holds that \(W_2(\mu, \nu)\leq \Vert G_\rho(\mu)-G_\rho(\nu)\Vert_{L^2}\) [74], [75]. This implies that \(G_\rho\) is injective and its inverse on \(G_\rho(\mathcal{P}_2(\mathbb{R}^d))\) is Lipschitz continuous. Therefore, \(G_\rho\) is an embedding of \(\mathcal{P}_2(\mathbb{R}^d)\) into \(L^2(\mathbb{R}^d, \mathbb{R}^d; \rho)\). In linear optimal transport, the embedding \(\tilde{G}_\rho: \mu\mapsto (G_\rho(\mu) - \mathrm{id})\) is considered, see for example [72]. Since the operators \(G_\rho\) and \(\tilde{G}_\rho\) are continuous, they can be approximated by encoder-decoder architectures according to 14 as follows:

• Since \((\mathcal{P}_2(\mathbb{R}^d), W_2)\) has the input-EDAP, choose encoders as in 10;

• As in Section 5.1.1, choose \(\varphi\) from a universal function approximator \(\mathcal{F}\) which consists of, for example, neural networks;

• Since \(L^2(\mathbb{R}^d, \mathbb{R}^d; \rho)\) is a separable Hilbert space, choose frame decoders from 6, or dense decoders from 8 corresponding to a dense set \(S\subset L^2(\mathbb{R}^d, \mathbb{R}^d; \rho)\) of, for example, neural networks.

5.2.2 Geodesic operator networks↩︎

Gracyk et al. introduced geodesic operator networks (GeONets) in [41] to learn the operator which maps a pair of absolutely continuous probability measures \(\mu_0, \mu_1\in\mathcal{P}_2^{ac}(\Omega)\) to the corresponding Wasserstein geodesic \(\mu:[0,1]\to \mathcal{P}_2^{ac}(\Omega)\), where \(\Omega\subseteq \mathbb{R}^m\) is equipped with the Euclidean norm \(\vert \cdot \vert\). To the best of our knowledge, it has not been shown yet that GeONets can indeed approximate the geodesic operator \((\mu_0, \mu_1)\mapsto \mu\) with respect to a suitable topology. We briefly outline how the theory developed in this work could provide a route towards such an approximation result.

In [41], optimality conditions were derived for the Wasserstein geodesic problem of finding \(\mu\) from \(\mu_0, \mu_1\). GeONets then seek to learn solutions \(\rho, u:[0,1]\times \Omega\to \mathbb{R}\) of these conditions, which read as follows: \[\begin{align} \begin{cases} \partial_t \rho + \mathrm{div}(\rho\nabla u) = 0 &\mathrm{(continuity equation)}; \\ \partial_t u+ \frac{1}{2}\Vert \nabla u \Vert^2_2 = 0 &\mathrm{(Hamilton-Jacobi equation)};\\ \rho(\cdot, 0) = \rho_0, \rho(\cdot, 1) = \rho_1, \end{cases} \end{align}\] where \(\rho_0, \rho_1\) are the densities of \(\mu_0, \mu_1\), respectively. GeONets consist of two encoder-decoder architectures that approximate the solution operators \[\begin{align} G_{CE}: \mathcal{P}_2^{ac}(\Omega)\times \mathcal{P}_2^{ac}(\Omega) &\longrightarrow \mathcal{Y}_{CE}([0,1]\times \Omega, \mathbb{R})\\ (\mu_0, \mu_1)&\longmapsto \rho, \\ G_{HJ}: \mathcal{P}_2^{ac}(\Omega)\times \mathcal{P}_2^{ac}(\Omega) &\longrightarrow \mathcal{Y}_{HJ}([0,1]\times \Omega, \mathbb{R})\\ (\mu_0, \mu_1)&\longmapsto u, \end{align}\] corresponding to the continuity and Hamilton-Jacobi equation, respectively. \(\mathcal{Y}_{CE}\) and \(\mathcal{Y}_{HJ}\) denote function spaces for the respective PDE solution functions. \(G_{CE}(\mu_0, \mu_1) = \rho\) is then the density representation of the desired geodesic \(\mu\).

Both GeONet encoder-decoder architectures are closely related to the architectures covered by 17, which are generalizations of the low-rank MIONets from 30. Indeed, the GeONet architectures are both of the form \[\begin{align} \sum_{i=1}^{p} \varphi_{i}^{(1)}\big(E^{(1)}(\mu_0)\big) \varphi_{i}^{(2)}\big(E^{(2)}(\mu_1)\big) \psi_{i}, \end{align}\] where \(\varphi^{(1)}, \varphi^{(2)}\) and \(\psi_i:[0,1]\times \Omega\to \mathbb{R}\) are neural networks. The pair of GeONet encoders \(E^{(1)}, E^{(2)}\) evaluate the densities \(\rho_0, \rho_1\) of \(\mu_0, \mu_1\) at finitely many points \(x_k\in \Omega\). This is similar to the encoders from 22 and 5, which compute \(\mu_0(I_k)=\int_{I_k} \rho_0(x) d\lambda(x)\) for finitely many disjoint sets \(I_k\subset \Omega\).

Some things remain to be verified in order to get an approximation result by 17 for the approximation of the solution operators \(G_{CE}\) and \(G_{HJ}\) by the two GeONet encoder-decoder architectures.

• Show that the solutions of the PDEs lay within Banach spaces \(\mathcal{Y}_{CE}, \mathcal{Y}_{HJ}\) of functions. Further, check whether there exist dense subsets \(S_{CE}\subset \mathcal{Y}_{CE}\) and \(S_{HJ}\subset \mathcal{Y}_{HJ}\) of neural networks. These can then be used in 17.

• Show that both solution operators \(G_{CE}\) and \(G_{HJ}\) are continuous with respect to the \(2\)-Wasserstein distance. For each operator, then use two encoders from 5 in 17.

6 Conclusion↩︎

We studied the approximation theory of encoder-decoder architectures in the topology of uniform convergence on compact sets. In this setting, two notions of universal approximation naturally arise; see statements and in 2. The first requires that, for every continuous operator and compact subset of its input space, there exists a sequence of approximators converging uniformly on that compact set. The second requires the existence of a single sequence that converges uniformly on every compact subset to the operator. While statement is commonly considered in the operator-learning literature, we established in a new 14 for diverse classes of encoder-decoder architectures and choices for input and output spaces. We proved in 3 that this type of universal operator approximation is a topologically stronger statement in typical operator learning settings, namely whenever \(\mathcal{X}\) is an infinite-dimensional normed space (or more generally whenever \(\mathcal{X}\) is not hemicompact). A further advantage of over is that it may facilitate the derivation of approximation theorems in Bochner spaces. Indeed, uniform convergence on every compact subset implies pointwise convergence, allowing one to invoke the dominated convergence theorem. This suggests a possible route for extending the universal Bochner approximation theorem [33] for classical DeepONets to broader classes of encoder-decoder architectures and more general input and output spaces. We leave this direction for future research.

Moreover, we introduced the EDAPs and CEDAPs, sufficient properties of the underlying normed or metric input and output spaces, that enable universal operator approximation by encoder-decoder architectures as in statements and ; see [thm:density_metric,thm:mionet,thm:sequential_density_normed]. These properties are fulfilled by many spaces considered within operator learning frameworks, for example, normed function spaces such as Lebesgue spaces, Sobolev spaces, spaces of continuously differentiable functions, or - more generally - normed spaces having the (bounded) approximation property. Beyond normed spaces, the EDAPs and CEDAPs are sufficiently general to include also metric spaces. For instance, we showed that Fréchet spaces having Schauder bases as well as \(p\)-Wasserstein spaces of probability measures can be considered as input or output spaces. Further, Skorohod spaces of càdlàg functions are suitable input spaces in our operator approximation framework. These examples not only demonstrate the flexibility of the (C)EDAPs regarding suitable input and output spaces, but also regarding possible choices of encoders and decoders. For example, they allow nonlinear encoders and decoders, discontinuous encoders and non-Lipschitz decoders, thereby relaxing assumptions that are often imposed in the literature. To the best of our knowledge, the existing operator learning theory for statement handles operators between separable Hilbert spaces using Riesz basis encoders and decoders. Our 14 hence significantly extends the existing theory for encoder-decoder architectures regarding possible input and output spaces, as well as possible encoder and decoder constructions. Regarding the theory related to statement , we also make some contributions. For example, some of our naturally constructed encoders for Wasserstein and Skorohod spaces are currently not covered by the existing theory for statement (A), which is due to their discontinuity.

Rather than proving separate approximation theorems for individual architectures, our approximation theorems apply to many well-known encoder-decoder architectures at once, for example, classical DeepONets, MIONets, BasisONets, Deep-H-ONets, or Riesz bases architectures. We pointed out that our framework not only unifies existing universal approximation theorems for these models, but also extends them by allowing more general input and output spaces and greater flexibility in the choice of encoders and decoders.

We conclude by highlighting four directions for future research. First, it would be interesting to inspect which other metric spaces have the (C)EDAPs. Second, if the EDAPs or CEDAPs were refined to incorporate quantitative approximation rates for the identity operator - for instance, rates depending on covering numbers of compact sets - it would be natural to investigate whether analogous rates can be transferred to the approximation of operators satisfying additional regularity assumptions, such as Lipschitz continuity or Fréchet differentiability. Third, for different topologies than uniform convergence on compact sets, such as Bochner spaces \(L^p(\mathcal{X}, \mathcal{Y}; \mu)\), it would be interesting to explore analogues of the (C)EDAPs for \(\mathcal{X}\) and \(\mathcal{Y}\) that directly lead to approximation capabilities of encoder-decoder architectures in the respective topology. Last but not least, the discussion in Section 5.2 indicates that our framework may also be useful for future developments in optimal transport, including the approximation-theoretic analysis of GeONets and related architectures.

7 Supplement topology↩︎

7.1 Proof of Lemma 1↩︎

In the following, we state the auxiliary results from basic topology, which are needed for the proof of 1. The first lemma can, for example, be found in [42].

7.2 Further Results↩︎

It is well-known that compact metric spaces are separable, see for example [42]. Below we give a simple proof based on 18.

8 Notes on sampling↩︎

8.1 BAP of \(\mathcal{C}^{k}\)-spaces↩︎

Consider the space \(\mathcal{C}^k(\Omega, \mathbb{R})\) equipped with the \(\mathcal{C}^k\)-norm. For simplicity, we only consider \(k=1\), \(\Omega=[0,1]\), but it might be possible to derive similar results in more general settings.

8.2 Schauder basis coefficients and sampling↩︎

Below, we provide a comparison between the sampling encoders \(E_n\) from Section 3.1.3 with the basis encoders \(\tilde{E}_n\) introduced in Section 3.1.1. Since both of them can be used in encoder-decoder architectures for approximating operators, see 14, it is natural to ask whether sampling encoders can coincide with basis encoders for every \(n\in \mathbb{N}\). Since the sampling encoders \(E_n\) and the basis encoders \(\tilde{E}_n\) have different output dimensions, it only makes sense to compare \(E_n\) with \(\tilde{E}_{k(n)}\), where \(k(n)\) is the number of sampling points used for the definition of \(E_n\). We have not found a complete answer to the above question. However, under certain conditions on the sampling points, the following theorem reveals that \(E_n\) cannot coincide with \(\tilde{E}_{k(n)}\) for sufficiently large \(n\in \mathbb{N}\).

In the theorem above, the sampling points for \(E_{n+1}\) were a refinement of the sampling points for \(E_n\). If, more generally, individual sampling points \(\{y_1^{(n)}, ..., y_{k(n)}^{(n)}\}\) are chosen for each \(n\in \mathbb{N}\), it remains an open question whether the previous theorem still holds. Let us close the discussion about 21 with a final remark on the assumption of \(k(n)\) to be unbounded. If \(\{y_1^{(n)}, ..., y_{k(n)}^{(n)}\}\) is an \(\frac{1}{n}\)-covering of a set \(\Omega\subseteq \mathbb{K}^d\) having a non-zero Lebesgue measure \(\mathrm{Vol}(\Omega)>0,\) the unboundedness of \(k(n)\) follows from the translation-invariance and \(\sigma\)-subadditivity of the Lebesgue measure: If \(k(n)\) had an upper bound \(s<\infty\), it would follow that \[\begin{align} \mathrm{Vol}(\Omega) \leq \sum_{i=1}^{k(n)} \mathrm{Vol}\left(B_{\frac{1}{n}}(y_i^{(n)})\right) \leq s \mathrm{Vol}\left(B_{\frac{1}{n}}(y_1^{(1)})\right)\xrightarrow{n\to\infty}0, \end{align}\] which would be a contradiction to \(\Omega\) having positive measure.

9 The Skorohod space \(\mathcal{D}([0, \infty))\)↩︎

For the definition of a metric \(d_\infty\) inducing the Skorohod topology on \(\mathcal{D}([0,\infty))\), we follow [64]. Beforehand, for \(R>0\) we define the mapping \[\begin{align} \psi_R: \mathcal{D}([0,\infty)) &\longrightarrow \mathcal{D}([0, R])\\ \psi_R f(t) &=\begin{cases} f(t)\phantom{0(m-t)} \mathrm{ if } t\leq R-1,\\ f(t)(R-t)\phantom{0} \mathrm{ if } R-1<t\leq R. \end{cases} \end{align}\] Note that for any \(f\in \mathcal{D}([0, \infty))\) we can write \(\psi_Rf(t) = \varphi_R(t)f(t)\) with the mapping \[\begin{align} \varphi_R:[0, R] &\longrightarrow[0, 1]\\ t&\longmapsto \begin{cases} 1 \mathrm{ if } t\leq R-1,\\ R-t \mathrm{ else.} \end{cases} \end{align}\]

In order to show that \(\mathcal{D}([0, \infty))\) equipped with \(d_\infty\) has the input-EDAP, we slightly adapt 25. Below, we define a mapping \(A^{(\infty)}_{\sigma, n}\), which also has a canonical decomposition in terms of an encoder and a decoder. To simplify the notation, we define the clipping function \(\mathrm{clip}:\mathbb{R}\times [0, \infty) \to \mathbb{R}\) by \[\begin{align} \mathrm{clip}(x, n)\longmapsto\begin{cases} n \mathrm{ \, if \, } x\geq n,\\ -n \mathrm{ \, if \, } x\leq -n,\\ x,\mathrm{ \, else.} \end{cases} \end{align}\]

Next, we prove an analogous result to Lemma 15 for the space \(\mathcal{D}([0, \infty))\). Again, this result will be the key for approximating the identity by \(A^{(\infty)}_{\sigma, n}\) uniformly on compact sets, see Corollary 11.

The following characterization of relatively compact sets in \(\mathcal{D}([0,\infty))\) can be found in [64].

Finally, we obtain the desired result.

10 Approximation by MIONets↩︎

In this section, we provide a proof of 17, which applies to low-rank MIONets from 30, when basis encoders are used for the input spaces \(\mathcal{X}_i\), and \(S\) is a dense subset of \(\mathcal{Y}\) that consists of neural networks. Note that it would be sufficient to assume that \(\mathcal{X}_1\) and \(\mathcal{X}_2\) have the input-CEDAP, respectively. For a simpler notation, we assume them to have the input-EDAP.

Beforehand, we introduce some notation for the tensor products of vector spaces, where we follow [81]. The tensor product between vector spaces \(\mathcal{Z}_1\) and \(\mathcal{Z}_2\) is denoted by \(\mathcal{Z}_1\otimes\mathcal{Z}_2\). Note that every \(u\in \mathcal{Z}_1\otimes\mathcal{Z}_2\) has a representation \[\begin{align} u = \sum_{i=1}^p v_i \otimes w_i, \end{align}\] where \(v_i\otimes w_i\) denotes the tensor product of some \(v_i\in \mathcal{Z}_1\) and \(w_i\in \mathcal{Z}_2.\)

11 Further useful results↩︎

Acknowledgements↩︎

Janek Gödeke acknowledges funding by the Deutsches Zentrum für Luft- und Raumfahrt (grant no. 50 EE 2204). Further, Janek Gödeke and Pascal Fernsel acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation, Project Number 281474342/GRK2224/2).

We thank Peter Maaß from the University of Bremen and Maarten de Hoop from the Rice University for fruitful discussions about operator learning and approximation. We also thank Nihat Ay for his invitation to TU Hamburg for discussions, and Paweł Przybyłowicz from AGH University of Krakow for his question whether the EDAP-theory is applicable to Skorohod spaces. Further, we thank Hendrik Vogt from the University of Bremen for our discussions and his contribution to 20.

References↩︎