1 Introduction↩︎

Nonparametric regression is one of the most extensively studied problems in past decades due to its remarkable flexibility in modeling the relationship between an input \(X\) and output \(Y\). While numerous algorithms have been developed, the strong guarantees of learnability and generalization rely on the fact that there are a sufficient number of training samples and that the future data possess the same distribution as the training. However, training sample scarcity in the target domain of interest and distribution shifts occur frequently in practical applications and deteriorate the effectiveness of most existing algorithms both empirically and theoretically. Transfer learning has emerged as an appealing and promising paradigm for addressing these challenges by leveraging samples or pre-trained models from similar, yet not identical, source domains.

In this work, we study the problem of transfer learning in the presence of the concept shifts for nonparametric regression over some specific reproducing kernel Hilbert spaces (RKHS). Specifically, we posit there are limited labeled samples from the target domain but sufficient labeled samples from a similar source domain where the concept shifted, namely, the conditional distribution of \(Y|X\) changes across domains, which implies the underlying regression function shifts. Hypothesis Transfer Learning (HTL) [1] is the most prevalent framework to tackle concept shift in a two-phase manner by leveraging pre-trained models (hypotheses) from source domains followed by utilizing labeled target samples to learn the shifting, i.e., fine-tune the pre-trained models. HTL algorithms have been extensively studied across various model settings and fine-tuning approaches with theoretical guarantees. Examples include additive linear forms in [2]–[4] and general transformations in [5], [6], among others. Inspired by this framework, a line of research has focused on developing transfer learning algorithms with minimax optimality guarantees. These pursuits facilitate the design and understanding of algorithms from the perspective of fully exploiting source knowledge; see [7]–[12], and more reference therein.

Previous works have primarily focused on transfer learning procedures for model estimation, providing upper bounds or even optimal convergence rates to evaluate the generalization performance given that the model is well-specified, i.e., assuming the specified models in the estimation match the true regression function. In contrast, less attention has been paid to achieving provable and optimal guarantees under misspecified cases despite its prevalence in practical applications. For instance, a considerable body of literature for nonparametric regression assumes the true source and target regression functions reside in certain RKHSs. They derive (nearly) optimal convergence rates of excess risk by assuming the true functions’ regularity is a priori known and the employed RKHSs that contain the estimators are correctly specified in the estimation. [5], [13], [14]. Consequently, such algorithms and analyses are neither robust against misspecification nor adaptive, as the information about the true regression functions is typically unknown in practical applications.

The procedure of HTL typically involves first learning the pre-trained models over the source domain and some intermediate models sequentially and then combining them via some transformations to obtain estimators for the target domains; see Section 2.1. Therefore, it appears promising to exploit some existing well-known robustness results from classical nonparametric methods within the RKHS framework, e.g., [15]–[17], to develop robust transfer procedures with optimality guarantees. These existing robustness results demonstrate that the optimal convergence rate is still attainable even when estimators are learned within misspecified RKHSs. However, such robustness typically depends on the qualification of the learning algorithm (a quantity measuring the algorithm’s fitting capability; see definition in Section 2.2). If the relative regularity of the estimator to the true function deviates significantly from the qualification, these robustness results will no longer hold due to the well-known saturation effect [18], a phenomenon that the convergence rate of the algorithms fails to attain the information-theoretic lower bound. Thus, it is necessary to develop robustness not only against model misspecification but also to overcome the challenges posed by the saturation effect.

Aiming at the gap between theory and practice in the current analysis of HTL, we state the main question of this paper:

In this work, we answer this question. We first give a new result on the robustness of misspecification for a widely used class of nonparametric methods, termed spectral algorithms, which overcomes saturation effects. We then utilize this result to achieve the main goal. To our knowledge, this work presents the first attempt to address HTL under concept shift that simultaneously fulfills the following two desirable properties: (1) robustness to misspecification of specified RKHSs; and (2) adaptively achieve minimax optimal convergence rates. Our main contributions can be summarized as follows.

1. Our first result proves that for true regression functions in fractional Sobolev spaces of order \(m\), employing fixed bandwidth Gaussian kernels in arbitrary single-task misspecified spectral algorithms, which will be defined in Section 2.2, attains minimax optimal convergence rates. Additionally, an estimator obtained via standard training and validation achieves the same minimax optimality up to a logarithmic factor. Notably, the optimal order of the regularization parameter \(\lambda\) for achieving non-adaptive rates should decay exponentially, i.e., \(\lambda \asymp \exp\{ -Cn^{\frac{2}{2m+d}} \}\) for a constant \(C\) and sample size \(n\). This robustness result is of independent interest and serves as a complement to classical misspecified kernel methods research.

2. Methodologically, to answer the main question, we propose a special HTL procedure by imposing spectral algorithms with Gaussian kernels as the learning algorithms in each phase of the HTL. The design of this HTL algorithm leverages the novel robustness against model misspecification exhibited by the Gaussian kernel and is thus free from saturation effects, enabling its capability to consistently and adaptively achieve rate-optimal transfer learning.

3. Theoretically, we establish the minimax lower bound for the learning problem under concept shifts in terms of excess risk and demonstrate that this special HTL algorithm attains a minimax optimal convergence rate (up to logarithmic factors). The asymptotic optimal rate we obtain reveals that the factor influencing the transfer learning efficiency in HTL is the relative signal strength between true intermediate and source regression functions, where such factors are largely overlooked in existing HTL analyses.

The rest of the paper is organized as follows. In Section 2, we formulate the learning problem, including the concept shifts in nonparametric regression, the HTL framework, and model assumptions. We also briefly introduce the spectral algorithms. In Section 3, we establish both the non-adaptive and adaptive convergence rates for spectral algorithms with fixed bandwidth Gaussian kernels. In Section 4, we present the modified special HTL algorithm for learning the true function with concept shifts, followed by the derivation of minimax optimality analysis and discussions. In Section 5, we conduct simulations to confirm the derived asymptotic convergence rates are optimal. Section 6 provides a detailed review of related works in the context of misspecified kernel methods and transfer learning. Finally, Section 7 concludes the paper and provides discussions.

2 Problem Formulations↩︎

In this section, we set up the problem to be studied in this work. In Subsection 2.1, we formulate the learning problem of nonparametric regression with concept shifts and state the hypothesis transfer learning framework along with the model assumptions. In Subsection 2.2, we introduce the spectral algorithms in classical single-task settings, which serve as the backbone of the learning framework.

2.1 Concept Shifts, Learning Framework and Assumptions↩︎

2.1.0.1 Concept Shifts

Let a compact set \(\mathcal{X}\subset \mathbb{R}^{d}\) be the input space with Lipschitz boundary, and \(\mathcal{Y}\subseteq \mathbb{R}\) be the output space. Suppose there are two unknown probability measures on \(\mathcal{X}\times \mathcal{Y}\), the measure \(P\) for the source domain and \(Q\) for the target domain. One observes \(n_{P}\) i.i.d. labeled source samples \(\mathcal{D}^{P} = \{ (x_{i}^{P},y_{i}^{P}) \}_{i=1}^{n_{P}}\) drawn from distribution \(P\), and \(n_{Q}\) i.i.d. labeled target samples \(\mathcal{D}^{Q} = \{ (x_{i}^{Q},y_{i}^{Q}) \}_{i=1}^{n_{Q}}\) drawn from distribution \(Q\), with \[y_{i}^{P} = f^{P}( x_{i}^{P} ) + \epsilon_{i}^{P} \quad \text{and} \quad y_{i}^{Q} = f^{Q}( x_{i}^{Q} ) + \epsilon_{i}^{Q},\] where \(f^{P}\) and \(f^{Q}\) are the underlying true regression functions, and \(\epsilon_{i}^{P}\) and \(\epsilon_{i}^{Q}\) are i.i.d. random noise with zero mean.

In this work, we consider the concept shifts setting, which is alternatively known as model shifts [19], [20] or posterior drift [9], [21] in different contexts. Specifically, let \(P_{X}\) and \(Q_{X}\) denote the marginal distributions. The concept shift assumes \(P_{X}\) and \(Q_{X}\) are identical while \(P_{Y|X}\) and \(Q_{Y|X}\), i.e., \(f^{P}\) and \(f^{Q}\), are different across the domains. Given the observed samples from both domains, the goal is to construct an estimator \(\hat{f}^{Q}\) that minimizes the excess risk over the target distribution \(Q\), i.e., \(\mathop{\mathrm{E}}_{X\sim Q_{X}}[ ( \hat{f}^{Q}(X) - f^{Q}(X) )^{2} ]\).

2.1.0.2 Learning Framework

In this work, the learning framework we consider is a slight generalization of the hypothesis transfer learning (HTL) procedure adopted from [5], which are both practically and theoretically prevalent for addressing concept shifts.

The motivation of Algorithm 1 is to decompose the objective of learning \(f^{Q}\) into learning \(f^{P}\) and \(f^{\delta}\) separately. Here, the source function \(f^{P}\) is less regularized (“more complex”) while the intermediate function \(f^{\delta}\), which represents the shift, is highly regularized (“simple”, e.g., linear functions). Since \(f^{Q}\) is challenging to learn directly with limited target samples, Algorithm 1 facilitates the learning efficiency by learning the “hard” part, \(f^{P}\), effectively using sufficient large source samples and also learning the “simple” part, \(f^{\delta}\), well with limited target samples.

The procedure in Algorithm 1 generalizes many widely used transfer learning algorithms by selecting different \(g\) and \(G\). For example, taking \(g(Y^{Q}, f^{P}(X)) = Y^{Q} - f^{P}(X)\) and \(G(f^{\delta}(X), f^{P}(X)) = f^{P}(X) + f^{\delta}(X)\) represents the offset transfer learning, where the target function is in the additive combination of source and intermediate function. This recovers numerous transfer learning algorithms originated from biased regularization [2], which modify the pre-trained model via additive linear adjustments based on regularized ERM, such as those by [1], [3], [10], [12], to name a few. Additionally, it is shown that in [22], by setting \(g(Y^{Q}, f^{P}(X)) = (Y^{Q} - \tau f^{P}(X))/(1-\tau)\) and \(G(f^{\delta}(X), f^{P}(X)) = (1-\rho)f^{P}(X) + \rho f^{\delta}(X)\) with two varying hyperparameters \(\tau\) and \(\rho\), Algorithm 1 becomes the transfer learning algorithms such as posterior distribution ratio estimation [23] and neural network-based pre-training and fine-tuning [24] given \(\tau\) and \(\rho\) belongs to specific regimes. To establish the theoretical analysis under this learning framework, we impose the following assumption on \(g\) and \(G\), which is standard in most existing HTL literature, such as [1], [5].

The Lipshictz continuity condition guarantees \(\hat{f}^{Q}\) converges to \(f^{Q}\) if both \(\hat{f}^{P}\) and \(\hat{f}^{\delta}\) converge their counterparts. The invertibility condition assures the injection between the values of the target function \(f^{Q}\) and the intermediate function \(f^{\delta}\). Finally, the consistency condition assures that step 3 produces an unbiased estimator for \(f^{\delta}\).

2.1.0.3 Model Assumptions

In this work, we investigate the setting where \(f^{P}\), \(f^{Q}\) and \(f^{\delta}\) reside in some specific RKHSs. Let \(K\) be a symmetric, positive-definite, and continuous kernel function over \(\mathcal{X}\times \mathcal{X}\) that satisfies \(\sup_{x\in \mathcal{X}}K(x,x)\leq \kappa^2\) and denote its associate RKHS as \(\mathcal{H}_{K}(\mathcal{X})\). We consider the stationary kernel, i.e., the value of \(K(x,y)\) only depends on the difference \(x-y\), and write \(K(x-y):= K(x,y)\). For such kernels, one can characterize the RKHS of \(K\) in terms of Fourier transforms when \(\mathcal{X}= \mathbb{R}^{d}\). Let \(\mathcal{F}(f)(\omega)\) denote the Fourier transform of a function \(f\), then by Theorem 10.12 of [25], \[\mathcal{H}_{K}(\mathbb{R}^{d}) = \left\{ f \in L^{2}(\mathbb{R}^{d}) \cap C(\mathbb{R}^{d}) : \frac{\mathcal{F}(f)}{\sqrt{\mathcal{F}(K)}} \in L^{2}(\mathbb{R}^{d}) \right\}.\] When \(\mathcal{X}\) is a subset of \(\mathbb{R}^{d}\), such a definition still captures the regularity of functions in \(\mathcal{H}_{K}(\mathcal{X})\) via a norm equivalency result that holds by the extension theorem [26]. Denote \(\mathcal{W}^{m,p}(\mathcal{X})\) as the Sobolev space with order \(m\). When \(p=2\), the Sobolev space is equivalent to the RKHS of a stationary kernel. The following lemma from [25] describes the norm equivalency.

A prominent class of stationary kernels that satisfy (?? ) is the Matérn kernel [27], which is defined by \[K_{\nu}(x,y ;h) = \frac{2^{1-\nu}}{\Gamma(\nu) }\left( \sqrt{2\nu} \frac{\|x-y\|_2}{h}\right)^\nu K_\nu\left( \sqrt{2\nu} \frac{\|x-y\|_2}{h}\right),\] where \(h\) is the bandwidth and \(K_{\nu}\) is the modified Bessel function of the second kind. The Sobolev space \(H^{m}(\mathcal{X})\) associated with \(K_{\nu}\) satisfied \(m = \nu + d/2\). We let \(H^{m}(\mathcal{X},R)\) represent a ball in the the Sobolev space with radius \(R\), i.e. \(H^{m}(\mathcal{X},R) = \{f : f\in H^{m}(\mathcal{X}), \|f\|_{H^{m}(\mathcal{X})} \leq R \}\). For notational simplification, we abbrivate \(H^{m}(\mathcal{X})\) as \(H^{m}\) and \(H^{m}(\mathcal{X},R)\) as \(H^{m}(R)\) unless otherwise specified. The following assumptions will be made throughout the paper.

Assumption 2 indicates the eigenvalue decay rate (EDR) of the reproducing kernel of these Sobolev spaces is polynomial, which is a standard assumption in nonparametric literature and allows one to obtain the minimax optimal convergence rate directly via classical results [28]. While this assumption does not assert the relationship between \(m_{P}\), \(m_{Q}\) and \(m_{\delta}\), the effectiveness of Algorithm 1 relies on \(m_{\delta}\) being larger, meaning \(f^{\delta}\) is more regularized than \(f^{P}\) and \(f^{Q}\) and thus intuitively aligns with the philosophy of Algorithm 1. For example, when Algorithm 1 takes the offset transfer learning form, it is often assumed \(m_{\delta} \geq m_{P}\) and \(m_{P} = m_{Q}\) [29]. Such “simpler” offset assumptions have also been proven to be effective in other models, e.g., high-dimensional linear works [4], [10], [12] where authors assume the offset coefficients should be sparser than source and target coefficients.

Assumption 3 is also a standard assumption that frequently appears in the kernel method literature [16], [30] to establish theoretical results, which controls the noise tail-probability decay speed.

2.2 Spectral Algorithms↩︎

We now introduce a class of kernel methods for nonparametric regression known as spectral algorithms, which serve as \(\mathcal{A}^{P}\) and \(\mathcal{A}^{\delta}\) in Algorithm 1 in this work. To facilitate simplicity, we introduce the spectral algorithms in a single-task learning setting with the target domain samples \(\mathcal{D}^{Q}\) while an extension to source domain samples is straightforward.

Let \(\mathcal{H}_{K}\) be an RKHS with kernel \(K\). Denote the \(L^{2}\)-space as \(L^{2}(\mathcal{X}, Q_{X})\) (in short \(L^{2}\)). We define the corresponding integral operator \(T_{K}:L^{2} \rightarrow L^{2}\) as \[T_{K}(f)(x^{\prime}) = \int_{\mathcal{X}} K(x,x^{\prime}) f(x) dQ_{X}(x),\] which is a compact operator since it is positive, self-adjoint, and trace-class. Applying Mercer’s theorem yields that there is an at most countable index set \(N\) such that the decomposition \(K(x,x^{\prime}) = \sum_{j\in N}s_{j} e_{j}(x) e_{j}(x^{\prime})\) holds, where \(\{s_{j}\}_{j\in N}\) are the eigenvalues in non-increasing order and \(\{e_{j}\}_{j\in N}\) are the corresponding eigenfunctions, which form an orthonormal basis of \(L^{2}\). For \(x\in \mathcal{X}\), define the evaluation operator as \(K_{x}: \mathcal{H}_{K} \rightarrow \mathbb{R}, f \mapsto \langle f, K_{x} \rangle_{\mathcal{H}_{K}}\) and its adjoint operator \(K_{x}^{*}: \mathbb{R}\rightarrow \mathcal{H}_{K}, y \mapsto yK_{x}\). We define the target sample covariance operator \(T_{K,n}^{Q}:\mathcal{H}_{K}\rightarrow \mathcal{H}_{K}\) and the target sample bias function \(g_{n}^{Q}:\mathbb{R}^{n} \rightarrow \mathcal{H}_{K}\) as \[T_{K,n}^{Q} := \frac{1}{n_{P}}\sum_{i=1}^{n_{Q}}K_{x_{i}^{Q}}^{*}K_{x_{i}^{Q}}\quad \text{and} \quad g_{n}^{Q}:=\frac{1}{n_{Q}} \sum_{i=1}^{n} K_{x_{i}^{Q}}^{*}y_{i}^{Q}.\]

In the classical single-task non-parametric regression problem, the goal is to obtain \(\hat{f}^{Q}\) from \(\mathcal{H}_{K}\) based on the target samples \(\mathcal{D}^{Q}\) to achieve small risk in \(L^{2}\). Thus, it is natural to obtain \(\hat{f}^{Q}\) by minimizing the empirical squared loss over \(\mathcal{H}_{K}\), i.e. \[\label{eqn:32empirical32loss32minimization32problem} \hat{f}^{Q} = \underset{f\in \mathcal{H}_{K}}{\operatorname{argmin}} \frac{1}{n} \sum_{i=1}^{n_{Q}} \left( f(x_{i}^{Q}) - y_{i}^{Q} \right)^2,\tag{1}\] which gives \(\hat{f}^{Q}\) as the solution of the empirical, linear equation, \(T_{K,n}^{Q}f = g_{n}^{Q}\). However, solving this equation is an ill-posed inverse problem as the inverse of the sample covariance operator, \(T_{K,n}^{Q}\), in general, does not exist. In inverse problems and statistical learning, a common approach to address this issue is to replace the inverse of \(T_{K,n}^{Q}\) with a regularized operator. This replacement corresponds to selecting a specific filter function, which leads to the formulation of spectral algorithms [18], [31], [32].

The motivation for introducing the class of filter functions is that \(\phi_{\lambda}\) is analogous to the function \(\phi(t) = t^{-1}\) but with better behavior around \(0\), e.g. \(\phi_{\lambda}\) is bounded by \(\lambda^{-1}\). We now define the corresponding spectral algorithms given a specific filter function \(\phi_{\lambda}\).

Here, the filter function \(\phi_{\lambda}\) acts on the eigenvalues of the self-adjoint finite-rank sample covariance operator \(T_{K,n}\). Different filter functions lead to different types of regularization. We list some of the commonly used spectral algorithms in the following examples. For more examples of spectral algorithms, we refer readers to [33].

1. Kernel Ridge Regression (KRR): The choice of Tikhonov filter function \(\phi_{\lambda}(z) = (z + \lambda)^{-1}\) corresponds to kernel ridge regression. In this case, \(\tau= E = F_{\tau}=1\).

2. Gradient Descent (GD) and Gradient Flow (GF): The choice of \(\phi_{\lambda}(z) = \eta \sum_{k=1}^{t-1} (1 - \eta z)^{k}\) where \(\lambda = (\eta t)^{-1}\)

   corresponds to gradient descent where \(\eta > 0\) is a constant step size. Gradient descent with more complex update rules can be expressed in terms of filter functions as well [34], [35]. For infinitely small step size, the choice of \(\phi_{\lambda}(z) = (1 - \exp\{-\frac{z}{\lambda}\}) / z\) corresponds to gradient flow. In both cases, \(\tau\) could be any positive number, \(E=1\), and \(F_{\tau}=(\tau/e)^{\tau}\).

3. Kernel Principal Component Regression (KPCR): The choice of spectral cut-off function \(\phi_{\lambda}(z) = z^{-1} \mathbb{1}_{z\geq \lambda}\) corresponds to Kernel Principal Component Regression, which is motivated by using finite components to recover \(T_{K,n}^{Q}\). In such case, \(\tau\) could be any positive number, \(\tau= E = F_{\tau}=1\).

3 Spectral Algorithms with Gaussian Kernels↩︎

Observing that Algorithm 1 obtains the \(\hat{f}^{P}\) and \(\hat{f}^{\delta}\) through two sequential and independent spectral algorithms, \(\mathcal{A}^{P}\) and \(\mathcal{A}^{\delta}\), we first consider the robustness and adaptivity for spectral algorithms under the classical learning setting (i.e., recovering \(f^{Q}\) with \(\mathcal{D}^{Q}\) solely), and then apply the results to Algorithm 1 to answer the main question of this work. Specifically, this section begins with a warm-up example to demonstrate why existing misspecified results are insufficient and highlights the motivation for using Gaussian kernels. We then present the non-adaptive and adaptive minimax optimal convergence rate for spectral algorithms with Gaussian kernels, followed by detailed discussions.

3.1 Warm-up: Misspecification with Optimal Rate and Saturation Effect↩︎

We first review some existing robustness results on misspecified spectral algorithms and the saturation effect.

When \(m_{Q}' = m_{Q}\), the misspecified result becomes well-specified and can be found in a line of past work [36], [37]. The misspecified result comes from a combination (with a modification) of Theorem 15 and 16 in [17] and Theorem 1 in [30]. The saturation effect is proved by Corollary 3.2 in [38]. Given the fact that the minimax optimal rate is \(n^{-2m_{Q}/(2m_{Q}+d)}\) when \(f^{Q}\in H^{m_{Q}}\), Proposition 1 indicates that even when the imposed RKHS \(\mathcal{H}_{K}\) is misspecified (and thus the estimator \(\hat{f}^{Q}\)), the minimax optimal rate is still attainable when \(m_{Q}' \geq m_{Q}/2\tau\) with appropriately chosen \(\lambda\). However, if \(m_{Q}' < m_{Q}/2\tau\), i.e., the true function is much smoother than the estimator itself, the saturation effect occurs, i.e., there is a persistent gap between the lower bound of the estimator and the information-theoretic lower bound of the learning problem.

The misspecified result in Proposition 1 suggests a potential solution to the main question of this work: imposing appropriately misspecified kernels, such as Matérn kernels, in \(\mathcal{A}^{P}\) and \(\mathcal{A}^{\delta}\) to attain robustness against model misspecification while achieving optimal rate. Nonetheless, such a solution still faces two problems: (1) Non-adaptivity, i.e., one still needs to know the true smoothness when tuning \(\lambda\); (2) Saturation risk, i.e., using spectral algorithms with finite qualifications \(\tau\) but a less smooth kernel is imposed. Although the former can be potentially addressed by some data-driven adaptive approaches, the second one is more problematic, as choosing a less smooth kernel may permanently prevent achieving the optimal rate. While saturation effects can be mitigated by selecting specific spectral algorithms with arbitrarily large qualifications, this restricts the flexibility in choosing \(\mathcal{A}^{P}\) and \(\mathcal{A}^{\delta}\), eliminating the potential benefits that can be gained from customizing these general algorithms under different application scenarios.

Therefore, there is a clear need for a misspecified kernel that can be robust against both model misspecification and the potential saturation effect across arbitrary spectral algorithms. Formally, for \(f^{Q} \in H^{m_{Q}}\) with any \(m_{Q} > d/2\), imposing such a kernel in the spectral algorithms would ensure that, regardless of the spectral algorithm’s qualification, there always exists an optimal \(\lambda\) such that the minimax optimal convergence rate is achievable.

3.2 Opimal Convergence Rate with Gaussian Kernels↩︎

We start with the motivation for considering the Gaussian kernels. As shown in Proposition 1, when the imposed RKHS’s smoothness \(m_{Q}'\) is greater than the quantity \(m_{Q}/2\tau\), there always exists an optimal \(\lambda\) such that the minimax optimal rate is attainable. This indicates that the most prominent issue that causes the saturation effect is the large relative smoothness of \(f^{Q}\) to \(\hat{f}^{Q}\), i.e., estimating a smooth function with a much less smooth estimator. Thus, a feasible strategy is always to employ a Matérn kernel with large smoothness so that the estimator \(\hat{f}^{Q}\) is obtained from a smoother Sobolev space than the one where \(f^{Q}\) resides. However, precisely determining the appropriate smoothness to impose on \(\hat{f}^{Q}\) is challenging since the true smoothness \(m_{Q}\) is typically unknown. To this end, we consider using the fixed bandwidth Gaussian kernel, which is further motivated by the following fact.

• The RKHS associated with the isotropic Matérn kernel \(K_{\nu}\) [27] is isomorphic to the Sobolev space \(H^{\nu + \frac{d}{2}}\). Moreover, the Gaussian kernel is the limited of the Matérn kernel, i.e. \[\lim_{\nu \rightarrow\infty} K_{\nu}(x; h) = \exp \left( - \frac{\|x-y\|_{2}^{2}}{2h^2} \right), \quad x,y \in \mathbb{R}^{d},\] and the RKHS associated with the Gaussian kernel is contained in the Sobolev space \(H^{m_{Q}}\) for any \(m_{Q} > d/2\) [39].

Thus, in using the Gaussian kernel, one might expect to achieve the minimax optimal rate with a feasible \(\lambda\) and overcome the saturation effect since its corresponding RKHS consists of functions that are consistently “smoother” than \(f^{Q}\).

Our first result proves that there exists an optimal order of \(\lambda\) such that the spectral algorithms with Gaussian kernels attain the optimal convergence rate.

Despite the minimax optimality of the spectral algorithm demonstrated in Theorem 1, a key practical challenge is that choosing optimal \(\lambda\) still requires knowing \(m_{Q}\). To address this, we developed an adaptive rate when the smoothness parameter \(m_{Q}\) is unknown a priori. The adaptive procedure is based on a standard training and validation approach [41]. Specifically, we construct a finite set that is a finite arithmetic sequence, i.e., \(\mathcal{M}= \{m_{\min}< \cdots < m_{\max}\}\) where \(\{m_{i}\}\) satisfy \(m_{\min}>d/2\), \(m_{\max}\) large enough such that \(m_{Q}\leq m_{\max}\) and \(m_{i} - m_{i-1}\asymp 1/\log n_{Q}\). We then split dataset \(\mathcal{D}^{Q}\) into \(\mathcal{D}_{1}^{Q} := \{(x_{1}^{Q},y_{1}^{Q}),\cdots,(x_{j}^{Q},y_{j}^{Q})\}\) and \(\mathcal{D}_{2}^{Q} = \mathcal{D}^{Q} \backslash \mathcal{D}_{1}^{Q}\). The adaptive estimator is obtained by following the training and validation approach.

1. (Training): For each \(m \in \mathcal{M}\), compute the non-adaptive estimator \(\hat{f}_{\lambda_{m}}^{Q} = \mathcal{A}_{K,\lambda_{m}}(\mathcal{D}_{1}^{Q})\) with the spectral algorithm \(\mathcal{A}_{K,\lambda_{m}}\) and dataset \(\mathcal{D}_{1}^{Q}\). The regularization parameter \(\lambda_{m}\) is chosen as \(\lambda = \exp\{ -Cn^{2/2m + d} \}\) for some constant \(C\) based on Theorem 1.

2. (Validation): Selecting the \(\hat{f}_{\lambda_{m}}^{Q}\) that minimizes empirical \(L^{2}\) error over \(\mathcal{D}_{2}^{Q}\) as the adaptive estimator \(\hat{f}_{\hat{\lambda}}^{Q}\).

The following theorem shows the adaptive estimator from the aforementioned training and validation approach attains optimal rate up to a logarithmic factor in \(n_{Q}\).

3.3 Discussion↩︎

Theorem 1 shows that the excess risk of spectral algorithms with fixed bandwidth Gaussian kernels can attain the minimax optimal convergence rate without restriction on the qualification. From another perspective, it offers a robust solution to avoid the potential saturation effect in spectral algorithms with low qualifications.

It is interesting to note that even though the motivation for imposing Gaussian kernels is that the Gaussian kernel is the limit of Matérn kernel \(K_{m_{Q}'}\) as \(m_{Q}'\rightarrow\infty\), the misspecified case in Proposition 1 can never substitute our result. Specifically, setting the smoothness parameter \(m_{Q}'\) as infinity in Proposition 1, the polynomial decay form of \(\lambda\) never makes the optimal order of \(\lambda\) trackable since \(\lim_{m_{Q}' \rightarrow \infty} n_{Q}^{-2m_{Q}'/(2m_{Q}+d) } = 0\), and only indicates that the optimal \(\lambda\) should converge to \(0\) much faster than the polynomial form. In contrast, our result explicitly reveals that the optimal order of \(\lambda\) should decay exponentially.

This exponential decay form of the optimal \(\lambda\) originates from controlling the approximation error. The misspecified results in Proposition 1 rely on the standard real interpolation technique to control the approximation error. It allows one to expand the intermediate term \(f_{\lambda}^{Q}\) (see definition in Appendix 8) and the true function \(f^{Q}\) under the same basis, making the approximation error scale as \(\lambda^{m_{Q}/m_{Q}'}\|f^{Q}\|_{H^{m_{Q}}}\) and the optimal \(\lambda\) decays polynomially like Proposition 1. This technique is widely used in many misspecified kernel literature. However, in our case, the intermediate term lies in the RKHS of Gaussian kernels, making expanding \(f_{\lambda}^{Q}\) and \(f^{Q}\) under the same basis no longer feasible. The technique we used to prove the Theorem 1 is the Fourier transform of the Gaussian kernel, which allows us to bound the approximation error by \(log(1/\lambda)^{-m_{Q}} \| f^{Q} \|_{H^{m_{Q}}}^{2}\) and consequently reveals this exponential decay order of \(\lambda\). We refer readers to Appendix 8 for more details.

Lastly, to further highlight our findings, we summarize some state-of-the-art results (with slight modifications to align with our setting) that consider general, Matérn, or Gaussian misspecified kernels for the spectral algorithms (or their special cases) and highlight our result in Table 1. We refer readers to Section 6 for a detailed literature review.

The pattern of the \(\lambda\) in this work differs significantly from previous attempts that justify the minimax optimality of the misspecified fixed bandwidth Matérn kernel or variable bandwidth Gaussian kernel in statements. Especially in the regime of variable bandwidth, given both \(\gamma\) and \(\lambda\) decay polynomially in \(n_{Q}\), [43] showed that the convergence rate could be arbitrarily close to the optimal rate, while [44] attained the optimal rate (up to a logarithmic factor) under the so-called DIM condition. However, we would like to note that these variable Gaussian results are not directly comparable to ours due to the difference in model settings and assumptions. For example, these two works consider regularized ERM problems, allowing for loss functions beyond the square ones. They also assumed data distributions are bounded, whereas we considered a moment-based error assumption.

4 Robust and Adaptive Hypothesis Transfer Learning↩︎

In this section, we first present a special version of Algorithm 1 based on the results in Section 3. Then, we theoretically analyze the minimax optimality of this special HTL algorithm in Subsection 4.2 and discuss its insights in Subsection 4.3.

4.1 Methodology↩︎

A key observation of Algorithm 1 is that the estimators \(\hat{f}^{P}\) and \(\hat{f}^{\delta}\) are learned separately on two datasets \(\mathcal{D}^{P}\) and \(\mathcal{D}^{\delta}\) using the spectral algorithms \(\mathcal{A}^{P}\) and \(\mathcal{A}^{\delta}\), respectively. Consequently, achieving robust and adaptive HTL with rate-optimal guarantees reduces to achieving the same goal in learning \(f^{P}\) and \(f^{\delta}\) individually. Recall that in a single dataset learning case, Theorem 2 shows that the desired results can be obtained by imposing Gaussian kernels. Building on this observation, we introduce Robust and Adaptive Hypothesis Transfer Learning (RAHTL) estimator, defined via Algorithm 1 as follows: \[\hat{f}^{Q} = G\left ( \hat{f}_{\hat{\lambda}_{\delta}}^{\delta}, \hat{f}_{\hat{\lambda}_{P}}^{P} \right).\] Here, \(\hat{f}_{\hat{\lambda}_{P}}^{P} = \mathcal{A}_{K,\hat{\lambda}_{P}}^{P}(\mathcal{D}^{P})\) and \(\hat{f}_{\hat{\lambda}_{\delta}}^{\delta} = \mathcal{A}_{K,\hat{\lambda}_{\delta}}^{\delta}(\mathcal{D}^{\delta})\) are adaptive estimators obtained using selected spectral algorithms with Gaussian kernels of fixed bandwidth \(K\). \(\mathcal{D}^{P}\) is the source dataset and \(\mathcal{D}^{\delta}\) is the intermediate dataset generated by applying data transformation function \(g\) to \(\mathcal{D}^{p}\) and the pre-trained estimator \(\hat{f}_{\hat{\lambda}_{P}}^{P}\).

4.2 Optimal Convergence Rate Analysis↩︎

Based on Assumption 2, the parameter space for the learning problem is defined as follows, \[\Theta(R_{P},R_{\delta},m_{P},m_{\delta}) = \left\{ (P, Q): \left\|f^{P}\right\|_{H^{m_{P}}}\leq R_{P}, \left\|f^{\delta}\right\|_{H^{m_{\delta}}} \leq R_{\delta} \right\}.\] Here, \(R_{\delta}\) mathematically quantifies the similarity between the source and target functions as a small \(R_{\delta}\) indicates the \(f^{P}\) and \(f^{Q}\) are more similar. We note that such an upper bound for the distance between parameters or models from both domains is often necessary to establish rigorous optimality in many concept shift settings. Examples include \(\ell^{1}\) or \(\ell^{0}\) distance in high-dimensional setting [10], [12], Fisher-Rao distance in low-dimensional setting [45], RKHS distance in functional setting [14], etc.

Our first result specifies the minimax lower bound, which elucidates the information-theoretic difficulty of the transfer learning problem.

The next theorem provides the upper bound of the excess risk for the RAHTL estimator and shows that it is minimax optimal up to logarithmic factors.

Theorem 4 indicates that the convergence rate of the estimator \(\hat{f}^{Q}\) consists of two components: the first term is pre-training error which represents the error of learning \(f^{P}\) with the source samples, while the second term is the fine-tuning error, which arises from learning the intermediate function \(f^{\delta}\) with the constructed samples.

4.3 Discussion↩︎

We now discuss the minimax optimal rate in Subsection 4.2 to provide insights to better understand the transfer efficiency. Specifically, we explore how the source samples improve the learning efficiency over the target domain and identify what factors contribute to higher efficiency. These insights are derived by comparing the convergence rates between target-only learning ?? and transfer learning ?? . However, without explicitly assuming the relationship between \(m_{Q}\), \(m_{P}\), and \(m_{\delta}\) in Assumption 2, it is hard to compare these rates directly. In the following proposition, we show that the target smoothness \(m_{Q}\) can be linked to \(m_{P}\) and \(m_{\delta}\) given the model transformation function \(G\) belongs to certain smooth function classes.

In existing research, if the model transformation function \(G\) takes some certain explicit form, like \(G(x_{1},x_{2}) = x_{1}+x_{2}\) in offset transfer learning, then Theorem 2 indicates \(m_{\delta}\geq m_{P} = m_{Q}\) since \(G \in C^{\infty}\), recovering the conditions in [29]. However, when \(G\) lacks an explicit form, it is unavoidable to require direct assumptions such as \(m_{\delta} \geq m_{Q}\) if one would like to compare these rates [5].

We first discuss the learning efficiency. When supervised learning is performed solely on the target sample \(\mathcal{D}^{Q}\), the convergence rate of the target-only learning adaptive estimator is \((n_{Q}/\log n_{Q})^{-\frac{2m_{Q}}{2m_{Q}+d}}\), as established in Theorem 2. Theorem 4 indicates the benefit gained from the knowledge transfer depends jointly on the sample size in source domain \(n_{P}\) and the factor \(\xi\). The quantity \(\xi\) plays a pivotal role in the comparison, whose magnitude, in general, reflects the degree of similarity between \(f^{P}\) and \(f^{Q}\). We define a phase transition points \(\xi^{*}\) as \[\xi^{*}:=\frac{(n_{Q}/\log n_{Q})^{\frac{2m_{\delta}}{2m_{\delta}+d}}}{(n_{P}/\log n_{P})^{\frac{2m_{P}}{2m_{P}+d}}}.\] When \(\xi\) is smaller than \(\xi^{*}\), it makes the pre-training error dominate, and the convergence rate of \(\hat{f}^{Q}\) is faster than the target-only estimator given \(n_{Q} < n_{P}\). For less similar functions, i.e., a large \(\xi\), the fine-tuning error dominates, but the upper bound is still sharper than the target-only estimator since \(m_{Q} \leq m_{\delta}\).

Next, we discuss the factor \(\xi\) as its magnitude controls the phase transition. To the best of our knowledge, the form of \(\xi^{2}:= R_{\delta}^{2} / R_{P}^{2}\) is the first form that links the signal strength from intermediate and source functions to explain the transition in the HTL framework. Previous works, such as [3], [5], [13], overlooked the existence of \(\xi\) in the upper bound and thus were unable to provide a concrete explanation of how model discrepancy affects the phase transition. Although some recent works introduced using the magnitude of intermediate models \(R_{\delta}\) to measure similarity in the same way as we do, such as [10], [12] and more references therein, these works only identified \(\xi \propto R_{\delta}^2\), i.e., in our context \[\mathcal{O}_{\mathbb{P}} \left( \left( \frac{n_{P}}{\log n_{P}}\right)^{-\frac{2m_{P}}{2m_{P}+d}} + \left( \frac{n_{Q}}{\log n_{Q}}\right)^{-\frac{2m_{\delta}}{2m_{\delta}+d}} R_{\delta}^{2} \right),\] and asserted that a smaller \(R_{\delta}\) leads to higher efficiency. However, such an interpretation is not precise as a small \(R_{\delta}\) does not necessarily indicate the high similarity even given simple model transformation functions \(G\). To illustrate, let us consider a simple but concrete example to illustrate how our form of \(\xi\) provides a more accurate interpretation of the transfer efficiency.

The example discussed above highlights that both the relative signal strength of the target and source functions and the geometric angle between them play significant roles in determining the efficiency of offset transfer learning. This form encompasses most transfer learning algorithms that rely on biased regularization, e.g., [1], [2], [10]. The explanation becomes more complex for general model transformation functions \(G\) due to the absence of an explicit form of \(G\), making it challenging to analyze the precise factors that influence the efficiency. In such cases, we can only conclude that the efficiency depends on the signal strength ratio of \(f^{\delta}\) to \(f^{P}\), which still refines the results in previous literature that ignored or misjudged \(\xi\).

5 Numerical Illustration↩︎

This section provides several numerical simulations to validate our theoretical findings and insights in both target-only learning and hypothesis transfer learning cases.

5.1 Simulation for Spectral Algorithms↩︎

We first aim to confirm the minimax optimality of the non-adaptive and adaptive rates established in Theorems 1 and 2. Specifically, we conduct the numerical experiments in classical nonparametric regression with a single dataset, where we consider \(\mathcal{X}= [0,1]\) and the true function \(f_{0} \in H^{m}(\mathcal{X})\) for different \(m\).

We consider the following data generation procedure: \[y_{i} = f_{0}(x_{i}) + 0.5\epsilon_{i} \qquad i = 1, \cdots, n\] where \(x_{i}\) are i.i.d. samples drawn from the uniform distribution on \(\mathcal{X}\), \(\epsilon_{i}\) are i.i.d. standard Gaussian random variables, and sample size \(n=100k\) with \(k=1,2,\cdots,30\). Since the sample path of Gaussian processes (GP) with specified covariate kernels indeed lies in certain Sobolev spaces [47], we generate true functions \(f_{0}\) by setting them as sample paths that are generated from a Gaussian process with isotropic Matérn kernels \(K_{\nu}\). Based on Corollary 4.15 in [47], we set \(\nu = 2.01\) and \(\nu = 3.01\) to generate \(f_{0}\) in \(H^{2}(\mathcal{X})\) and \(H^{3}(\mathcal{X})\).

We investigate three spectral algorithms, including Kernel ridge regression (KRR), Gradient flow (GF), and Kernel principal component regression (KPCR), to recover the true regression functions. For non-adaptive rates, we set the regularization parameter \(\lambda\) as \(\lambda = \exp\{-Cn^{\frac{2}{2m + 1}}\}\) with a fixed \(C\) to construct the estimators. For adaptive rates, we set the candidate smoothness as \([1,2,\cdots,5]\) and construct the estimators through the training and validation as described in Section 3. For each combination of \(n\) and \(m\), we compute \(\hat{\mathcal{E}_{i}} = \|\hat{f} - f_{0}\|_{L_{2}}^{2}\) by Simpson’s rule with \(5000\) testing points and repeats the experiments \(100\) times. The excess risk is then approximated by \(\hat{\mathcal{E}} = \frac{1}{100}\sum_{i=1}^{100}\hat{\mathcal{E}_{i}}\). To verify the convergence rate of the error is sharp, we regress the \(\log(\hat{\mathcal{E}})\) on \(\log(n)\) and check if the regression coefficient to its theoretical counterpart \(-2m/(2m+1)\).

We test different values of \(C\) in sequence \([0.05,0.1,\cdots, 4]\), and report the optimal curve for the best choice of \(C\) in Figure 3 for non-adaptive and adaptive rates, respectively. It can be seen that the scattered points and the corresponding empirical error decay curves align with the theoretical decay curves in both non-adaptive and adaptive cases. Also, the estimated regression coefficient closely agrees with the theoretical counterparts \(-\frac{2m}{2m+1}\). Additionally, in Figure 4, the results based on different choices of \(C\) are presented, and it shows that the empirical excess risk decay rates still closely align with theoretical ones in both cases.

5.2 Simulation for Hypothesis Transfer↩︎

In this part, we present the simulation to confirm the minimax optimality of the HTL algorithm derived in Theorems 3 and 4, as well as the role of \(\xi\) as we discussed in Subsection 4.3. Specifically, we explore offset transfer learning with the data generation procedure under concept shifts defined as follows, \[\{x_{i}^{Q}\}_{i=1}^{n_{Q}}, \{x_{i}^{P}\}_{i=1}^{n_{P}} \stackrel{i.i.d.}{\sim} U([0,1]), \quad y_{i}^{P} = f^{P}(x_{i}^{P}) + 0.5 \epsilon_{i}, \quad y_{i}^{Q} = y_{i}^{P} + f^{\delta}(x_{i}^{Q}),\] where \(f^{P} \in H^{1}(\mathcal{X})\), \(f^{\delta}\in H^{2}\) or \(H^{3}\), making \(m_{P} = m_{Q} = 1\) and \(m_{\delta} = 2\) or \(3\). Throughout the simulation, we compare the excess risk of the two following learning cases.

• Target-only Regression (“Target-Only”): We use KRR with Gaussian kernels to recover \(f^{Q}\) with only the target dataset \(\mathcal{D}^{Q}\). This serves as the non-transfer baseline.

• Hypothesis Transfer (“Transfer”): The special HTL algorithm introduced in Subsection 4.1. To mimic practical scenarios, the pre-training phase conducts gradient methods, i.e., GF, to recover \(f^{P}\) while the fine-tuning phase uses KRR for \(f^{\delta}\).

5.2.0.1 Transfer Optimality

We first verify the minimax optimality. We set \(n_{P} = n_{Q}^{1.5}\) while \(n_{Q}\) varies and takes the value in \(\{40,45,\cdots,150\}\). We also manipulate the Sobolev norm of \(f^{P}\) and \(f^{\delta}\) such that the factor \(\xi\) belongs to \(\{0.25, 0.5, 0.75, 1\}\). Such settings ensure the convergence rate of excess risk for the HTL algorithm satisfies \(\|\hat{f}^{Q} - f^{Q}\|_{L^{2}}^{2} = \mathcal{O}_{\mathbb{P}}( n_{Q}^{-\frac{3m_{P}}{2m_{P}+1}} + \xi n_{Q}^{-\frac{2m_{\delta}}{2m_{\delta}+1}} ) = \mathcal{O}_{\mathbb{P}}( n_{Q}^{-1} + \xi n_{Q}^{-\frac{2m_{\delta}}{2m_{\delta}+1}} ) = \mathcal{O}_{\mathbb{P}}( n_{Q}^{-\frac{2m_{\delta}}{2m_{\delta}+1}} )\). Therefore, optimality can thus be confirmed by verifying that the empirical excess risk decay rate matches the theoretical rates \(n_{Q}^{-\frac{2m_{\delta}}{2m_{\delta}+1}}\). The excess risks under different settings are presented in Figure 5, where each curve denotes the average error over \(100\) repeated experiments. We can see the empirical excess risks of the HTL algorithm, i.e., blue ones, consistently aligned with the theoretical counterparts, i.e., orange ones, and also outperform the target-only baselines in all settings. Moreover, the excess risks for \(m_{\delta} = 3\) are lower and decay faster than those for \(m_{\delta} = 2\), which aligned with the theoretical indications. These results justify the optimality of the HTL algorithm we derived in Section 4.

5.2.0.2 Influence of \(\xi\) on Transfer Efficiency

As discussed in Subsection 4.3, the magnitude of \(\xi\) affects the transfer efficiency. To confirm such influence of \(\xi\), we fix \(n_{Q} = 200\) while varying \(n_{P}\) from \(200\) to \(1500\) and \(\xi\) takes values in \(\{0.25, 0.5, 1, 4\}\). Under such settings, it is expected that as \(\xi\) increases, the fine-tuning error will start to dominate and remain unchanged, even as \(n_{P}\) increases. The results, presented in Figrue 6, show that the degree of decrease in the excess risks of the HTL algorithm diminishes as \(\xi\) increases. Eventually, these error curves become horizontal; that is, it does not decrease with the increase of \(n_{P}\). These observations confirm the influence of \(\xi\) discussed in Subsection 4.3.

6 Related Work↩︎

6.1 Hypothesis Transfer Learning↩︎

Under plausible distribution shifts relating two domains, the technique of leveraging data from source domains to enhance learnability in the target domain is known as transfer learning or domain adaptation [48], [49]. There is a long list of works on these topics in nonparametric methods and RKHS context, covering scenarios such as covariate shift [8], [50], [51], and concept shift [3], [5], [19], [52], to name a few. Focusing on transfer learning under concept shifts, the methodology of HTL has been theoretically studied over the past decade. [1], [53] studied the shifted offset being a linear model and established generalization bounds through Rademacher complexity. [13] further assumed true source functions in a Sobolev ellipsoid while shifted offsets belong to smoother ones, modeling them with finite basis functions. [5] proposed using general transformations to model domain shift and offer upper bounds on excess risk. More recently, following this transformation framework, [6] derived the optimal model transformation functions under square loss. These works, however, investigated neither the minimax optimality nor the robustness to model misspecification of HTL.

Recently, a line of work has been devoted to studying the minimax optimality for a special case of HTL, offset transfer learning, which typically involves regularizing the difference between target and pre-trained models via different norms. To name a few: [10], [12] considered \(\ell^{1}\)-norm for high-dimensional (generalized) linear regression. [54] considered \(\ell^{2}\)-norm for linear models in feature learning setting. [14] utilized RKHS-norm for functional linear models. Other similar studies for more specific problems include linear regression [45], binary classification [7], [11], [55], Gaussian mixture model [56], graphical model [57] and among others.

For theoretical studies of transfer learning on robustness against model misspecification, most works are concentrated on semiparametric models due to the fact that parametric parts are sensitive to misspecified models and can affect the efficiency of the estimator. For example, [58] utilized model averaging to achieve asymptotic minimax optimality when the target model is misspecified. Another line of works [59]–[62] developed robustness estimation procedures under the covariate shift assumption, allowing for one or multiple specified models are misspecified. For RKHS-based nonparametric regression, [29] achieved optimal HTL using misspecified kernels, but their framework was confined to offset transfer learning with the learning algorithms in both phrases being KRR. For nonparametric classification, multiple works obtained minimax optimal guarantees by using robust models, such as decision trees [44] or nearest neighbors estimators [9], under the assumption that true classification functions reside in certain Hölder spaces. Despite the distinctions in the problem setting and estimation procedures, the messages in these researches are analogous to ours: achieving robustness to model misspecification and adaptivity while ensuring minimax optimality is crucial for transfer learning algorithms, as these properties underpin their performance guarantees.

6.2 Misspecified Spectral Algorithm↩︎

For misspecified spectral algorithms, a line of works [15], [16], [34], [63]–[68] has investigated the cases where the true function belongs to the interpolation space of the RKHS associated with the imposed kernel, i.e., \([\mathcal{H}]^{s}\), and derive the minimax optimal rate for spectral algorithms under different norms. To our knowledge, the state-of-the-art results in [30] recovered the minimax optimal rate for all spectral algorithms under the \([\mathcal{H}]^{\gamma}\)-norm while removing the \(L^{\infty}(\mathcal{X})\) boundedness assumption on the true function \(f_{0}\) in previous literature. [69] further extended the results to the vector-valued spectral algorithms. Most of the aforementioned works typically assumed the eigenvalue decay rate of the imposed kernel is polynomial or the effective dimension bounded by \(\lambda^{-\beta}\). However, eigenvalues of Gaussian kernels decay exponentially fast and do not fit these assumptions. The closest case to our work is Assumption 8 in [70], which assumes a logarithmic decay of the effective dimension. Yet, this is only proven in specific instances, such as the univariate Gaussian kernel \(K(x,y) = xy + \exp\{-8(x-y)^2\}\), which is not the classical multivariate Gaussian kernel \(\exp\{-\|x-y\|_{2}^2/2h^{2}\}\) we consider.

When the imposed kernels are variable bandwidth Gaussian kernels, [43] proved that allowing regularization parameters and the bandwidths to decay polynomially yields convergence rates that can be arbitrarily close to the optimal rate in KRR when \(f_{0}\) belongs to Sobolev or Besov spaces. Later, [44] further improved the convergence rate up to logarithmic factors under the DIM condition.

Regarding the saturation effect, especially for KRR, was observed in practice and reported in [18], [33]. It was only recently that the saturation effect on KRR was theoretically proved by [71] and later extended to spectral algorithms [38]. In other specific settings, like (stochastic) distributed-based, online-based spectral algorithms and average gradient descent, the saturation effect was also noted and overcome by method-specified techniques [35], [72], [73]. Note that this does not weaken the novelty of our results since employing fixed bandwidth Gaussian kernels offers a consistent and robust solution to avoiding the potential saturation effect in arbitrary spectral algorithms with finite qualification.

7 Conclusion and Discussion↩︎

In this work, we address the underexplored problem of achieving robustness against model misspecification in hypothesis transfer learning under the concept shift condition. We first show that in classical nonparametric regression, where true functions belong to Sobolev spaces, specifying the hypothesis space as the RKHS of a fixed bandwidth Gaussian kernel allows spectral algorithms to produce estimators that attain minimax optimal convergence rates, provided the regularization parameter decays exponentially. This provides a stronger, robust result for the misspecified spectral algorithms as it can help to overcome the well-known saturation effect. Leveraging this result, we show that a modified version of a class of hypothesis transfer learning algorithms is robust against model misspecification while still attaining minimax optimality. The derived convergence rates reveal several factors that affect the transfer efficiency, which have been overlooked by previous research, and these findings are empirically validated through numerical simulations.

We aspire this work serves as a starting point for advancing robust transfer learning with statistical guarantees: it encourages the researchers to be aware of the necessity of robustness in learning algorithms under distribution shifts, particularly when leveraging conventional statistical tools or models for which misspecification is not the central research theme.

Moreover, we provide the following potential directions for further improvements. This work considers the setting where the distributions of \(X\) stay the same while conditional distributions of \(Y|X\) can vary. However, in practice, the invariant marginal distribution condition is often unverifiable, and the performance of HTL can degrade significantly when the support or covariate distributions differ substantially between domains. Exploring modifications to the HTL framework that enhance its robustness to covariate shifts while providing theoretical guarantees represents an important avenue for further study. Such advancements would significantly expand the applicability of our results.

8 Proofs for Section 3↩︎

While we introduce and present the results using the target dataset \(\mathcal{D}^{Q}\), for simplicity of notation, we present the proof in the context of classical nonparametric regression setting, i.e. \[y_{i} = f_{0}(x_{i}) + \epsilon_{i}, \quad \forall n = 1,\cdots,n,\] where the true function \(f_{0}\in H^{m_{0}}(\mathcal{X})\), and \(\epsilon_{i}\) satisfies Assumption 3. Note that replacing \(f_{0}\), \(m_{0}\) and the dataset \(\{(x_{i}, y_{i})\}_{i=1}^{n}\) with \(f^{Q}\), \(m_{Q}\) and \(\mathcal{D}^{Q}\) recovers what we presented in Section 3. Besides the notations defined in Section 2, we denote the operator norm of a bounded linear operator as \(\|\cdot\|_{op}\) and denote the effective dimension of \(T_{K}\) as \(\mathcal{N}(\lambda) =tr( (T_{K} + \lambda \mathbf{I})^{-1} T_{K} )\).

8.1 Proof of Theorem 1↩︎

The proof of the non-adaptive rate is mainly consistent with approximation-estimation error decomposition and control of each of them, respectively. We first state the approximation-estimation error decomposition. The estimator of spectral algorithms can be written as \[\hat{f} = \phi_{\lambda}(T_{K,n}) g_{n}.\] Then, we decompose the excess risk into approximation error and estimation error, i.e. \[\underbrace{\left\| \hat{f} - f_{0} \right\|_{L_{2}} }_{\text{excess risk of \hat{f}}} \leq \underbrace{\left\| \hat{f} - f_{\lambda} \right\|_{L_{2}}}_{\text{estimation error}} + \underbrace{\left\| f_{\lambda} - f_{0} \right\|_{L_{2}}}_{\text{approximation error}},\] where the intermediate term \(f_{\lambda}\) is defined as follows, \[f_{\lambda} = \phi_{\lambda}(T_{K}) T_{K}(f_{0})= \phi_{\lambda}(T_{K}) g.\] To prove the upper bound, we also need the following technical assumption.

We now start to control each error separately. The following two theorems are used. We refer their proofs to Appendix 8.3.

8.2 Proof of Theorem 2↩︎

8.3 Proof of Supporting Theorem↩︎

8.4 Supporting Lemmas↩︎

The following lemma provides the concentration inequality about self-adjoint Hilbert-Schmidt operator-valued random variables, which is widely used in related kernel method literature, e.g., Theorem 27 in [16], Lemma 26 in [68] and Lemma 32 in [30].

9 Proofs for Section 4↩︎

9.1 Proof of Theorem 3↩︎

For the lower bound, we proof the alternative but asymptotically equivalent version, i.e. \[\label{eqn:32alternative32version32of32lower32bound} \inf_{\tilde{f}} \sup_{\Theta(R_{P},R_{\delta},m_{P},m_{\delta})} \mathbb{P} \left( \| \tilde{f} - f^{Q} \|_{L_{2}}^2 \geq C \delta R^2 \left( (n_{P} + n_{Q})^{-\frac{2m_{P}}{2m_{P}+d}} + \xi n_{Q}^{-\frac{m_{\delta}}{2m_{\delta}+d}} \right) \right) \geq 1 - \delta\tag{14}\] for some \(C\) that does not depend on \(n_{P}\), \(n_{Q}\), \(R_{P}\), \(R_{\delta}\) and \(\delta\), with \(\xi \propto R_{\delta}^{2}/R_{P}^2\).

9.2 Proof of Theorem 4↩︎

Now, we begin to prove the upper bound for the excess risk of the hypothesis transfer learning. The following Theorem 7 is used, which provides an upper bound for the error induced by using estimator \(\hat{f}^{P}\) to construct intermediate labels.

9.3 Proof of Proposition 2↩︎

The property of composition preserves the Sobolev order and is termed the superposition property for (fractional) Sobolev spaces [74] and has been studied from the 1970s to 1990s under various conditions. However, most of the existing results are built on univariate composition function; see a detailed review in [75]. We thus prove the superposition property for the bivariate composition functions, which take two functions within different Sobolev spaces as input.

9.4 Proof of Supporting Theorems↩︎

9.5 Supporting Lemmas↩︎

The following lemma controls the error induced by using \(\hat{f}^{P}\) under the Tikhonov regularization function.

The following lemma states the minimax lower bound of the excess risk, often referred to as the information-theoretic lower bound, under the single dataset scenario. The proof is standard and can be found in different kernel methods literature; see [30], [65], among others. However, most of the works omit the property of constant \(C\). In order to derive the form of \(\xi\) in Theorem 3, we apply a version from [29], where the authors have shown that \(C\) is proportional to the radius of the Sobolev ball.

References↩︎

---

1. Work does not relate to the author’s position at Amazon.↩︎