1 Introduction↩︎

Recent advancements in deep learning methodology have uncovered a surprising phenomenon that defies conventional statistical wisdom: overfitting can yield remarkably effective generalization [1]–[4]. In the overparametrized regime, complex models that achieve zero-training error, i.e. models that interpolate the training data, have gained popularity [5]–[7]. However, as researchers grapple with increasingly large and heterogeneous datasets, the imperative to effectively integrate diverse sources of information has become crucial. Transfer learning [8] has emerged as a promising approach to address this challenge, allowing one to leverage knowledge from related datasets to boost performance on target problems. Yet, in the context of overparametrization, a crucial question arises: how should one leverage transfer learning in the presence of overfitting? Specifically, should diverse datasets be aggregated to construct a unified interpolator, or should they be handled individually? Intuition suggests that for closely related datasets, a pooled interpolator may yield superior performance compared to those derived from individual datasets. Conversely, for unrelated datasets, the opposite might be true. This paper investigates the interplay between interpolation risk and task relatedness, focusing particularly on overparametrized linear models and min-norm interpolation.

To achieve this, we explore two common ways datasets can differ: covariate shift, where the covariate distribution varies across different datasets, and model shift, where the conditional distribution of the outcome given the covariates differs across datasets. Covariate shift has been extensively studied through concepts like likelihood ratios [9] and transfer exponents [10]–[12], among others. Similarly, a substantial body of literature has studied model shift problems [12]–[17]. Other recent work has also studied overparametrized transfer learning in linear regression, including approaches that regularize toward a source-based estimator [18] or transfer specific parameters learned from a source task [19].

However, the aforementioned literature has primarily considered statistical ideas where explicit penalties are added to loss functions to aid learning in high dimensions or non-parametric approaches. In contrast, modern machine learning has seen growing interest in and success with implicit regularization—the phenomenon whereby algorithms, under suitable initialization and step sizes, converge to predictors that generalize well despite overparameterization. In this context, min-norm interpolators have emerged as an important class of predictors that arise commonly as implicit regularized limits of these algorithms [1], [20]–[29]. However, properties of these interpolators under transfer learning are less well-studied.

A recent work [30] identified conditions on benign overfitting, i.e., data distribution choices such that min-norm interpolators are consistent under covariate shift. Another related work [12] characterized the prediction risk of ridge regression and min-\(\ell_2\)-norm interpolators, while [31] provided an asymptotic analysis of high-dimensional random feature regression under covariate shift. However, these works consider an out-of-distribution (OOD) setting, where no observation from the test distribution is utilized during training.

Transfer learning is often required in settings where along with enormous source data, limited test data is indeed available, cf. [32]. Data from the test distribution can enhance transfer performance. However, min-norm interpolators in this context are not as well-understood. Some recent work has introduced and analyzed approaches that transfer specific parameters from an estimator fitted on source-only data [19] or that interpolate the target data while minimizing distance to a source-only interpolator [33]. However, these works study mechanisms where a source estimator is learned separately and then incorporated with separate target data, and their model shift analyses are restricted to isotropic Gaussian covariates. A closely related line of work studies data-level pooling in high-dimensional transfer learning in the underparametrized regime [34]. However, the analyses in [34] rely on properties of OLS in the underparametrized regime, and the model shift analysis in [34] is similarly restricted to Gaussian covariates. In contrast, the behavior of early fusion mechanisms—where source and target samples are combined at the data level—in the overparametrized regime remains largely unexplored: it is unclear when one should pool source and target samples into a single interpolating estimator and how model and covariate shift affect the risk of such an estimator, especially under more general covariate distributions.

This paper addresses this crucial gap in the literature and characterizes the precise prediction performance of a pooled min-\(\ell_2\)-norm interpolator calculated using both source and target data under overparametrization. Our contributions are summarized as follows:

1. We consider a min-\(\ell_2\)-norm interpolator constructed by combining the source and target data (henceforth referred to as the pooled min-\(\ell_2\)-norm interpolator). We characterize the finite-sample bias and variance of the pooled min-\(\ell_2\)-norm interpolator \(\hat{\boldsymbol{\beta}}\), defined in 3 , in an overparametrized regime. We consider the cases of model shift and covariate shift (Theorems 1 and 7 respectively) and quantify the generalization error of \(\hat{\boldsymbol{\beta}}\). In the presence of model shift, Theorem 1 characterizes the risk of \(\hat{\boldsymbol{\beta}}\) under general anisotropic covariance matrices; special examples are presented in Corollaries 2–6, covering the cases of isotropic designs, spiked matrix models and random effects models. Our main theorem for covariate shift (Theorem 7) operates under an assumption of simultaneous diagonalizability. We discuss possibilities of relaxing this assumption in Section 7 (v) and Appendix  16 .

2. Our general results have a few important takeaways. For instance, our model shift results (Theorem 1) characterize the precise threshold at which the pooled interpolator outperforms the target-only interpolator (Proposition 4). In the special case of isotropic covariance, we further provide a data-driven procedure for estimating this phase transition (Appendix  8 ). In a similar spirit, our findings on covariate shift (Theorem 7) uncover surprising dichotomies. In particular, we observe that increased degrees of covariate shift can in some cases enhance the performance of the pooled min-\(\ell_2\)-norm interpolator (while the opposite happens in other regimes). This property is entirely dependent on the dimensionalities of the observed data (see Proposition 8).

3. We prove a universality result (Theorem 11), establishing that our asymptotic risk characterization of the pooled min-\(\ell_2\)-norm interpolator holds for a broad class of covariate distributions. This demonstrates that the transfer phenomena identified in our Gaussian-based analysis are robust to the choice of covariate distributions. Our result complements related work [23], [35]–[40] on the universality of regression estimators in high-dimensional settings.

4. On the technical front, a significant mathematical contribution of our work is the derivation of an anisotropic law applicable to a broad class of random matrices (see Theorem 20 ). This complements previous anisotropic local laws [34], [41] and may be of independent interest both in random matrix theory as well as in the study of other high-dimensional transfer learning problems, particularly those involving multiple covariance matrices.

   Additionally, we establish how Lindeberg’s replacement method can be combined with Gaussian interpolation to establish universality for resolvent-based functionals of mixtures of sample covariance matrices (Appendix 14 ). Our formulation allows for more precise control of expected derivative terms that arise along the interpolation path, thus permitting us to pass to the ridgeless limit. This complements previous approaches to proving universality [39], [42], [43] while also handling functionals of mixtures of sample covariance matrices, which prior techniques do not readily accommodate.

5. We interpret the pooled min-\(\ell_2\)-norm interpolator as a ridgeless limit of a pooled ridge estimator (see 5 ). As part of our results, we characterize the exact bias and variance of the pooled ridge estimator under the aforementioned settings. This contributes independently to the expanding body of work on the generalization error of high-dimensional ridge regression [6], [44]. Our work complements the out-of-distribution error characterized by recent works [12], [30] where target data is not used in constructing the estimator.

6. Furthermore, we demonstrate that our framework accommodates both (1) settings beyond the well-specified linear model (Theorem 12) and (2) estimators beyond the pooled min-\(\ell_2\)-norm interpolator (Theorem 13). In a misspecified linear model studied in the existing literature [6], [18], we characterize how misspecification alters the risk of the pooled min-\(\ell_2\)-norm interpolator under model shift. We also propose a two-step bias-corrected estimator, based on a held-out subset of target samples, and derive its exact asymptotic risk under model shift.

7. We complement our theoretical findings with extensive simulations that demonstrate the finite-sample efficacy of our theory. Our results exhibit remarkable finite-sample performance under both model shift (Figures 1 and 2) and covariate shift (Figure 3), as well as across varying shift levels, signal-to-noise ratios, and dimensionality ratios. Additional simulations in Appendix 17 further validate the universality of our risk characterization for a range of synthetic and semi-synthetic covariates and illustrate the relative performance of our bias-corrected estimator. Our results complement analogous results for the underparametrized setting in [34] to show that the generalization error of this interpolator exhibits the well-known double-descent phenomenon observed in the machine learning literature [2], [3].

The remainder of the paper is structured as follows: In Section 2, we lay out our framework and model assumptions. Sections 3 and 4 provide detailed analyses of the generalization errors for model shift and covariate shift, respectively. Section 5 includes extensions of our work to a wide range of covariate distributions, to mildly misspecified settings, and to a two-step bias-corrected estimator. A proof outline is provided in Section 6, followed by discussions and future directions in Section 7. Auxiliary results concerning pooled ridge regression are included in Appendix 10 , with detailed proofs of our main results deferred to the remaining appendices.

2 Setup↩︎

In this section, we present our formal mathematical setup.

2.1 Data model↩︎

We consider the canonical transfer learning setting where random samples are observed from two different populations. Formally, we observe two datasets \((\boldsymbol{y}^{(1)}, \boldsymbol{X}^{(1)})\) and \((\boldsymbol{y}^{(2)}, \boldsymbol{X}^{(2)})\)—one serves as the source data and the other the target. Typically, the former has a larger sample size than the latter. Some of our results extend to the general case of multiple source datasets; see Appendix 9 . We assume that the observed data comes from linear models given by \[\label{eqn:model95linear} \boldsymbol{y}^{(k)} = \boldsymbol{X}^{(k)} \boldsymbol{\beta}^{(k)} + \boldsymbol{\epsilon}^{(k)}, k =1,2,\tag{1}\] where \(k=1\) and \(k=2\) correspond to the source and target, respectively. Here, \(\mathbf{y}^{(k)} \in \mathbb{R}^{n_k}\) denotes the response vectors, with \(n_k\) representing the number of samples observed in population \(k\), \(\boldsymbol{X}^{(k)} \in \mathbb{R}^{n_k \times p}\) denotes the observed covariate matrix from population \(k\) with \(p\) the number of features, \(\boldsymbol{\beta}^{(k)} \in \mathbb{R}^p\) are signal vectors independent of \(\boldsymbol{X}^{(k)}\), and \(\boldsymbol{\epsilon}^{(k)} \in \mathbb{R}^{n_k}\) are noise vectors independent of both \(\boldsymbol{X}^{(k)}\) and \(\boldsymbol{\beta}^{(k)}\). We denote \(n = n_1 + n_2\) as the total number of samples.

Throughout the paper, we assume that the design matrices take the form \[\label{eqn:assumption95X} \boldsymbol{X}^{(k)} = \boldsymbol{Z}^{(k)} (\boldsymbol{\Sigma}^{(k)})^{1/2},\tag{2}\] where \(\boldsymbol{\Sigma}^{(k)} \in \mathbb{R}^{p \times p}\) are deterministic positive definite covariate matrices and \(\boldsymbol{Z}^{(k)} \in \mathbb{R}^{n_k \times p}\) are random matrices with independent entries. We impose a common set of regularity assumptions throughout the paper related to regularity of the covariance matrices \(\boldsymbol{\Sigma}^{(k)}\), the noise terms \(\boldsymbol{\epsilon}^{(k)}\), and the aspect ratios \(p / n_1\) and \(p / n_2\). Distributional assumptions on the design matrices \(\boldsymbol{Z}^{(k)}\) are stated separately, since different results require different levels of moment control.

The precise assumptions are as follows.

The following assumption will be used only for the model shift results in Section 3.

As displayed in ?? , we are particularly interested in the overparametrized regime where \(p>n\), partly because it is more prevalent in modern machine learning applications. Besides, the underparametrized regime, i.e. \(p < n\), is better understood in the literature, cf. [34] (they assumed \(n_2 > p\), but their results remain valid for the general case \(n>p\)).

We remark that we do not require any additional assumptions on the signals \(\boldsymbol{\beta}^{(k)}\), except for their independence from \(\boldsymbol{X}^{(k)}\). In fact, our results later show that the convergence of the risks of interest is conditional on \(\boldsymbol{\beta}^{(k)}\), and depends on \(\boldsymbol{\beta}^{(k)}\) through their norms and inner products (i.e., \(\boldsymbol{\beta}^{(k)}\) can also be taken to be arbitrary and deterministic).

2.2 Estimator↩︎

Data fusion, the integration of information from diverse datasets, has been broadly categorized into three approaches in the literature: early, intermediate and late fusion [45], [46]. Early fusion combines information from multiple sources at the input level. Late fusion learns independently from each dataset and combines information at the end. Intermediate fusion achieves a middle ground and attempts to jointly learn information from both datasets in a more sophisticated manner than simply concatenating them. In this work, we consider a form of min-norm interpolation that captures both early and a class of intermediate fusion strategies. To achieve early fusion using min-norm interpolation, one would naturally consider the pooled min-\(\ell_2\)-norm interpolator, defined as

\[\label{eqn:interpolator} \hat{\boldsymbol{\beta}}= \mathop{\mathrm{arg\,min}}\left \{ \|\boldsymbol{b}\|_2 \;\;\text{s.t.}\;\boldsymbol{y}^{(k)} = \boldsymbol{X}^{(k)}\boldsymbol{b},\;k=1,2 \right \}.\tag{3}\] This interpolator admits the following analytical expression:

\[\label{eqn:analytical95pinv} \hat{\boldsymbol{\beta}}= \left(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}\right)^\dagger \left(\boldsymbol{X}^{(1)\top}\boldsymbol{y}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{y}^{(2)}\right),\tag{4}\] where for any matrix \(\boldsymbol{A}\), we denote its pseudoinverse by \(\boldsymbol{A}^\dagger\). This interpolator is alternatively designated as the "ridgeless" estimator because it is the limit of the ridge estimator when the penalty term vanishes (noticed earlier by [6] in the context of a single training dataset):

\[\label{eqn:ridgeless} \begin{align} \hat{\boldsymbol{\beta}} &= \lim_{\lambda\rightarrow 0}\left(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}+n\lambda\boldsymbol{I}\right)^{-1} \left(\boldsymbol{X}^{(1)\top}\boldsymbol{y}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{y}^{(2)}\right)\\ &= \lim_{\lambda \rightarrow 0}\mathop{\mathrm{arg\,min}}_{\boldsymbol{b}} \left\{\frac{1}{n}\|\boldsymbol{y}^{(1)} - \boldsymbol{X}^{(1)}\boldsymbol{b}\|_2^2 + \frac{1}{n}\|\boldsymbol{y}^{(2)} - \boldsymbol{X}^{(2)}\boldsymbol{b}\|_2^2 + \lambda \|\boldsymbol{b}\|_2^2 \right\}. \end{align}\tag{5}\]

Coincidentally, although the pooled min-\(\ell_2\)-norm interpolator is motivated via early fusion, it can also capture a form of intermediate fusion as described below. If we consider a weighted ridge loss with weights \(w_1,w_2\) on the source and target data respectively, then the ridgeless limit of this loss recovers the pooled min-\(\ell_2\)-norm interpolator. This can be seen as follows: for any two weighting coefficients \(w_1,w_2 > 0\), \[\label{eqn:intermediate95fusion} \begin{align} \hat{\boldsymbol{\beta}}&= \mathop{\mathrm{arg\,min}}\left \{ \|\boldsymbol{b}\|_2 \;\;\text{s.t.}\;w_k\boldsymbol{y}^{(k)} = w_k\boldsymbol{X}^{(k)}\boldsymbol{b},\;k=1,2 \right \}\\ &= \left(w_1\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + w_2\boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}\right)^\dagger \left(w_1\boldsymbol{X}^{(1)\top}\boldsymbol{y}^{(1)} + w_2\boldsymbol{X}^{(2)\top}\boldsymbol{y}^{(2)}\right)\\ &= \lim_{\lambda\rightarrow 0}\left(w_1\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + w_2\boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}+n\lambda\boldsymbol{I}\right)^{-1} \left(w_1\boldsymbol{X}^{(1)\top}\boldsymbol{y}^{(1)} + w_2\boldsymbol{X}^{(2)\top}\boldsymbol{y}^{(2)}\right)\\ &= \lim_{\lambda \rightarrow 0}\mathop{\mathrm{arg\,min}}_{\boldsymbol{b}} \left\{\frac{w_1}{n}\|\boldsymbol{y}^{(1)} - \boldsymbol{X}^{(1)}\boldsymbol{b}\|_2^2 + \frac{w_2}{n}\|\boldsymbol{y}^{(2)} - \boldsymbol{X}^{(2)}\boldsymbol{b}\|_2^2 + \lambda \|\boldsymbol{b}\|_2^2 \right\}, \end{align}\tag{6}\] indicating that the interpolator above uniquely stands as the common solution of both the min-\(\ell_2\)-norm interpolator and the ridgeless estimator when two datasets are weighted by \(w_1,w_2\).

Furthermore, the pooled min-\(\ell_2\)-norm interpolator 3 appears naturally as the limit of the gradient descent based linear estimator in the following way: Initialize \(\boldsymbol{\beta}_0=0\) and run gradient descent with iterates given by \[\boldsymbol{\beta}_{t+1}= \boldsymbol{\beta}_t + \eta \sum_{k=1}^2\boldsymbol{X}^{(k)\top}(\boldsymbol{y}^{(k)} - \boldsymbol{X}^{(k)}\boldsymbol{\beta}_{t}), \qquad t=0,1,\ldots,\] with \(0< \eta \le \min\{\lambda_{\max}(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)}), \lambda_{\max}(\boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)})\}\), then \(\lim_{t \rightarrow \infty} \boldsymbol{\beta}_t= \hat{\boldsymbol{\beta}}\).

In recent years, min-norm interpolation-type solutions and their statistical properties have been extensively studied in numerous contexts [1], [3], [4], [7], [23], [47]–[49]. In fact, it is conjectured that this min-norm regularization is responsible for the superior statistical behavior of estimators in the overparametrized regime. In light of these results and the aforementioned abundant formulations, we view our work as a first step of understanding the behavior of this important family of estimators in transfer learning, under the lens of early and intermediate fusion.

We conclude this section by highlighting that computing such an interpolator requires only the summary statistics \(\mathbf{X}^{(k)\top}\mathbf{X}^{(k)}\) and \(\mathbf{X}^{(k)\top}\mathbf{y}^{(k)}\) for \(k=1,2\), rather than entire datasets. This characteristic is particularly beneficial in fields such as medical or genomics research, where individual-level data may be highly sensitive and not permissible for sharing across institutions.

2.3 Risk↩︎

To assess the performance of \(\hat{\boldsymbol{\beta}}\), we focus on its prediction risk at a new (unseen) test point \((\boldsymbol{y}_0, \boldsymbol{x}_0)\) drawn from the target distribution, i.e., \(k=2\). This out-of-sample prediction risk (hereafter simply referred to as risk) is defined as \[\label{eq:define95risk} R(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) := \mathbb{E}\left[(\boldsymbol{x}^{\top}_0\hat{\boldsymbol{\beta}}- \boldsymbol{x}^{\top}_0\boldsymbol{\beta}^{(2)})^2|\boldsymbol{X}\right] = \mathbb{E}\left[\|\hat{\boldsymbol{\beta}}- \boldsymbol{\beta}^{(2)}\|_{\boldsymbol{\Sigma}^{(2)}}^2|\boldsymbol{X}\right],\tag{7}\] where \(\|\boldsymbol{x}\|_{\boldsymbol{\Sigma}}^2:=\boldsymbol{x}^\top\boldsymbol{\Sigma}\boldsymbol{x}\) and \(\boldsymbol{X}:= (\boldsymbol{X}^{(1)\top},\boldsymbol{X}^{(2)\top})^\top\). Note that this risk has a \(\sigma^2\) difference from the mean-squared prediction error for the new data point, which does not affect the relative performance and is therefore omitted. Also, this risk definition takes expectation over both the randomness in the new test point \((\boldsymbol{y}_0,\boldsymbol{x}_0)\) and the randomness in the training noises \((\boldsymbol{\epsilon}^{(1)},\boldsymbol{\epsilon}^{(2)})\). The risk is conditional on the covariates \(\boldsymbol{X}\). Despite this dependence, we will use the notation \(R(\hat{\boldsymbol{\beta}},\boldsymbol{\beta}^{(2)})\) for conciseness since the context is clear. The risk admits a bias-variance decomposition:

\[\label{eq:risk95analytical} R(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) = \underbrace{\|\mathbb{E}(\hat{\boldsymbol{\beta}}|\boldsymbol{X})-\boldsymbol{\beta}^{(2)}\|_{\boldsymbol{\Sigma}^{(2)}}^2}_{B(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})} + \underbrace{\textrm{Tr}[\textrm{Cov}(\hat{\boldsymbol{\beta}}|\boldsymbol{X})\boldsymbol{\Sigma}^{(2)}]}_{V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})}.\tag{8}\] Plugging in 5 , we immediately obtain the following result which we will crucially use for the remainder of the sequel:

In the next two sections, we will lay out our main results which involve precise risk characterizations of the pooled min-\(\ell_2\)-norm interpolator. Bias and variance terms, as decomposed in Lemma 1, will be presented separately.

Throughout the rest of the paper, we denote \(f(p) = O(g(p))\) if there exists a constant \(C\) such that \(|f(p)| \leq Cg(p)\) for large enough \(p\). Moreover, we say an event \(\mathcal{E}\) happens with high probability (w.h.p.) if \(\mathbb{P}(\mathcal{E})\rightarrow 1\) as \(p \rightarrow \infty\).

3 Model Shift↩︎

We first investigate the effect of model shift. More precisely, we assume the covariates \(\boldsymbol{X}^{(1)}\) and \(\boldsymbol{X}^{(2)}\) come from the same distribution, but the underlying signal changes from \(\boldsymbol{\beta}^{(1)}\) to \(\boldsymbol{\beta}^{(2)}\). We denote the shift in the signal parameter by \(\tilde{\boldsymbol{\beta}}:= \boldsymbol{\beta}^{(1)}-\boldsymbol{\beta}^{(2)}\). We will first characterize the precise risk of the pooled min-\(\ell_2\)-norm interpolator and then compare it with that of the min-\(\ell_2\)-norm interpolator using solely the target data.

3.1 Risk under model shift↩︎

Recall the formula for the risk \(R(\hat{\boldsymbol{\beta}}, \boldsymbol{\beta}^{(2)})\) in 8 and its decomposition in Lemma 1. We will split the risk into bias and variance following Lemma 1, and characterize their behavior respectively. Therefore, our contribution focuses on characterizing the asymptotic behavior of \(B_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\), \(B_2(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\), \(B_3(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\), and \(V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) as defined by ?? and ?? .

The risk of the pooled min-\(\ell_2\)-norm interpolator depends on the geometry of the shared covariance matrix \(\boldsymbol{\Sigma}\), the model shift \(\tilde{\boldsymbol{\beta}}\), and the target signal \(\boldsymbol{\beta}^{(2)}\). Letting \(\boldsymbol{\Sigma}= \sum_{i=1}^p s_i \boldsymbol{v}_i \boldsymbol{v}_i^\top\) denote the eigendecomposition of the shared covariance matrix, we encode this geometry via the following signed measures on \(\mathbb{R}_{\geq 0}\) for nonzero \(\boldsymbol{\beta}^{(2)}\) and \(\tilde{\boldsymbol{\beta}}\): \[\begin{gather} \hat{G}_n^{\boldsymbol{\beta}^{(2)}}(s) = \frac{1}{\|\boldsymbol{\beta}^{(2)}\|_2^2} \sum_{i=1}^p \langle \boldsymbol{\beta}^{(2)}, \boldsymbol{v}_i\rangle^2 \boldsymbol{1}_{\{s \geq s_i\}}, \quad \hat{G}_n^{\tilde{\boldsymbol{\beta}}}(s) = \frac{1}{\|\tilde{\boldsymbol{\beta}}\|_2^2} \sum_{i=1}^p \langle \tilde{\boldsymbol{\beta}}, \boldsymbol{v}_i\rangle^2 \boldsymbol{1}_{\{s \geq s_i\}} \\ \hat{G}_n^{(b)}(s) = \frac{1}{\|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2}\sum_{i=1}^p \langle \boldsymbol{\beta}^{(2)}, \boldsymbol{v}_i\rangle \langle \tilde{\boldsymbol{\beta}}, \boldsymbol{v}_i\rangle \boldsymbol{1}_{\{s \geq s_i\}}, \quad \hat{H}_n(s) = \frac{1}{p} \sum_{i=1}^p \boldsymbol{1}_{\{s \geq s_i\}}. \end{gather}\] When \(\boldsymbol{\beta}^{(2)} = 0\) or \(\tilde{\boldsymbol{\beta}}= 0\), we define the corresponding signed measures to be the Dirac measure on \(\{0\}\).

These measures encode how the spectrum of the covariance matrix \(\boldsymbol{\Sigma}\) interacts with the target signal \(\boldsymbol{\beta}^{(2)}\) and the model shift \(\tilde{\boldsymbol{\beta}}\). While the empirical spectral distribution \(\hat{H}_n\) simply records the eigenvalue distribution of the shared covariance \(\boldsymbol{\Sigma}\), the probability measures \(\hat{G}_n^{\boldsymbol{\beta}^{(2)}}\) and \(\hat{G}_n^{\tilde{\boldsymbol{\beta}}}\) weight this spectrum by the alignment between each eigenvector and the corresponding signal \(\boldsymbol{\beta}^{(2)}\) or shift \(\tilde{\boldsymbol{\beta}}\). The signed measure \(\hat{G}_n^{(b)}\) then captures the directional alignment between \(\boldsymbol{\beta}^{(2)}\) and \(\tilde{\boldsymbol{\beta}}\) with respect to the eigenbasis \(\boldsymbol{v}_1, \dots, \boldsymbol{v}_p\). Thus, these four measures translate the spectral geometry of \(\boldsymbol{\Sigma}\), \(\tilde{\boldsymbol{\beta}}\), and \(\boldsymbol{\beta}^{(2)}\) into inputs for the following asymptotic risk quantities:

Our main result for model shift can then be stated in terms of these quantities.

We remark that Theorem 1 holds true for finite values of \(n\) and \(p\). It characterizes the bias and variance of the pooled min-\(\ell_2\)-norm interpolator for the overparametrized regime \(p>n\). The variance \(V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) is decreasing in \(p/n\), but the bias \(B(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) is not necessarily monotone. We provide extensive numerical examples of \(R(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) in Section 3 showing that the risk function still exhibits the well-known double-descent phenomenon. The main challenge in establishing Theorem 1 lies in the analysis of \(B_2(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) and \(B_3(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\). More details and explanations are provided in Section 6. The theorem allows us to answer the following questions:

1. When does the pooled min-\(\ell_2\)-norm interpolator outperform the target-only min-\(\ell_2\)-norm interpolator?

2. What is the optimal target sample size that minimizes the generalization error of the pooled min-\(\ell_2\)-norm interpolator?

We study these questions for three example problems: isotropic designs (Section 3.2), spiked covariance designs (Section 3.3), and random effects models (Section 3.4). In each case, we first establish results under the assumption that \((\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) are both Gaussian random matrices but note that our universality result (Theorem 11) implies that the same results hold for very general classes of covariates.

3.2 Risk under isotropic designs↩︎

In this section, we consider the special case where \(\boldsymbol{\Sigma}^{(1)} = \boldsymbol{\Sigma}^{(2)} = \boldsymbol{I}\). Here, the following corollary of Theorem 1 holds:

In fact, it shows that, in the isotropic setting, the bias can be decomposed into two simple terms ?? and ?? , which depend only on the \(\ell_2\)-norm of \(\boldsymbol{\beta}^{(2)}\) and \(\tilde{\boldsymbol{\beta}}\) respectively. Therefore, under isotropic designs, the generalization error is independent of the angle between \(\boldsymbol{\beta}^{(1)}\) and \(\boldsymbol{\beta}^{(2)}\).

Now, we compare the risk of \(\hat{\boldsymbol{\beta}}\), the pooled min-\(\ell_2\)-norm interpolator, with that of the interpolator using only the target dataset, defined as follows:

\[\label{eqn:interpolator95single} \hat{\boldsymbol{\beta}}^{(2)} = \mathop{\mathrm{arg\,min}}\left\{ \|\boldsymbol{b}\|_2 \;\;\text{s.t.}\;\boldsymbol{y}^{(2)} = \boldsymbol{X}^{(2)}\boldsymbol{b}\right\}.\tag{9}\]

The risk of \(\hat{\beta}^{(2)}\) has been previously studied:

The above proposition, along with our results on model shift (Theorem 1 and Corollary 2), allows us to characterize when pooled min-\(\ell_2\)-norm interpolator yields lower risk than the target-only min-\(\ell_2\)-norm interpolator, i.e. when data pooling leads to positive/negative transfer:

The implications of Proposition 4 are as follows:

1. When the SNR is small, i.e. \(\rm{SNR} \le \frac{1}{(1-1/\gamma)(1-1/\gamma_2)}\), adding more data always hurts the pooled min-\(\ell_2\)-norm interpolator. Note that, even if the source and target populations have the exact same model settings, pooling the data induces higher risk for the interpolator.

2. If SNR is large, whether pooling the data helps or not depends on the level of model shift. Namely, if SSR exceeds \(\rho\) defined by ?? , the models are far apart resulting in negative transfer. Otherwise, pooling the datasets helps. In particular, if \(\textrm{SSR} \ge \frac{1-1/\gamma}{1-1/\gamma_1}\), then for any values of \(\textrm{SNR}\), data pooling worsens performance.

3. Note that the \(\textrm{SNR}\) and \(\textrm{SSR}\) can be consistently estimated using the available samples. Based on this observation, in Appendix 8 , we provide a data-driven method for deciding whether to use the pooled interpolator or the target-based interpolator.

In sum, in the overparametrized regime, positive transfer happens when the SSR is low and the SNR is sufficiently high. When SSR is low, the source data is close to the target data, so pooling introduces little model shift bias. When SNR is high, the signal generally dominates the noise in the sample, so additional source samples can help stabilize the interpolation. In contrast, when SSR is high and SNR is low, adding source samples forces the interpolator to fit observations that are both noisy and generated from a signal far from the target of interest, thus leading to negative transfer.

From a practical perspective, these results suggest that the data-driven procedure in Appendix 8 can be used to select the number of source samples to pool with the target samples so as to approximately minimize the risk of the resulting linear interpolator.

Figure 1 illustrates the interplay between the \(\textrm{SNR}\), \(\textrm{SSR}\), and the transfer performance as described in Proposition 4. Figures 1 (a) and 1 (b) show that for a fixed source sample size \(n_1\), the risk of the pooled interpolator \(\hat{\boldsymbol{\beta}}\) decreases relative to that of the target interpolator \(\hat{\boldsymbol{\beta}}^{(2)}\) as the tasks become more similar (as \(\textrm{SSR}\) decreases) or as the target signal becomes easier to estimate (as \(\textrm{SNR}\) increases).

For completeness, we also depict the empirical performance of the pooled interpolator in the underparametrized regime (where \(n_1 + n_2 > p\)) along with the corresponding theoretical predictions from [34]. Figures 1 (a) and 1 (b) show that, when combined with our results in the overparametrized setting, a double descent phenomenon appears, though the first descent in the overparametrized setting can fail to occur for sufficiently small \(\textrm{SNR}\) or sufficiently large \(\textrm{SSR}\), where pooling remains detrimental throughout the overparametrized regime.

Finally, Figures 1 (c) and 1 (d) demonstrate that, when the SNR is sufficiently small, the pooled interpolator is outperformed by the target-only interpolator, regardless of the SSR. However, we also note that, for especially small SNR (e.g., \(\textrm{SNR} = 1\) in Figure 1 (a)), the null estimator \(\hat{\boldsymbol{\beta}}_{\text{null}}\) can outperform both interpolators throughout the range of source sample sizes \(n_1\). In these regimes with especially low SNR, this comparison should not be interpreted as identifying a practically useful transfer procedure: see Appendix 17.4 for more. On the other hand, for larger SNR, pooling improves performance when the SSR is sufficiently small. The phase transitions observed in Figures 1 (a) and 1 (b) closely match the phase transitions predicted by our theory in Proposition 4, thus providing strong finite-sample support for our results.

3.3 Risk under spiked covariance designs↩︎

In this section, we now consider the setting where \(\boldsymbol{\Sigma}^{(1)} = \boldsymbol{\Sigma}^{(2)} = \boldsymbol{I} + \alpha \boldsymbol{v}\boldsymbol{v}^\top\) for \(\alpha > 0\) and a unit vector \(\boldsymbol{v} \in \mathbb{R}^p\). In this case, we obtain the following corollary of Theorem 1.

Corollary 5 highlights how anisotropy enters the transfer learning problem. In Corollary 2, the asymptotic out-of-sample prediction risk depended on \(\boldsymbol{\beta}^{(2)}\) and \(\tilde{\boldsymbol{\beta}}\) only through their norms \(\|\boldsymbol{\beta}^{(2)}\|_2\) and \(\|\tilde{\boldsymbol{\beta}}\|_2\) and was thus invariant to the relative positions of the signal vectors \(\boldsymbol{\beta}^{(1)}\) and \(\boldsymbol{\beta}^{(2)}\). In contrast, Corollary 5 uncovers that even a rank-one perturbation from isotropy is sufficient to make the prediction risk depend on the target alignment \(\frac{\langle \boldsymbol{v}, \boldsymbol{\beta}^{(2)} \rangle}{\|\boldsymbol{\beta}^{(2)}\|_2}\) and the shift alignment \(\frac{\langle \boldsymbol{v}, \tilde{\boldsymbol{\beta}}\rangle}{\|\tilde{\boldsymbol{\beta}}\|_2}\), which suggests that the invariance of the prediction risk in the isotropic setting is particular to that setting.

Note that the variance term in Corollary 5 matches that of Corollary 2; that is, the difference in the prediction risk between the two settings is driven by differences in the bias expressions. In particular, each of the three terms in the bias decomposition can be expressed as one component that matches the corresponding bias term in Corollary 2 and one new component that depends on \(\langle \boldsymbol{v}, \boldsymbol{\beta}^{(2)}\rangle\), \(\langle \boldsymbol{v}, \tilde{\boldsymbol{\beta}}\rangle\), or both, in the cases of \(B_1\), \(B_2\), and \(B_3\), respectively. Thus, when the spike \(\boldsymbol{v}\) is orthogonal (or asymptotically orthogonal) to the target signal \(\boldsymbol{\beta}^{(2)}\) and the shift \(\tilde{\boldsymbol{\beta}}\), the risk in the spiked covariance setting reverts to the risk in the isotropic setting. On the other hand, when either the target signal \(\boldsymbol{\beta}^{(2)}\) or the shift \(\tilde{\boldsymbol{\beta}}\) is highly aligned with the spike \(\boldsymbol{v}\), the resulting asymptotic risk can differ significantly from that of the isotropic setting.

Figure 2 illustrates these findings in a simulation setting. In Figure 2 (a), when the shift alignment \(\frac{\langle \boldsymbol{v}, \tilde{\boldsymbol{\beta}}\rangle}{\|\tilde{\boldsymbol{\beta}}\|_2}\) is fixed to be zero, increasing the magnitude of the alignment between the target \(\boldsymbol{\beta}^{(2)}\) and the spike \(\boldsymbol{v}\) reduces the generalization error. Conversely, in Figure 2 (b), when the target alignment \(\frac{\langle \boldsymbol{v}, \boldsymbol{\beta}^{(2)}\rangle}{\|\boldsymbol{\beta}^{(2)}\|_2}\) is fixed to be zero, increasing the magnitude of the alignment between the shift \(\tilde{\boldsymbol{\beta}}\) and the spike \(\boldsymbol{v}\) increases the generalization error. In both cases, the effects of alignment become more pronounced as the spike strength \(\alpha\) grows. However, Figures 2 (c) and 2 (d) demonstrate that, when both the target \(\boldsymbol{\beta}^{(2)}\) and the shift \(\tilde{\boldsymbol{\beta}}\) have nonzero alignment with the spike \(\boldsymbol{v}\), their effects are no longer separable. This interaction is partly driven by the growing \(B_3\) term, which depends on the alignment between \(\tilde{\boldsymbol{\beta}}\) and \(\boldsymbol{\beta}^{(2)}\) through their one-dimensional projections onto the spike \(\boldsymbol{v}\).

3.4 Risk under a random effects model↩︎

In this section, we consider the setting where \(\tilde{\boldsymbol{\beta}}\) and \(\boldsymbol{\beta}^{(2)}\) are randomly drawn vectors. In particular, suppose that \(\tilde{\boldsymbol{\beta}}\), \(\boldsymbol{\beta}^{(2)}\) are i.i.d. uniform draws from the \(p-1\) dimensional unit sphere. In this case, one can easily determine that the quantities \(B_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})\) and \(B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})\) concentrate around the analogous tracial quantities, while \(B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})\) concentrates around \(0\). The proof of Theorem 1 then immediately yields the following corollary.

Therefore, under the random effects model, the specific geometry of \(\tilde{\boldsymbol{\beta}}\) and \(\boldsymbol{\beta}^{(2)}\) becomes asymptotically irrelevant: the risk depends only on the spectrum of \(\boldsymbol{\Sigma}\) and not on any particular alignment with the eigenvectors of \(\boldsymbol{\Sigma}\).

This stands in direct contrast with the result in Corollary 5, where the risk could depend heavily on the projections of \(\tilde{\boldsymbol{\beta}}\) and \(\boldsymbol{\beta}^{(2)}\) in the spiked direction. Here, directions for \(\tilde{\boldsymbol{\beta}}\) and \(\boldsymbol{\beta}^{(2)}\) are sampled from an isotropic distribution, which leads to a result more reminiscent of Corollary 2. However, the result here still depends on the spectrum of \(\boldsymbol{\Sigma}\) and thus allows us to probe how the covariance structure of the covariates influences the success of transfer procedures.

4 Covariate Shift↩︎

In this section, we characterize the bias and variance of the pooled min-\(\ell_2\)-norm interpolator under covariate shift. We assume the underlying signal \(\boldsymbol{\beta}^{(1)}=\boldsymbol{\beta}^{(2)}=\boldsymbol{\beta}\) stays the same for both datasets, whereas the population covariance matrix changes from \(\boldsymbol{\Sigma}^{(1)}\) to \(\boldsymbol{\Sigma}^{(2)}\). Under this condition, \(B_2(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})=B_3(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})=0\) in ?? . We then compare the remaining bias term \(B_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) and the variance term \(V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) of the pooled min-\(\ell_2\)-norm interpolator with the target-based interpolator and investigate conditions that determine achievability of positive/negative transfer.

4.1 Risk under simultaneously diagonalizable covariances↩︎

We assume that the source and target covariance matrices \((\boldsymbol{\Sigma}^{(1)},\boldsymbol{\Sigma}^{(2)})\) are simultaneously diagonalizable, as defined below:

The simultaneous diagonalizability assumption has previously been used in the transfer learning literature (cf. [30]). It ensures that the difference in the covariate distributions of the source and the target is solely captured by the eigenvalues \(\lambda_{i}^{(1)},\lambda_i^{(2)}\). Characterizing the generalization error when covariance matrices lack a common orthonormal basis appears to be fairly technical, and we defer this to future work.

Note that in the typical overparametrized setup with a single dataset, the risk of the min-\(\ell_2\)-norm interpolator is expressed in terms of the empirical distribution of eigenvalues and the angle between the signal and the eigenvector of the covariance matrix (equation (9) in [6]). Consequently, it is natural to expect that the generalization error of our estimator will depend on \(\hat{H}_p\) and \(\hat{G}_p\) defined by ?? , which indeed proves to be the case. The following presents our main result concerning covariate shift.

Note that although \((\tilde{a}_1,\tilde{a}_2)=(\tilde{b}_1,\tilde{b}_2)\) in the displayed equations above, we intentionally used different notations for clarity, as they are derived through distinct routes in the proof.

4.2 Performance comparison example: when does heterogeneity help?↩︎

In this section, we illustrate the impact of covariate shift on the performance of our pooled min-\(\ell_2\)-norm interpolator. Since the closed-form expressions for the bias and variance of the pooled min-\(\ell_2\)-norm interpolator are complicated (see ?? , ?? ), we exhibit the impact through stylized examples. Specifically, we consider the setting where the target covariance \(\boldsymbol{\Sigma}^{(2)}=\mathbf{I}\) and the source covariance \(\boldsymbol{\Sigma}^{(1)}\) has pairs of reciprocal eigenvalues: \(\boldsymbol{\Sigma}^{(1)}:=\boldsymbol{M}\in \mathcal{S}\), where \(\mathcal{S}\) denotes the collection of all diagonal matrices with pairs of reciprocal eigenvalues. To elaborate, suppose that \(p\) is an even integer. In this case, \(\boldsymbol{\Sigma}^{(1)}\) is diagonal, with \(\lambda_{p+1-i}^{(1)} = 1/\lambda_i^{(1)}\) for \(i=1,...,p/2\). Note that \(\mathbf{I} \in \mathcal{S}\). This setting leads to a simplification in the bias and the variance, and allows us to study the effects of covariate shift.

We will analyze the risk of the pooled min-\(\ell_2\)-norm interpolator \(R(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})\), defined by 7 , under this covariate shift model. To emphasize that the difference in the source and target feature covariances is captured by the matrix \(\boldsymbol{M}\), we denote this risk as \(\hat{R}(\boldsymbol{M})\) through the rest of the section.

Our first result studies the following question: under what conditions does heterogeneity reduce the risk for the pooled estimator? To this end, we compare \(\hat{R}(\boldsymbol{M}), \boldsymbol{M}\in \mathcal{S}\), with \(\hat{R}(\mathbf{I})\), i.e. the risk under covariate shift with the one without covariate shift.

Proposition 8 is proved in Appendix 13.4 . Its implications are as follows: in the overparametrized regime, heterogeneity leads to lower risk for the pooled min-\(\ell_2\)-norm interpolator if and only if the sample size from the source data is small relative to the dimension, specifically, smaller than \(p/2\). Note that since we consider the overparametrized regime, we always have \(p > n_1+n_2\); in other words, \(n_1 < p - n_2\). Thus, Proposition 8 gives an exhaustive characterization of when heterogeneity helps in terms of the relations between \(p,n_1,n_2\). More generally, an additional question may be posed: how does the degree of heterogeneity affect the risk of the pooled min-\(\ell_2\)-norm interpolator? Investigating this is hard even for the structured class of covariance matrices \(\mathcal{S}\). Therefore, we consider the case when \(\boldsymbol{\Sigma}^{(1)}\) has only two eigenvalues: \(\lambda_{p+1-i}^{(1)} = 1/\lambda_i^{(1)}=\kappa, \forall 1 \leq i \leq p/2\) for some \(\kappa>1\). We denote this covariance matrix as \(\boldsymbol{M}(\kappa)\). Note that \(\boldsymbol{M}(1)=\boldsymbol{I}\) and \(\kappa\) here denotes the degree of heterogeneity. In this special setting, we can establish the following.

The key takeaways from Proposition 9 can be summarized as follows:

1. If the size of the source dataset is small, then the risk is a decreasing function of \(\kappa\), i.e., more discrepancy between source and target dataset helps achieve lower risk for the pooled min-\(\ell_2\)-norm interpolator.

2. When the size of the source dataset is large, then larger heterogeneity hurts the pooled min-\(\ell_2\)-norm interpolator.

Lastly, similar to Section 3.2, we compare here the pooled min-\(\ell_2\)-norm interpolator with the target-based interpolator 9 . Note that the risk of the latter under similar settings has been studied in Theorem 2 of [6] which we cite below:

We remark that using similar proof techniques as Theorem 7, the additional bounded moments condition in Proposition 10 can be relaxed to Assumption 4. We omit the details. Regretfully, although the precise risks for both interpolators are available from Theorem 7 and Proposition 10, one cannot obtain a simple result even for \(\boldsymbol{\Sigma}^{(2)}=\boldsymbol{I}\) due to the complexity of expressions. Thus we make the comparison via solving the expressions numerically in the next section.

4.3 Numerical examples under covariate shift↩︎

Similar to Section 3.2, we illustrate the applicability of our results in finite samples and compare the performances of the pooled min-\(\ell_2\)-norm interpolator and its target-based counterpart. We pick the special setting of Proposition 9 where the target distribution has covariance \(\boldsymbol{I}\), while the source distribution has a covariance matrix with two distinct eigenvalues \(\kappa\) and \(1/\kappa\), each having multiplicity \(p/2\). The choice of covariance matrices ensures that both have equal determinants. In Figure 3, we illustrate how the generalization error varies with \(n_1,n_2, p\), the SNR, and the covariate shift magnitude \(\kappa\). Again, we include the generalization error of the OLS estimator for the underparametrized regime \(p<n\) for completeness.

The key insights from the plots are summarized as follows. Figure 3 (a) illustrates the generalization error of the pooled min-\(\ell_2\)-norm interpolator, showcasing the double descent phenomena across varying \(p,n_1,n_2\). The generalization error of the target-based interpolator is represented by dotted horizontal lines. We observe that in the overparametrized regime, if \(n_1\) is small, the pooled estimator consistently outperforms the target-based estimator. Figure 3 (b) presents our findings from Proposition 9. For small values of \(n_1\), the generalization error decreases with the degree of heterogeneity, i.e., the value of \(\kappa\). However, for \(n_1> \min \{p/2, p-n_2\}\), higher values of \(\kappa\) result in higher generalization error. A notable scenario occurs at \(n_1 = \min \{p/2, p-n_2\}\), where the generalization error becomes independent of \(\kappa\), causing all curves to intersect at a single point. Finally, Figure 3 (c) demonstrates the dependence of the generalization error on the SNR. Higher SNR values lead to significantly higher error when \(n_1\) is small, but the curves tend to converge at the interpolation threshold, i.e., when the total sample size matches the number of covariates. As in Figure 3 (a), the null estimator \(\hat{\boldsymbol{\beta}}_{\text{null}} = 0\) can outperform both interpolators throughout the range of source sample sizes \(n_1\) for sufficiently small SNR (e.g., \(\textrm{SNR} = 1\) in Figure 3 (c)). In such SNR regimes, this comparison should not be interpreted as identifying a practically useful transfer procedure: see Appendix 17.4 for more.

5 Beyond Gaussianity and Other Extensions↩︎

In this section, we discuss several extensions to our above results.

5.1 Universality of out-of-sample prediction risk↩︎

Our model shift result in Theorem 1 assumes that \((\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) consist of i.i.d. Gaussian entries (Assumption 2). Here, we show that the same results hold when Assumption 2 is replaced by Assumption 3. Our universality result is the following.

It immediately follows that the characterization of the out-of-sample prediction risk under model shift in Theorem 1 also holds when Assumption 2 is replaced by Assumption 3.

We complement Theorem 11 with extensive finite-sample simulations in Appendix 17.1 . These simulations compare the theoretical risk predictions for the pooled min-\(\ell_2\)-norm interpolator against empirical risks under a variety of synthetic covariate distributions, as well as semi-synthetic covariates derived from real human genomics data. These simulations validate Theorem 11 for several covariate distributions satisfying Assumptions 1 and 3, and further suggest that the same risk predictions remain accurate for heavier-tailed covariates and real-world covariates that may not exactly satisfy these assumptions.

5.2 Misspecified model↩︎

Following [6], we consider a misspecified model where the true model (for both the source and target data) is linear but we only observe a subset of the features. Formally, we assume that the data comes from linear models given by \[\label{eq:misspecified95model} \boldsymbol{y}^{(k)} = \boldsymbol{X}^{(k)} \boldsymbol{\beta}^{(k)} + \boldsymbol{W}^{(k)} \boldsymbol{\theta}^{(k)} + \boldsymbol{\epsilon}^{(k)}, k = 1,2\tag{10}\] where, once again, \(k = 1\) denotes the source and \(k = 2\) denotes the target. For \(k = 1,2\), the matrix \(\boldsymbol{X}^{(k)} \in \mathbb{R}^{n_k \times p}\) contains the observed covariates, while \(\boldsymbol{W}^{(k)} \in \mathbb{R}^{n_k \times p_{m,k}}\) contains unobserved covariates. For each \(k = 1,2\), we assume that \(\boldsymbol{X}^{(k)}\) and \(\boldsymbol{W}^{(k)}\) are independent, with covariance matrices \(\boldsymbol{\Sigma}^{(k)}\) and \(\boldsymbol{\Sigma}_m^{(k)}\), respectively. Then, for a test point \((\boldsymbol{x}_0, \boldsymbol{w}_0)\) drawn independently from the target distribution, we study the out-of-sample prediction risk for an estimator \(\hat{\boldsymbol{\beta}}\), defined in this case as follows: \[R(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)}) = \mathbb{E}[(\boldsymbol{x}_0^\top \hat{\boldsymbol{\beta}}- \boldsymbol{x}_0^\top \boldsymbol{\beta}^{(2)} - \boldsymbol{w}_0^\top \boldsymbol{\theta}^{(2)})^2 \mid \boldsymbol{X}].\] This risk admits the following decomposition [6]: \[\begin{align} R(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)}) &= \underbrace{\mathbb{E}[(\boldsymbol{x}_0^\top \hat{\boldsymbol{\beta}}- \boldsymbol{x}_0^\top \boldsymbol{\beta}^{(2)} - \mathbb{E}[\boldsymbol{w}_0 \mid \boldsymbol{x}_0]^\top \boldsymbol{\theta}^{(2)})^2 \mid \boldsymbol{X}]}_{R_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})} \\ &\qquad + \underbrace{\mathbb{E}[(\mathbb{E}[\boldsymbol{w}_0 \mid \boldsymbol{x}_0]^\top \boldsymbol{\theta}^{(2)} - \boldsymbol{w}_0^\top \boldsymbol{\theta}^{(2)})^2]}_{R_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})}. \end{align}\] We generalize our results for this model as follows.

Thus, under this form of misspecification, the unobserved covariates influence the out-of-sample prediction risk through two additive components. The first component \(R_1\) closely resembles the well-specified case, but includes an additional term reflecting the impact of misspecification through both the source and target datasets, thereby capturing errors arising from fitting the model on incomplete covariates.

The second component \(R_2\) is new and depends solely on the unobserved covariates in the target distribution. It represents an irreducible error due to misspecification, capturing the discrepancy between the true response and its best approximation based solely on the observed covariates.

In Appendix 17.2 , we provide finite-sample simulations that display the risk of the pooled min-\(\ell_2\)-norm interpolator for different amounts of target and source misspecification. These simulations confirm that our theoretical predictions in Theorem 12 closely match those seen in finite samples, thus again providing evidence for the finite-sample validity of our predictions.

5.3 Bias-corrected estimator↩︎

In the setting of model shift, we can also consider a modification of the min-norm interpolator 4 that corrects for the bias of the combined min-norm interpolator with a holdout subset of the target dataset. That is, our bias-corrected estimator—which we denote as \(\hat{\boldsymbol{\beta}}_{\text{BC}}\)—is formed by a two-step procedure, parametrized by a regularization parameter \(\lambda_\delta > 0\) and a holdout parameter \(\kappa \in (0,1)\). We present this procedure in Algorithm 4.

Note that the bias-corrected estimator \(\hat{\boldsymbol{\beta}}_{\text{BC}}\) is closely related to the estimator considered in [18], which regularizes the target estimator toward a source-only estimator. It is also conceptually related to the Transfer MNI estimator of [33], which first trains a source-only min-\(\ell_2\)-norm interpolator and then interpolates the target data while minimizing the \(\ell_2\) distance to the source estimator. Our estimator and the estimators in [18], [33] all involve a two-step estimation procedure, where the second stage produces an estimator based on the first estimator. Our construction differs in that the reference estimator is itself obtained by pooling part of the target data with the source data.

We can then prove the following result for the out-of-sample prediction risk of the bias-corrected estimator:

Theorem 13 shows that, in the isotropic setting, the risk of the bias-corrected estimator is equal to the sum of two terms. The first term is the risk of the interpolator fitted with the source data and a split of the target data, multiplied by a shrinkage factor induced by the second-stage correction. The second term is the additional variance incurred by fitting the correction on the held-out target split. Therefore, the bias-corrected estimator improves over the interpolator in 3 precisely when the reduction in the first-stage risk outweighs the additional second-stage variance cost.

In Appendix 17.3 , we provide finite-sample simulations that compare the bias-corrected estimator with the target-only and pooled min-\(\ell_2\)-norm interpolators. Across a range of SNR, SSR, sample-split parameter \(\kappa\), and second-stage regularization parameter \(\lambda_\delta\) values, the empirical risks closely track the theoretical values provided in Theorem 13.

6 Proof Outline↩︎

In this section, we outline the key ideas behind the proofs of our main results in Sections 3, 4, and 5, highlighting crucial auxiliary results along the way. All convergence statements will be presented in an approximate sense, with comprehensive proof details provided in the Appendix.

To circumvent the difficulty of dealing with the pseudo-inverse formula in 4 , we utilize the connection between the pooled min-\(\ell_2\)-norm interpolators and ridgeless least squares interpolation. This says that to study the interpolator, it suffices to study the following ridge estimator \[\label{eqn:ridge} \hat{\boldsymbol{\beta}}_\lambda = \left(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}+n\lambda\boldsymbol{I}\right)^{-1} \left(\boldsymbol{X}^{(1)\top}\boldsymbol{y}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{y}^{(2)}\right),\tag{11}\] and then study its \(\lambda \rightarrow 0\) limit.

In the remainder of this section, we focus on the proofs of Theorems 1, 7, and 11, as these results are the most technically challenging and intricate. We note that, while the proofs of Theorems 12 and 13 require careful analysis, they reuse the same machinery rather than introducing new random matrix inputs. Therefore, we omit a discussion of these proofs here and defer them to Appendix 15 .

6.1 Model shift and covariate shift↩︎

Owing to the decomposition 8 , we analyze the bias and the variance terms separately. With this in mind, our proofs of Theorems 1 and 7 consist of the following four steps:

1. For a fixed value of \(\lambda>0\), we analyze the risk of the pooled ridge estimator defined by 11 , i.e., we show there exists \(\mathcal{R}\) such that \(R(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) \overset{p \rightarrow \infty}{\longrightarrow} \mathcal{R}(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}),\) where \(R\) is defined as in 7 . [see Theorem 16 and Theorem 17 for the proof under model shift and covariate shift respectively].

2. Show that with high probability, the difference between the (random) risks of the min-\(\ell_2\)-norm interpolator and the ridge estimator vanishes as \(\lambda \rightarrow 0^+\), i.e., \(R(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) -R(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\overset{\lambda \rightarrow 0^+}{\longrightarrow} 0\).

3. Analyze the limit of the (deterministic) risk of \(\hat{\boldsymbol{\beta}}_\lambda\), i.e., \(\mathcal{R}(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) \overset{\lambda \rightarrow 0^+}{\longrightarrow} \mathcal{R}(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\).

4. Find a suitable relation between \(\lambda\) and \(p\) such that the aforementioned three convergence relationships are simultaneously satisfied.

The most nontrivial parts of these results are to prove the convergence (i) and obtain the precise expression for \(\mathcal{R}(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)})\). For (ii), one can rewrite the difference \(R(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) -R(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) in terms of the eigenvalues of some symmetric random matrices such as \(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)}\). Next, using high probability bounds of the largest and smallest eigenvalues (see Lemma 5 and Corollary 18 ), we are able to establish high probability convergence given by (ii) above. Part (iii) only involves deterministic quantities and therefore classical analysis techniques. Finally, Step (iv) reduces to optimization problems over \(\lambda\).

The rest of this section is devoted to outlining the proof for part (i). These steps for model shift and covariate shift are detailed in Appendix 12 and Appendix 13 , respectively.

6.1.1 Model shift↩︎

After decomposing \(R(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})\) into the variance term and the three bias components, the terms requiring novel analyses are \(B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})\) and \(B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})\). These involve products of the source-only and pooled covariance matrices \[\frac{1}{n} \boldsymbol{Z}^{(1)\top}\boldsymbol{Z}^{(1)} \quad \text{and} \quad \frac{1}{n} \boldsymbol{Z}^{(1)\top}\boldsymbol{Z}^{(1)} + \frac{1}{n} \boldsymbol{Z}^{(2)\top}\boldsymbol{Z}^{(2)},\] respectively. The key difficulty arises from this asymmetry: some terms depend only on the source design, while others involve both source and target designs. As a result, standard random matrix results—typically formulated for functions of a single sample covariance matrix—cannot be applied directly. This limitation also appears in related work: e.g., [34] notes that this technical hurdle prevents their OLS analysis for the underparametrized model shift setting from generalizing beyond isotropic designs. Similarly, the analysis of a different transfer estimator in [19] relies heavily on properties of Wishart matrices and thus fails to extend to non-Gaussian covariates.

Our proof resolves this asymmetry by exploiting the exchangeability of the rows of the whitened design matrices \(\boldsymbol{Z}^{(1)}\) and \(\boldsymbol{Z}^{(2)}\). At the level of expectations, source-only insertions can be converted into weighted combinations of pooled-resolvent expressions. This reduction to a single covariance matrix is delicate because it must be performed inside anisotropic quadratic forms. Therefore, we carefully apply Gaussian concentration tools (e.g., Poincaré-type inequalities and the Efron-Stein inequality) to upgrade our expectation-level reductions to high-probability statements. Finally, we are able to apply anisotropic local laws and related resolvent techniques (e.g., [6], [41]) to analyze the reduced matrix expressions.

6.1.2 Covariate shift↩︎

For covariate shift, \(B_2(\hat{\boldsymbol{\beta}}_{\lambda},\boldsymbol{\beta}^{(2)})=B_3(\hat{\boldsymbol{\beta}}_\lambda, \boldsymbol{\beta}^{(2)}) = 0\) since \(\tilde{\boldsymbol{\beta}}=0\), eliminating the need for dealing with quadratic terms with free addition. One can therefore relax the Gaussianity assumption on \((\boldsymbol{Z}^{(1)},\boldsymbol{Z}^{(2)})\) to the one in Theorem 7. However, new challenges arise with anisotropic covariances \((\boldsymbol{\Sigma}^{(1)},\boldsymbol{\Sigma}^{(2)})\). For conciseness, we illustrate the representative variance term, which is formulated as (see (F.2) in Section 13 ) \[\label{eqn:outline95V} \begin{align} V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})=& \frac{\partial}{\partial\lambda} \left\{\frac{\sigma^2\lambda}{n}\textrm{Tr}\left(\boldsymbol{\Sigma}^{(2)}(\hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I})^{-1}\right)\right\}\\ =&\frac{\partial}{\partial\lambda} \left\{\frac{\sigma^2\lambda}{n}\textrm{Tr}\left((\boldsymbol{\Lambda}^{(2)})^{1/2}\left(\boldsymbol{W}+\lambda\boldsymbol{I}\right)^{-1}(\boldsymbol{\Lambda}^{(2)})^{1/2}\right)\right\}, \end{align}\tag{12}\] where \[\boldsymbol{W}:= (\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2} +(\boldsymbol{\Lambda}^{(2)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(2)})^{1/2}.\]

This simplification highlights the necessity of the jointly diagonalizable assumption for \((\boldsymbol{\Sigma}^{(1)},\boldsymbol{\Sigma}^{(2)})\). The core challenge, therefore, reduces to characterizing the behavior of \((\boldsymbol{W}+\lambda\boldsymbol{I})^{-1}\) under the setting of Theorem 7. In other words, we aim to characterize the resolvent of \(\boldsymbol{W}\), a sum of two sample-covariance-like matrices with anisotropic covariances. To this end, we present Theorem 20 (omitted here for the sake of space), a novel type of anisotropic local law [41], which might be of independent interest. Convergence of the trace term in 12 follows as a corollary of our anisotropic local law, and further analysis of the derivative yields the value of \(V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\).

6.2 Universality↩︎

Our proof of Theorem 11 (detailed in Appendix 14 ) uses a Lindeberg replacement argument at the level of the full risk decomposition. We replace entries of the whitened design matrices \(\boldsymbol{Z}^{(k)}\) by their Gaussian counterparts and show that the distribution of each risk component is asymptotically unchanged, as \(n,p \to \infty\). The nontrivial point is that the risk components are not smooth bounded functions of the data uniformly in the ridgeless regime, as many applications of Lindeberg replacement arguments often assume. Rather, they contain resolvents whose operator norms grow as \(\lambda \downarrow 0\); additionally, as with the proofs of Theorems 1 and 7, the analysis is complicated by the interactions between source-only, target-only, and pooled covariance terms.

Therefore, we first demonstrate universality of the risk of the ridge estimator \(\hat{\boldsymbol{\beta}}_\lambda\) for a fixed \(\lambda > 0\). Using both the Sherman-Morrison formula and Taylor expansions, we reduce the replacement error to bounds on the derivatives of resolvent-based functionals. These bounds must be sharp enough to be summed over all replaced elements and must retain the explicit dependence on \(\lambda\). Therefore, both our Taylor expansions and the subsequent derivative bounds must be carefully executed to maintain sufficient control over the error. Once we have controlled the replacement error for a given \(\lambda > 0\), we take the limit \(\lambda \downarrow 0\) to establish universality of the risk of the interpolator \(\hat{\boldsymbol{\beta}}\).

7 Discussion and Future Directions↩︎

In this paper, we characterize the generalization error of the pooled min-\(\ell_2\)-norm interpolator under overparametrization and distribution shift. Our study underscores how the degree of model shift or covariate shift influences the risk associated with the interpolator. Mathematically, we derive a novel anisotropic local law, which may be of independent interest in random matrix theory. We conclude by identifying several avenues for future research and discussing some limitations of our current findings:

1. Expanding to Multiple Datasets: In Appendix 9 , we establish the risk of the pooled min-\(\ell_2\)-norm interpolator with multiple source datasets under model shift. Extending our results under covariate shift to the case of multiple source datasets is equally important. We anticipate that this would be feasible by extending our approach to incorporate more general anisotropic local laws. However, we defer this study to future work in the interest of space.

2. Combined Model and Covariate Shift: In Sections 3 and 4, we separately analyze model shift and covariate shift. A natural next step is to characterize the risk of the min-\(\ell_2\)-norm interpolator when both shifts occur simultaneously. Such settings are common in real-world transfer learning settings and have been partially explored in [34] for the underparametrized regime. Extending our precise risk characterizations to such settings would require significant new ideas for handling mixed random matrix terms involving multiple covariance structures and signal-shift directions.

3. Comparison with Other Estimators: Our pooled min-\(\ell_2\)-norm interpolator covers early and intermediate min-\(\ell_2\)-norm interpolation under a single umbrella thanks to the alternate representation 6 . Additionally, our bias-corrected estimator \(\hat{\boldsymbol{\beta}}_{\text{BC}}\) in Section 5.3 captures a non-interpolating intermediate-fusion estimator. However, a more complete picture of interpolation under transfer learning and overparametrization necessitates comprehensive comparisons with the following different types of estimators: 1) Late-fused min-\(\ell_2\)-norm interpolators, which combine min-\(\ell_2\)-norm interpolators that are individually obtained from different datasets; 2) Other types of interpolators including min-\(\ell_1\)-norm ones [23]; 3) Other estimators in overparametrized regimes including the penalized ones, beyond the ridge estimators covered in Appendix 10 .

4. Extension to Nonlinear Models: Recent literature has explored the connection between linear models and more complex models such as neural networks [50]–[53]. To be precise, for i.i.d. data \((y_i, z_i)\), \(i \le n\), \(y_i \in \mathbb{R}\), \(z_i \in \mathbb{R}^d\). Consider learning a neural network with weights \(\boldsymbol{\theta}\in \mathbb{R}^p\), \(f(.; \boldsymbol{\theta}): \mathbb{R}^d \mapsto \mathbb{R}\), \(z \mapsto f(z, \boldsymbol{\theta})\), where the form of \(f\) need not be related to the distribution of \((y_i, z_i)\). In some settings, the number of parameters \(p\) is so large that training effectively changes \(\boldsymbol{\theta}\) only by a small amount with respect to a random initialization \(\boldsymbol{\theta}_0 \in \mathbb{R}^p\). Then, given \(\boldsymbol{\theta}_0\) such that \(f(z, \boldsymbol{\theta}_0) \approx 0\), one obtains that the neural network can be approximated by \(z \mapsto \nabla_{\boldsymbol{\theta}} f(z, \boldsymbol{\theta}_0)^\top \beta\), where we have reparametrized \(\boldsymbol{\theta}=\boldsymbol{\theta}_0+ \boldsymbol{\beta}\). Such models, despite being nonlinear in \(z_i\)s, are linear in \(\boldsymbol{\beta}\). Moving beyond high-dimensional linear regression, one might ask about the statistical consequences of such linearization in the context of transfer learning. We leave such directions for future research.

5. Relaxing Simultaneous Diagonalizability: Our covariate shift result (Theorem 7) assumes that the source and target covariance matrices are simultaneously diagonalizable. A natural next step is to remove this assumption and allow non-commuting source-target covariance structures, which would allow us to extend our theory to covariate shift settings where eigenspaces change, in addition to eigenvalues. In Appendix 16 , we sketch a potential route that involves iteratively applying anisotropic local laws, thus allowing us to isolate individual components. However, the required analysis is substantially more delicate; we therefore defer a complete treatment to future work.

Acknowledgements↩︎

P.S. would like to acknowledge partial funding from NSF DMS-\(21134\), NSF CAREER Award \(2440824\) and the Office of Naval Research Award N00014-26-1-2144.

8 A Data-Driven Choice of Interpolator↩︎

Our findings in Section 3.2 highlight that, to achieve the least possible risk for pooled min-\(\ell_2\)-norm interpolator, one should neither always pool both datasets nor always use all samples in either datasets. In fact, if SNR and SSR were known, it would be possible to identify subsets of source and target datasets, having sample size \(\tilde{n}_1, \tilde{n}_2\) respectively, such that the risk of the pooled min-\(\ell_2\)-norm interpolator based on the total data of size \(\tilde{n}_1 +\tilde{n}_2\) is minimized. To this end, the purpose of this section is two-fold: first to provide an estimation strategy for SNR and SSR, and second to invoke the estimator and provide a guideline for choosing appropriate sample size.

For SNR and SSR estimation, we employ a simplified version of the recently introduced HEDE method [54]. Let \(\hat{\boldsymbol{\beta}}_d^{(k)}\) denote the following debiased Lasso estimator [55], [56]: \[\hat{\boldsymbol{\beta}}_d^{(k)} = \hat{\boldsymbol{\beta}}_{\rm{L}}^{(k)} + \frac{\boldsymbol{X}^{(k)\top} (\boldsymbol{y}^{(k)} - \boldsymbol{X}^{(k)} \hat{\boldsymbol{\beta}}_{\rm{L}}^{(k)})}{n_k - \|\hat{\boldsymbol{\beta}}_{\rm{L}}^{(k)}\|_0}\] that is based on the Lasso estimator for some fixed \(\lambda_{\rm{L}}\): \[\hat{\boldsymbol{\beta}}_{\rm{L}}^{(k)} := \mathop{\mathrm{arg\,min}}_{\boldsymbol{b}} \frac{1}{2n} \|\boldsymbol{y}^{(k)} - \boldsymbol{X}^{(k)} \boldsymbol{b}\|_2^2 + \frac{\lambda_{\textrm{L}}}{\sqrt{n_k}} \|\boldsymbol{b}\|_1,\] and let \[\hat{\tau}_k^2 =\frac{ \|\boldsymbol{y}^{(k)} - \boldsymbol{X}^{(k)} \hat{\boldsymbol{\beta}}_{\textrm{L}}^{(k)}\|_2^2}{(n_k - \|\hat{\boldsymbol{\beta}}_{\textrm{L}}^{(k)}\|_0)^2}\] be the corresponding estimated variance for \(k=1,2\). Then by Theorem 4.1 and Proposition 4.2 in [54], we know \[\begin{align} &\|\hat{\boldsymbol{\beta}}_d^{(k)}\|_2^2 - p \hat{\tau}^2_k \rightarrow \|\boldsymbol{\beta}^{(k)}\|_2^2,\;\;k=1,2, \\ &\widehat{\textrm{Var}}(\boldsymbol{y}^{(k)}) -( \|\hat{\boldsymbol{\beta}}_d^{(k)}\|_2^2 - p \hat{\tau}^2_k) \rightarrow \sigma^2,\;\;k=1,2, \\ &\|\hat{\boldsymbol{\beta}}^{(1)}_d- \hat{\boldsymbol{\beta}}^{(2)}_d\|_2^2 -p (\hat{\tau}^2_1+ \hat{\tau}^2_2) \rightarrow \|\boldsymbol{\beta}^{(1)}-\boldsymbol{\beta}^{(2)}\|_2^2. \end{align}\] where the last line invokes independence between the datasets. Hence, we obtain the following consistent estimators for SNR and SSR: \[\begin{align} \label{eq:snr95hat} \widehat{\text{SNR}}:= \frac{\|\hat{\boldsymbol{\beta}}_d^{(2)}\|_2^2 - p \hat{\tau}^2_2}{\widehat{\textrm{Var}}(\boldsymbol{y}^{(2)}) -( \|\hat{\boldsymbol{\beta}}_d^{(2)}\|_2^2 - p \hat{\tau}^2_2)} \rightarrow & \rm{SNR}, \nonumber\\ \widehat{\text{SSR}}:=\frac{\|\hat{\boldsymbol{\beta}}^{(1)}_d- \hat{\boldsymbol{\beta}}^{(2)}_d\|_2^2 -p (\hat{\tau}^2_1+ \hat{\tau}^2_2)}{\|\hat{\boldsymbol{\beta}}_d^{(2)}\|_2^2 - p \hat{\tau}^2_2} \rightarrow & \rm{SSR}. \end{align}\tag{13}\]

Given the estimators of SNR and SSR, denoted by \(\widehat{\text{SNR}}\) and \(\widehat{\text{SSR}}\) respectively, an important question arises: what is the optimal size of the target dataset for which the pooled min-\(\ell_2\)-norm interpolator achieves least mean-squared prediction error \(R(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})\)? This is answered by the following proposition:

This proposition validates our intuition for the problem: If SNR is low or SSR is large, large amount of target data is not necessary for achieving good generalization. In fact, given the estimators \(\widehat{\rm{SNR}}\), \(\widehat{\rm{SSR}}\), a data-dependent estimated optimal target data size is given by \[\hat{n}_{2,\text{opt}}=\left(p-n_1 - \sqrt{\frac{p^2}{\widehat{\rm{SNR}}}+ n_1(p-n_1) \widehat{\rm{SSR}}}\right)_{+}\] Simulations in [54] suggest that both \(\widehat{\text{SNR}}\) and \(\widehat{\text{SSR}}\) exhibit favorable finite-sample performance, which suggests favorable finite-sample performance of our data-dependent estimated optimal target data size \(\hat{n}_{2, \text{opt}}\). We defer further analysis of the finite-sample properties of \(\hat{n}_{2, \text{opt}}\) to future work.

9 Extension to Multiple Source Datasets↩︎

Our results for model shift, i.e., Theorem 1, can be extended to the general framework of multiple sources and one target datasets. Denote the target as \(T\) and \(K\) sources as \(1,...,K\), and further denote the signal shift vectors as \(\tilde{\boldsymbol{\beta}}^{(k)} := \boldsymbol{\beta}^{(k)} - \boldsymbol{\beta}^{(T)}\). Suppose we are still in the overparametrized regime \(p> n:=n_1+...+n_K+n_T\), and consider the pooled min-\(\ell_2\)-norm interpolator \[\label{eqn:multiple95joint95interpolator} \hat{\boldsymbol{\beta}}= \mathop{\mathrm{arg\,min}}\left\{\|\boldsymbol{b}\|_2 \;\;\text{s.t.}\;\boldsymbol{y}^{(T)} = \boldsymbol{X}^{(T)}\boldsymbol{b},\;\boldsymbol{y}^{(k)} = \boldsymbol{X}^{(k)}\boldsymbol{b},\;k=1,...,K.\right\}\tag{14}\] The following assumption combines and generalizes Assumptions 1, 2, and 5 to this setting.

Under these assumptions, the out-of-sample prediction risk of \(\hat{\boldsymbol{\beta}}\) on a new observation from the target distribution can similar be decomposed as \[R(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)}) = \underbrace{\|\mathbb{E}[\hat{\boldsymbol{\beta}}\mid \boldsymbol{X}] - \boldsymbol{\beta}^{(T)}\|_{\boldsymbol{\Sigma}}^2}_{B(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})} + \underbrace{\textrm{Tr}[\textrm{Cov}(\hat{\boldsymbol{\beta}}\mid X) \boldsymbol{\Sigma}]}_{V(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})}.\] Note that the pooled min-\(\ell_2\)-norm interpolator in 14 now takes the form \[\hat{\boldsymbol{\beta}}= \biggl(\boldsymbol{X}^{(T)\top} \boldsymbol{X}^{(T)} + \sum_{k=1}^K \boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)} \biggr)^\dagger \biggl(\boldsymbol{X}^{(T)\top} \boldsymbol{y}^{(T)} + \sum_{k=1}^K \boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)}\biggr).\] and thus, we can further decompose \[\begin{align} V(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)}) &= \frac{\sigma^2}{n} \textrm{Tr}(\hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}), \\ B(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)}) &= \underbrace{\boldsymbol{\beta}^{(T)\top} (\hat{\boldsymbol{\Sigma}}^\dagger \hat{\boldsymbol{\Sigma}}- \boldsymbol{I}) \boldsymbol{\Sigma}(\hat{\boldsymbol{\Sigma}}^\dagger \hat{\boldsymbol{\Sigma}}- \boldsymbol{I}) \boldsymbol{\beta}^{(T)}}_{B_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})} \\ &\qquad + \underbrace{\sum_{k=1}^{K} \tilde{\boldsymbol{\beta}}^{(k)\top} \biggl(\frac{\boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)}}{n}\biggr) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}\hat{\boldsymbol{\Sigma}}^\dagger \biggl(\frac{\boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)}}{n}\biggr)\tilde{\boldsymbol{\beta}}^{(k)}}_{B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})} \\ &\qquad + \underbrace{\sum_{k=1}^{K} 2 \boldsymbol{\beta}^{(T)\top} (\hat{\boldsymbol{\Sigma}}^\dagger \hat{\boldsymbol{\Sigma}}- \boldsymbol{I}) \boldsymbol{\Sigma}\hat{\boldsymbol{\Sigma}}^\dagger \biggl(\frac{\boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)}}{n}\biggr) \tilde{\boldsymbol{\beta}}^{(k)}}_{B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})} \\ &\qquad + \underbrace{\sum_{\substack{1 \leq k, \ell \leq K,\\ k \neq\ell}} \tilde{\boldsymbol{\beta}}^{(k)\top} \biggl(\frac{\boldsymbol{X}^{(k)\top} \boldsymbol{X}^{(k)}}{n}\biggr) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}\hat{\boldsymbol{\Sigma}}^\dagger \biggl(\frac{\boldsymbol{X}^{(\ell)\top} \boldsymbol{X}^{(\ell)}}{n}\biggr) \tilde{\boldsymbol{\beta}}^{(\ell)}}_{B_4(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})} \end{align}\] where \(\hat{\boldsymbol{\Sigma}}= \boldsymbol{X}^\top \boldsymbol{X} / n\) is again the uncentered sample covariance matrix obtained by appending all source and target samples together and where \(\tilde{\boldsymbol{\beta}}^{(k)} = \boldsymbol{\beta}^{(k)} - \boldsymbol{\beta}^{(T)}\) for \(k = 1, \dots, K\).

Like Theorem 1, the asymptotic risk of the pooled min-\(\ell_2\)-norm interpolator is characterized with the following signed measures that capture the geometry between \(\boldsymbol{\Sigma}\), \(\boldsymbol{\beta}^{(T)}\), and \(\tilde{\boldsymbol{\beta}}^{(k)}\) (for \(k = 1,\dots, K\)): \[\begin{gather} \hat{G}_n^{\boldsymbol{\beta}^{(T)}}(s) = \frac{1}{\|\boldsymbol{\beta}^{(T)}\|_2^2} \sum_{i=1}^p \langle \boldsymbol{\beta}^{(T)}, \boldsymbol{v}_i\rangle^2 \boldsymbol{1}_{\{s \geq s_i\}}, \quad \hat{G}_n^{\tilde{\boldsymbol{\beta}}^{(k)}}(s) = \frac{1}{\|\tilde{\boldsymbol{\beta}}^{(k)}\|_2^2} \sum_{i=1}^p \langle \tilde{\boldsymbol{\beta}}^{(k)}, \boldsymbol{v}_i\rangle^2 \boldsymbol{1}_{\{s \geq s_i\}} \\ \hat{G}_n^{(b,k)}(s) = \frac{1}{\|\boldsymbol{\beta}^{(T)}\|_2 \|\tilde{\boldsymbol{\beta}}^{(k)}\|_2}\sum_{i=1}^p \langle \boldsymbol{\beta}^{(T)}, \boldsymbol{v}_i\rangle \langle \tilde{\boldsymbol{\beta}}^{(k)}, \boldsymbol{v}_i\rangle \boldsymbol{1}_{\{s \geq s_i\}}, \\ \hat{G}_n^{(k,\ell)}(s) =\frac{1}{\|\boldsymbol{\beta}^{(k)}\|_2 \|\tilde{\boldsymbol{\beta}}^{(\ell)}\|_2}\sum_{i=1}^p \langle \boldsymbol{\beta}^{(k)}, \boldsymbol{v}_i\rangle \langle \tilde{\boldsymbol{\beta}}^{(\ell)}, \boldsymbol{v}_i\rangle \boldsymbol{1}_{\{s \geq s_i\}} \\ \hat{H}_n(s) = \frac{1}{p} \sum_{i=1}^p \boldsymbol{1}_{\{s \geq s_i\}}\\ \end{gather}\]

Then, the following result characterizes the bias and variance of the pooled min-\(\ell_2\)-norm interpolator under this setting:

Note that the results for \(B_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})\), \(B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})\), \(B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})\), and \(V(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})\) follow directly from Theorem 1. It thus suffices to analyze the remaining term \(B_4(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(T)})\), though the strategy and machinery we employ is identical to that of Theorem 1: we thus omit the full argument for brevity.

One can similarly verify that, in the isotropic and random effects settings, the asymptotic out-of-sample prediction risk depends only on the spectrum of the covariance matrix and the norms of the target \(\boldsymbol{\beta}^{(T)}\) and the shifts \(\tilde{\boldsymbol{\beta}}^{(1)}, \dots, \tilde{\boldsymbol{\beta}}^{(K)}\), again emphasizing the relationship between the transfer geometry and covariate geometry in determining the out-of-sample prediction risk.

10 Ridge Regression↩︎

As mentioned in Section 6, the risk of the pooled min-\(\ell_2\)-norm interpolator is closely related to that of the pooled ridge estimator in 11 , with a fixed penalty \(\lambda >0\). In this section, we obtain exact generalization error of \(\hat{\boldsymbol{\beta}}_\lambda\) under the assumptions of Section 3 and Section 4. These results are crucial intermediate results in our proofs of evaluating interpolator risk.

Similar to Lemma 1, we first state the bias-variance decomposition of the prediction risk:

In the subsections below, we present exact risk formulae under model shift and covariate shift, respectively.

10.1 Model Shift↩︎

We first define deterministic quantities that only depend on \(\gamma_1\), \(\gamma_2\), \(\gamma\), \(\lambda\), and \(\boldsymbol{\Sigma}\), then present our main results stating that these deterministic quantities are the limits of the bias and variance components of prediction risk.

In Theorem 16, both ?? and ?? follow directly from [6]. We defer the proof of ?? and ?? to Appendix 12.1.

10.2 Covariate Shift↩︎

Again, notice that \((a_1,a_2)=(b_1,b_2)\) above.

11 Basic Tools↩︎

We first collect the notations that we will use throughout the proof. Since \(p,n_1,n_2\) has comparable orders, we use \(p\) as the fundamental large parameter. All constants only depend on parameters introduced in Assumptions 1, 3, and 4. For any matrix \(\boldsymbol{X}\), \(\lambda_{\min}(\boldsymbol{A})\) denotes its smallest eigenvalue, \(\lambda_{\max}(\boldsymbol{X}) := \|\boldsymbol{X}\|_{\textrm{op}}\) denotes its largest eigenvalue (or equivalently the operator norm), \(\lambda_i(\boldsymbol{X})\) denotes its \(i\)-th largest eigenvalue, and \(\boldsymbol{X}^\dagger\) denotes its pseudoinverse. We say an event \(\mathcal{E}\) happens with high probability (w.h.p.) if \(\mathbb{P}(\mathcal{E})\rightarrow 1\) as \(p \rightarrow \infty\). We write \(f(p) = O(g(p))\) if there exists a constant \(C\) such that \(|f(p)| \leq Cg(p)\) for large enough \(p\). For \(f(p),g(p) \geq 0\), we also write \(f(p) \lesssim g(p)\) and \(g(p) = \Omega(f(p))\) if \(f(p) = O(g(p))\), and \(f(p) =\Theta( g(p))\) if \(f(p) = O(g(p))\) and \(g(p) = O(f(p))\). Whenever we talk about constants, we refer to quantities that does not depend on \(\lambda,n_1,n_2,p,\) or random quantities such as \(\boldsymbol{X}^{(k)},\boldsymbol{y}^{(k)},\boldsymbol{\epsilon}^{(k)},k=1,2\). They might depend on other constants such as \(\tau\) in Assumption 1, \(\tau_m\) (for \(m \ge 1)\) in Assumption 3, and \(\tau_\varphi\) in Assumption 4. We often use \(c,C\) to denote constants whose specific value might change from case to case. We use check symbols (e.g. \(\check x,\check y\)) to denote temporary notations (whose definitions might also change) for derivation simplicity. We further make the following definitions:

Note that since \((\log p)^C \prec 1\) for any constant \(C>0\) and we are including \(p^c\), we can safely ignore log terms. Also, for a random variable \(\xi\) with finite moments up to any order, we have \(|\xi|\prec 1\), since \[\mathbb{P}(|\xi|\geq p^c) \leq p^{-kc}\mathbb{E}|\xi|^k \leq p^{-D}\] by Markov’s inequality, when \(k > D/c\).

The following lemma collect several algebraic properties of stochastic domination:

As showed above, if the entries of \(\boldsymbol{Z}\) have finite moments up to any order, then \(\boldsymbol{Z}\) has bounded support \(Q=1\). More generally, if every entry of \(\boldsymbol{Z}\) has a finite \(\varphi\)-th moment as in Assumption 4 and \(n \leq p\), then using Markov’s inequality and a union bound we have \[\label{eqn:logp95bounded95support} \begin{align} \mathbb{P}\left(\max_{1 \leq i \leq n,1 \leq j \leq p}|Z_{ij}|\geq (\log p) p^{2/\varphi}\right) &\leq \sum_{i=1}^n \sum_{j=1}^p \mathbb{P}\left(|Z_{ij}|\geq (\log p) p^{2/\varphi}\right)\\ &\lesssim \sum_{i=1}^n \sum_{j=1}^p \left[(\log p) p^{2/\varphi)}\right]^{-\varphi} \leq (\log p)^{-\varphi}. \end{align}\tag{15}\] In other words, \(\boldsymbol{Z}\) has bounded support \(Q=p^{2/\varphi}\) with high probability.

The following lemma provides concentration bounds for linear and quadratic forms of random variables with bounded support.

The following lemma deals with the empirical spectral distribution of \(\boldsymbol{Z}^\top\boldsymbol{Z}\) for \(n < p\). Since \(\boldsymbol{Z}^\top\boldsymbol{Z}\) is rank-deficient, its empirical spectral distribution has a peak at \(0\). However, its nonzero eigenvalues are the same as the eigenvalues of \(\boldsymbol{Z}\boldsymbol{Z}^\top\). Therefore, previous results for \(\boldsymbol{Z}\boldsymbol{Z}^\top\) directly controls the positive empirical spectral distribution of \(\boldsymbol{Z}^\top\boldsymbol{Z}\).

Using a standard cut-off argument, the above results can be extended to random matrices with the weaker bounded moment assumptions:

We further cite two additional corollaries:

We now provide two more results for bounding even moments of the operator norm of random matrices. Our first result holds under Assumption 3, whereas our second result requires the stricter Assumption 2.

We can prove a tighter bound when the entries are Gaussian:

We also require the following moment bound.

12 Proof for Model Shift↩︎

Similar to Section 13, we first prove the ridge case (Theorem 16) and subsequently prove the min-norm interpolator case (Theorem 1).

12.1 Proof of Theorem 16↩︎

We only prove ?? and ?? here as ?? and ?? are direct corollaries of [6].

12.1.1 Proof of Eq. ??↩︎

Let \(\hat{\boldsymbol{\Sigma}}_Z = \boldsymbol{Z}^\top \boldsymbol{Z} / n\), and let \(I \subseteq [n]\) be a random subset of size \(n_1\) drawn without replacement. Then, by exchangeability of the rows of \(\boldsymbol{Z}\), we have \[\begin{align} B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) &= \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} \biggl(\frac{\boldsymbol{Z}^{(1)\top} \boldsymbol{Z}^{(1)}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl(\frac{\boldsymbol{Z}^{(1)\top} \boldsymbol{Z}^{(1)}}{n} \biggr) \boldsymbol{\Sigma}^{1/2}\tilde{\boldsymbol{\beta}}\\ &\overset{d}{=} \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\biggl(\frac{1}{n} \sum_{i \in I} Z_i Z_i^\top \biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl(\frac{1}{n} \sum_{i \in I} Z_i Z_i^\top \biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\\ &= \sum_{i \in I} \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\biggl(\frac{Z_i Z_i^\top}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl( \frac{Z_i Z_i^\top}{n}\biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\\ &\qquad + \sum_{i,j \in I; i \neq j} \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} \biggl(\frac{Z_i Z_i^\top}{n}\biggr)(\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl(\frac{Z_j Z_j^\top}{n} \biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}. \end{align}\] We now introduce the following quantities: \[\begin{align} B_2^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) &:=\tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\hat{\boldsymbol{\Sigma}}_Z(\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \hat{\boldsymbol{\Sigma}}_Z \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\\ B_2^P(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) &:=\sum_{i=1}^n \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\biggl(\frac{Z_i Z_i^\top}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl( \frac{Z_i Z_i^\top}{n}\biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}. \end{align}\] Then, by symmetry, it follows that \[\begin{align} \label{eq:B295decomp} \mathbb{E}[B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})] &= \frac{n_1(n_1 - 1)}{n(n-1)} \mathbb{E}[B_2^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})] \\ &\qquad + \biggl(\frac{n_1}{n} - \frac{n_1 (n_1 - 1)}{n(n-1)}\biggr) \mathbb{E}[B_2^P(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})]. \end{align}\tag{16}\] From the following lemma, it suffices to analyze \(\mathbb{E}[B_2^F(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})]\) and \(\mathbb{E}[B_2^P(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})]\):

We now analyze \(\mathbb{E}[B_2^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})]\). Note that we can express \[\begin{align} B_2^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) &= \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\hat{\boldsymbol{\Sigma}}_Z(\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \hat{\boldsymbol{\Sigma}}_Z \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\\ &= \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}\tilde{\boldsymbol{\beta}}- 2 \lambda \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-1} \boldsymbol{\Sigma}^{-1/2} \tilde{\boldsymbol{\beta}}\\ &\qquad + \lambda^2 \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{-1/2} (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \boldsymbol{\Sigma}^{-1/2} \tilde{\boldsymbol{\beta}}\\ &= \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}\tilde{\boldsymbol{\beta}}- 2 \lambda \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}(\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{\beta}}\\ &\qquad + \lambda^2 \tilde{\boldsymbol{\beta}}^\top (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}(\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{\beta}}. \label{eq:b2f95decomp} \end{align}\tag{17}\]

We now turn our attention to \(\mathbb{E}[B_2^P(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})]\). Let \(\boldsymbol{A} = (\hat{\boldsymbol{\Sigma}}^-_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-1}\). Once again, by symmetry and by the Sherman-Morrison formula, we can simplify \[\begin{align} \label{eq:symmetry95B2P} &\mathbb{E}[B_2^P(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})] \\ &= \mathbb{E}\biggl[\sum_{i=1}^n \tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2}\biggl(\frac{Z_i Z_i^\top}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl( \frac{Z_i Z_i^\top}{n}\biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\biggr] \\ &= \frac{1}{n} \mathbb{E}\biggl[ (\tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} Z_1)^2 \cdot Z_1^\top (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} Z_1 \biggr] \\ &= \frac{1}{n} \mathbb{E}[(\tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} \boldsymbol{Z}_1)^2 \cdot \boldsymbol{Z}_1^\top \boldsymbol{A}^2 \boldsymbol{Z}_1] - \frac{2}{n^2} \mathbb{E}\biggl[(\tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} \boldsymbol{Z}_1)^2 \cdot \frac{\boldsymbol{Z}_1^\top \boldsymbol{A}^2 \boldsymbol{Z}_1\boldsymbol{Z}_1^\top \boldsymbol{A} \boldsymbol{Z}_1}{1 + \frac{1}{n} \boldsymbol{Z}_1^\top \boldsymbol{A} \boldsymbol{Z}_1}\biggr] \\ &\qquad + \frac{1}{n^3} \mathbb{E}\biggl[ (\tilde{\boldsymbol{\beta}}^\top \boldsymbol{\Sigma}^{1/2} \boldsymbol{Z}_1)^2 \cdot \frac{\boldsymbol{Z}_1^\top \boldsymbol{A} \boldsymbol{Z}_1 \boldsymbol{Z}_1^\top \boldsymbol{A}^2 \boldsymbol{Z}_1 \boldsymbol{Z}_1^\top \boldsymbol{A} \boldsymbol{Z}_1}{(1 + \frac{1}{n} \boldsymbol{Z}_1^\top \boldsymbol{A} \boldsymbol{Z}_1)^2} \biggr]. \end{align}\tag{18}\] Note that, conditional on \(\boldsymbol{A}\), the first term is a product of Gaussian quadratic forms, while the second and third terms are almost equal to products of Gaussian quadratic forms. By replacing the denominators in the second and third terms by quantities independent of \(\boldsymbol{Z}_1\) before applying formulas for the expectation of products of Gaussian quadratic forms, we yield the following result:

We wish to apply existing random matrix theory results to characterize the asymptotic behavior of \(B_2^{P,\star}(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) = \|\boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\|_2^2 \frac{\frac{1}{n}\textrm{Tr}[\boldsymbol{A}^2] }{(1 + \frac{1}{n} \textrm{Tr}\boldsymbol{A})^2}\), but these results apply to the stochastic quantity, not its expectation. Therefore, we first require the following concentration result that allows us to return our attention to the stochastic setting:

Now, we can now apply results from random matrix theory to analyze \(B_2^{P,\star}(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})\).

We can now complete our analysis of \(B_2^P\).

Combining Lemmas 9, 10, and 14 yields ?? .

12.1.2 Proof of Eq. ??↩︎

Our proof of ?? follows a similar strategy. Define \[B_3^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) := \lambda \boldsymbol{\beta}^{(2)\top} \boldsymbol{\Sigma}^{-1/2} (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \hat{\boldsymbol{\Sigma}}_Z \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}.\] By exchangeability of the rows of \(\boldsymbol{Z}\) and symmetry, we have \[\begin{align} \label{eq:symmetry95B3} \mathbb{E}[B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})] &= \mathbb{E}\biggl[\lambda \boldsymbol{\beta}^{(2)\top} \boldsymbol{\Sigma}^{-1/2} (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2}\biggl(\frac{\boldsymbol{Z}^{(1)\top} \boldsymbol{Z}^{(1)}}{n} \biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\biggr] \\ &= \mathbb{E}\biggl[\lambda \boldsymbol{\beta}^{(2)\top} \boldsymbol{\Sigma}^{-1/2} (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \biggl(\frac{1}{n} \sum_{i \in I} \boldsymbol{Z}_i \boldsymbol{Z}_i^{\top}\biggr) \boldsymbol{\Sigma}^{1/2} \tilde{\boldsymbol{\beta}}\biggr] \\ &= \frac{n_1}{n}\mathbb{E}[ B_3^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})]. \end{align}\tag{22}\] We similarly prove a concentration result that implies that it suffices to analyze \(B_3^F(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})\):

As with Lemma 13, we can now apply existing anisotropic local law results to analyze \(B_3^F\).

Combining Lemmas 15 and 16 yields ?? .

12.2 Proof of Theorem 1↩︎

We only prove ?? and ?? , as ?? and ?? follow immediately from [6].

12.2.1 Finite-sample risk difference↩︎

In this section, we provide high-probability bounds for the difference in the finite-sample risk terms for the pooled ridge estimator \(\hat{\boldsymbol{\beta}}_\lambda\) and for the pooled min-\(\ell_2\)-norm interpolator \(\hat{\boldsymbol{\beta}}\), as a function of the ridge parameter \(\lambda\). Specifically, we control the differences \(|B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})|\) and \(|B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})|\).

Throughout this section, we denote by \(\boldsymbol{X} / \sqrt{n} = \boldsymbol{U} \boldsymbol{D} \boldsymbol{V}^\top\) the full singular value decomposition of \(\boldsymbol{X} / \sqrt{n}\), where \(\boldsymbol{U} \in \mathbb{R}^{n \times n}\) and \(\boldsymbol{V} \in \mathbb{R}^{p \times p}\) are orthogonal matrices. Let \(d_1 \le \cdots \le d_n\) denote the singular values, and let \(d_{\text{nz}}\) denote the smallest nonzero singular value.

We first establish a few useful bounds.

Now, we control the difference \(|B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})|\).

We can then similarly control the difference \(|B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})|\).

12.2.2 Asymptotic risk difference↩︎

In this section, we provide numerical bounds for the difference between asymptotic risk terms for the pooled ridge estimator \(\hat{\boldsymbol{\beta}}_\lambda\) and for the pooled min-\(\ell_2\)-norm interpolator \(\hat{\boldsymbol{\beta}}\), as a function of the ridge parameter \(\lambda\). That is, we control the differences \(|\mathcal{B}_2(\lambda; \hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}}) - \mathcal{B}_2( \hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}})|\) and \(|\mathcal{B}_3(\lambda; \hat{H}_n, \hat{G}_n^{(b)}) - \mathcal{B}_3( \hat{H}_n, \hat{G}_n^{(b)})|\).

We first recall that, per [6], we have that \[\begin{align} m_n(-\lambda) &= (1 - \gamma^{-1}) \frac{1}{\lambda} + c_0 + O_*(\lambda) \\ m_{n,1}(-\lambda) &= c_1 + O_*(\lambda) \end{align}\] where \(O_*(\lambda)\) is a term bounded by \(\lambda \cdot C(\tau)\), where \(C(\tau)\) is a constant only depending on \(\tau\). Importantly, the above bound implies that \[\frac{1 + \gamma + \gamma \lambda m_n(-\lambda)}{\lambda} = \gamma c_0 + O_*(\lambda).\] Using these facts, we can establish the desired bounds below.

12.2.3 Proof of Eq. ??↩︎

Take \(\lambda = p^{-1/10}\). Then, for any small \(c > 0\), we have \[\begin{align} |B_2(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) - \mathcal{B}_2(\hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}})| &\le |B_2(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) - B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})| \\ &\qquad + |B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - \mathcal{B}_2(\lambda; \hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}})| \\ &\qquad + |\mathcal{B}_2(\lambda; \hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}}) - \mathcal{B}_2(\hat{H}_n, \hat{G}_n^{\tilde{\boldsymbol{\beta}}})| \\ &= O(\lambda^{-4} p^{-1/2+c} \|\tilde{\boldsymbol{\beta}}\|_2^2) + O_*(\lambda \|\tilde{\boldsymbol{\beta}}\|_2^2) + O(n^c \lambda \|\tilde{\boldsymbol{\beta}}\|_2^2) \\ &= O(p^{-1/10+c} \|\tilde{\boldsymbol{\beta}}\|_2^2). \end{align}\] with high probability.

12.2.4 Proof of Eq. ??↩︎

Take \(\lambda = p^{-1/10}\). Then, for any small \(c > 0\), we have \[\begin{align} |B_3(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) - \mathcal{B}_3(\hat{H}_n, \hat{G}_n^{(b)})| &\le |B_3(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) - B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)})| \\ &\qquad + |B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - \mathcal{B}_3(\lambda; \hat{H}_n, \hat{G}_n^{(b)})| \\ &\qquad + |\mathcal{B}_3(\lambda; \hat{H}_n, \hat{G}_n^{(b)}) - \mathcal{B}_3(\hat{H}_n, \hat{G}_n^{(b)})| \\ &= O(\lambda^{-3} p^{-1/2+c} \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{\beta}^{(2)}\|_2) + O_*(\lambda \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{\beta}^{(2)}\|_2) \\ &\qquad + O(n^c \lambda \|\boldsymbol{\beta}^{(2)}\|_2\|\tilde{\boldsymbol{\beta}}\|_2)\\ &= O(p^{-1/10+c} \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{\beta}^{(2)}\|_2). \end{align}\] with high probability.

13 Proof for Covariate Shift↩︎

We first prove the generalization error for ridge regression under covariate shift (Theorem 17), and then obtain the generalization error for the min-norm interpolator by letting \(\lambda \rightarrow 0\). Recall the expression for bias and variance was given by ?? and ?? . Note that \(\tilde{\boldsymbol{\beta}}=\boldsymbol{0}\) here. Define \[\label{eqn:def:W} \boldsymbol{W}:= (\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2} +(\boldsymbol{\Lambda}^{(2)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(2)})^{1/2}\tag{25}\]

Taking the antiderivative, we have \[\label{eqn:variance:analytical95ridge95antider} \begin{align} &V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\\ =& \frac{\partial}{\partial\lambda} \left\{\frac{\sigma^2\lambda}{n}\textrm{Tr}\left(\boldsymbol{\Sigma}^{(2)}(\hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I})^{-1}\right)\right\}\\ =&\frac{\partial}{\partial\lambda} \left\{\frac{\sigma^2\lambda}{n}\textrm{Tr}\left(\boldsymbol{\Sigma}^{(2)}\left((\boldsymbol{\Sigma}^{(1)})^{1/2}\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}(\boldsymbol{\Sigma}^{(1)})^{1/2} +(\boldsymbol{\Sigma}^{(2)})^{1/2}\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}(\boldsymbol{\Sigma}^{(2)})^{1/2} +\lambda\boldsymbol{I}\right)^{-1}\right)\right\}\\ =&\frac{\partial}{\partial\lambda} \left\{\frac{\sigma^2\lambda}{n}\textrm{Tr}\left((\boldsymbol{\Lambda}^{(2)})^{1/2}\left(\boldsymbol{W}+\lambda\boldsymbol{I}\right)^{-1}(\boldsymbol{\Lambda}^{(2)})^{1/2}\right)\right\}, \end{align}\tag{26}\] and \[\label{eqn:bias:analytical95bias95antider} \begin{align} &B(\hat{\boldsymbol{\beta}}_\lambda,\boldsymbol{\beta}^{(2)})\\ =& -\frac{\partial}{\partial \eta} \left\{\lambda\boldsymbol{\beta}^{(2)\top}(\hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I}+\lambda\eta\boldsymbol{\Sigma}^{(2)})^{-1}\boldsymbol{\beta}^{(2)}\right\}\bigg |_{\eta=0}\\ =& -\frac{\partial}{\partial \eta} \Bigl\{\lambda\boldsymbol{\beta}_\eta^{(2)\top}\Bigl((\boldsymbol{\Lambda}_\eta^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}_\eta^{(1)})^{1/2} \\ =&+(\boldsymbol{\Lambda}_\eta^{(2)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}_\eta^{(2)})^{1/2} +\lambda\boldsymbol{I}\Bigr)^{-1}\boldsymbol{\beta}_\eta^{(2)}\Bigr\}\bigg |_{\eta=0}, \end{align}\tag{27}\] where \(\boldsymbol{\Lambda}_\eta^{(k)} := \boldsymbol{\Lambda}^{(k)}(\boldsymbol{I}+\eta\boldsymbol{\Lambda}^{(2)})^{-1}\) for \(k=1,2, \boldsymbol{\beta}_\eta^{(2)} := (\boldsymbol{I}+\eta\boldsymbol{\Lambda}^{(2)})^{-1/2}\boldsymbol{V}^\top\boldsymbol{\beta}^{(2)}\), and \(\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(k)}:= \frac{1}{n}\boldsymbol{Z}^{(k)\top}\boldsymbol{Z}^{(k)}\) where \(\boldsymbol{Z}^{(k)}\) satisfies Assumptions 1 and 4. Here \(\boldsymbol{V},\boldsymbol{\Lambda}^{(1)},\boldsymbol{\Lambda}^{(2)}\) are defined in Definition 2.

13.1 Prototype Problem↩︎

Both the bias and variance terms above 27 , 26 involves characterizing the asymptotic behavior of \((\boldsymbol{W}+\lambda\boldsymbol{I})^{-1}\) (i.e. the resolvent of \(\boldsymbol{W}\) at \(-\lambda\)) where \(\boldsymbol{W}\) is defined by 25 .

A powerful tool to handle this type of problems is known as the anisotropic local law. [41] first introduced this technique and effectively characterized the behavior of \(\left((\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2} +z\boldsymbol{I}\right)^{-1}\) for a wide range of \(z\in \mathbb{C}\), whereas [34] generalized the result to \(\left((\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2} +\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)} +z\boldsymbol{I}\right)^{-1}\) at \(z\) around \(0\).

In this section, we state and prove a more general local law which is one of the major technical contributions of this manuscript. This local law will serve as the basis for studying the limit of 27 and 26 . The following subsections follows [34] closely.

13.1.1 Resolvent and Local Law↩︎

We can write \(\boldsymbol{W}= \boldsymbol{F}\boldsymbol{F}^\top\) where \(\boldsymbol{F}\in \mathbb{R}^{p \times n}\) is given by \[\label{eqn:def:F} \boldsymbol{F}:= n^{-1/2}[ (\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\boldsymbol{Z}^{(1)\top},(\boldsymbol{\Lambda}^{(2)})^{1/2}\boldsymbol{V}^\top\boldsymbol{Z}^{(2)\top}].\tag{28}\]

We will consistently use \(i,j\in\mathcal{I}_0\) and \(\mu,\nu\in \mathcal{I}_1\cup \mathcal{I}_2\). Correspondingly, the indices are labeled as \[\boldsymbol{Z}^{(1)}=\left[Z_{\mu i}^{(1)}:i\in\mathcal{I}_0,\mu\in\mathcal{I}_1\right],\;\;\boldsymbol{Z}^{(2)}=\left[Z_{\nu i}^{(2)}:i\in\mathcal{I}_0,\nu\in\mathcal{I}_2\right].\]

We further define the set of all indices as \(\mathcal{I}:=\mathcal{I}_0\cup \mathcal{I}_1\cup \mathcal{I}_2\), and label indices as \(\mathfrak{a},\mathfrak{b},\mathfrak{c}\), etc.

Using Schur complement formula, ?? becomes \[\label{eqn:G95inverse95schur} \boldsymbol{G}(z) = \begin{pmatrix} (\boldsymbol{F}\boldsymbol{F}^\top-z\boldsymbol{I})^{-1} & (\boldsymbol{F}\boldsymbol{F}^\top-z\boldsymbol{I})^{-1}\boldsymbol{F}\\ \boldsymbol{F}^\top (\boldsymbol{F}\boldsymbol{F}^\top-z\boldsymbol{I})^{-1} & z(\boldsymbol{F}^\top\boldsymbol{F}-z\boldsymbol{I})^{-1} \end{pmatrix}\tag{29}\] Notice that the upper-left block of \(\boldsymbol{G}\) is directly related to the resolvent of \(\boldsymbol{W}=\boldsymbol{F}\boldsymbol{F}^\top\) that we are interested in. However, rather than studying \((\boldsymbol{F}\boldsymbol{F}^\top-z\boldsymbol{I})^{-1}\) directly, we work with \(\boldsymbol{H}\) as it is a linear function of \(\boldsymbol{Z}^{(1)},\boldsymbol{Z}^{(2)}\) and is therefore easier to work with.

We now define the limit of \(\boldsymbol{G}\) as \[\label{eqn:G95limit} \mathfrak{G}(z) := \begin{pmatrix} [a_1(z)\boldsymbol{\Lambda}^{(1)}+a_2(z)\boldsymbol{\Lambda}^{(2)}-z]^{-1} & 0 & 0 \\ 0 & -\frac{n}{n_1}a_1(z)\boldsymbol{I}_{n_1} & 0 \\ 0 & 0 & -\frac{n}{n_2}a_2(z)\boldsymbol{I}_{n_2} \end{pmatrix}\tag{30}\] where \(a_1(z)\) and \(a_2(z)\) are the unique solution to the following system of equations (we omit the dependence on \(z\) for conciseness): \[\label{eqns:prototype95system} \begin{align} a_1+a_2 &= 1 - \gamma \int \frac{a_1\lambda^{(1)}+a_2\lambda^{(2)}}{a_1\lambda^{(1)}+a_2\lambda^{(2)}-z} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}),\\ a_1 &= \frac{n_1}{n} - \gamma \int \frac{a_1\lambda^{(1)}}{a_1\lambda^{(1)}+a_2\lambda^{(2)}-z} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}), \end{align}\tag{31}\] such that \(\textrm{Im}(a_1) \leq 0\) and \(\textrm{Im}(a_2) \leq 0\) whenever \(\textrm{Im}(z) > 0\). The existence and uniqueness of the above system will be proved in Lemma 23.

We now state our main result, Theorem 20, which characterizes the convergence (rate) of \(\boldsymbol{G}(z)\) to \(\mathfrak{G}(z)\) at \(z=-\lambda\). This Theorem is a more general extension in the family of anisotropic local laws [41]. In specific, it extends [34] to allow a general \(\boldsymbol{\Sigma}^{(2)}\) control at \(z=-\lambda\neq 0\). For a fix small constant \(\tau>0\), we define a domain of \(z\) as \[\label{eqn:def:spectral95domain} \boldsymbol{D}\equiv \boldsymbol{D}(\tau):= \{z = E + i\eta \in \mathbb{C}_+: |z+\lambda| \leq (\log p)^{-1}\lambda\}.\tag{32}\]

The rest of this section is devoted to the proof of Theorem 20. We focus on the case of \(\lambda \in (p^{-1/7+\epsilon},1)\), since the case for \(\lambda \geq 1\) can be trivially extended by dropping all negative powers of \(\lambda\) in the proof below.

13.1.2 Self-consistent equations↩︎

In this subsection, we show that the self-consistent equations 31 has a unique solution for any \(z \in \boldsymbol{D}\). For simplicity, we further define

\[\label{eq:define95r195r2} r_1:=n_1/n, r_2 := n_2 / n.\tag{33}\]

We first show that 31 has a unique solution at \(z=-\lambda\), where 31 reduces to the following system (equivalent to the first two equations of ?? ): \[\label{eqn:fg95prototype95system} \begin{align} f(a_1,a_2) &:= a_1 - r_1 + \gamma \int \frac{a_1\lambda^{(1)}}{a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}) = 0,\\ g(a_1,a_2) &:= a_2 - r_2 + \gamma \int \frac{a_2\lambda^{(2)}}{a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}) = 0. \end{align}\tag{34}\]

Focusing on \(f\) and consider any fixed \(a_2>0\) for a small enough constant \(c\). We know \[f(C\lambda,a_2) > -r_1 +\gamma \int \frac{C\lambda\lambda^{(1)}}{C\lambda\lambda^{(1)} + a_2\lambda^{(2)}+\lambda}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})>0,\] if \(C\) is large enough, \[f(c\min\{\lambda,r_1\},a_2) < c\min\{\lambda,r_1\}-r_1 +\gamma \int \frac{c\lambda\lambda^{(1)}}{\lambda}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})<0,\] if \(c\) is small enough, and \[\frac{\partial}{\partial a_1} f(a_1,a_2) = 1 + \gamma \int \frac{\lambda^{(1)}(a_2\lambda^{(2)}+\lambda)}{(a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda)^2}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}) > 0.\] for any \(a_1>0\). Together with the obvious continuity, we know, given \(a_2\), there exists a unique \(a_1 \in (c\min\{\lambda,r_1\},C\lambda)\) such that \(f(a_1,a_2) = 0\). Define the root as \[a_1 := \chi(a_2;f).\] By implicit function theorem, we know the function \(\chi\) is continuous, and \[\chi'(a_2;f) + \gamma \int \frac{\chi'(a_2;f)\lambda^{(1)}(a_2\lambda^{(2)}+\lambda)-\chi(a_2;f)\lambda^{(1)}\lambda^{(2)}}{(a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda)^2}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})=0,\] which, after straightforward rearranging, yields \(\chi'(a_2;f)>0\).

Now focus on \(g(\chi(a_2;f),a_2)\). Since \(\chi(a_2;f) \in (c\min\{\lambda,r_2\} ,C\lambda)\) for any \(a_2>0\), similar to our derivation above, we have \[g(\chi(c\min\{\lambda,r_2\};f),c\min\{\lambda,r_1\})<0, g(\chi(C\lambda;f),C\lambda)>0\] for \(c\) small enough and \(C\) large enough, and \(\frac{\textrm{d}}{\textrm{d}a_2} g(\chi(a_2;f),a_2)>0\). Therefore there exists a unique \(a_2\in (c\min\{\lambda,r_2\},C\lambda)\) such that \(g(\chi(a_2;f),a_2)=0\). Hence, there exists a unique pair of positive \((a_1,a_2)\) satisfying 34 , such that \[\label{eqn:a195a295bounded} (a_1,a_2) \in (c\min\{\lambda,r_1\},C\lambda)\times (c\min\{\lambda,r_2\},C\lambda)\tag{35}\]

Next, we prove the existence and uniqueness of the solution to 31 for a general \(z \in \boldsymbol{D}\), where \(\boldsymbol{D}\) is defined by 32 , presented by the following lemma:

As a by-product, we obtain the following stability result which is useful for proving Theorem 20. It roughly says that if we replace the \(0\)’s on the right hand sides of 36 with some small errors, the resulting solutions will still be close to \(a_1(z),a_2(z)\). We further make the following rescaling, which will prove handy in later derivations: \[\label{eqn:rescale95m95a} m_{1c}(z) := -r_1^{-1}a_1(z),\;\;m_{2c}(z) := -r_2^{-1} a_2(z),\tag{46}\] where \(r_1,r_2\) are defined by 33 . Combining 46 with 35 yields \[\label{eqn:m1295bounded} c\lambda \leq -m_{1c}(-\lambda) \leq C\lambda,\;\;c\lambda \leq -m_{2c}(-\lambda) \leq C\lambda\tag{47}\] for some (potentially different from before) constants \(c,C > 0\).

13.1.3 Multivariate Gaussian Case↩︎

In this section we prove the idealized setting where the entries of \(\boldsymbol{Z}^{(1)}\) and \(\boldsymbol{Z}^{(2)}\) are i.i.d. Gaussian. By the rotational invariance of multivariate Gaussian distribution, we have \[\label{eqn:gaussian95rotational95invariance} \boldsymbol{Z}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2}, \boldsymbol{Z}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(2)})^{1/2} \overset{d}{=} \boldsymbol{Z}^{(1)}(\boldsymbol{\Lambda}^{(1)})^{1/2}, \boldsymbol{Z}^{(2)}(\boldsymbol{\Lambda}^{(2)})^{1/2}.\tag{49}\] Notice that since the entries of \(\boldsymbol{Z}^{(1)},\boldsymbol{Z}^{(2)}\) are i.i.d. Gaussian, they have bounded support \(Q=1\) by the remark after Definition 6. Hence we can use the resolvent method (cf. [58]) to prove the following proposition.

Proposition 21 is a corollary of the following entrywise local law.

In the remaining of this subsection we prove Lemma 25, where the resolvent in Definition 7 becomes \[\label{eqn:resolvent95gaussian} \boldsymbol{G}(z) = \begin{pmatrix} -z\boldsymbol{I}_p & n^{-1/2}(\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{Z}^{(1)\top} & n^{-1/2}(\boldsymbol{\Lambda}^{(2)})^{1/2}\boldsymbol{Z}^{(2)\top} \\ n^{-1/2}\boldsymbol{Z}^{(1)}(\boldsymbol{\Lambda}^{(1)})^{1/2} & -\boldsymbol{I}& \boldsymbol{0}\\ n^{-1/2}\boldsymbol{Z}^{(2)}(\boldsymbol{\Lambda}^{(2)})^{1/2} & \boldsymbol{0} & -\boldsymbol{I} \end{pmatrix}^{-1}.\tag{50}\]

Below we introduce resolvent minors to deal with the inverse using Schur’s complement formula.

The following formulae are identical to those in [34]. In fact they can all be proved using Schur’s complement formula.

The next lemma provides a priori estimate on the resolvent \(\boldsymbol{G}(z)\) for \(z\in \boldsymbol{D}\).

Now we are ready to prove Lemma 25.

13.1.4 Non-Gaussian extension↩︎

Now with Proposition 21, we can extend from Gaussian random matrices to generally distributed random matrices using the exact same proof as [34], since it longer involves tracking powers of \(\lambda\), we omit the details here.

13.2 Proof of Theorem 17↩︎

First, using a standard truncation argument (cf. [34]), we know that Theorem 20 holds for \(Q=p^{2/\varphi}\) with high probability if \((\boldsymbol{Z}^{(1)},\boldsymbol{Z}^{(2)})\) now satisfy the bounded moment condition in Assumption 4.

We first prove the variance term ?? . Define the auxiliary variable \[\label{eqn:def:ridge95check95V} \begin{align} \check V(\lambda) &:= \frac{\sigma^2\lambda}{n}\textrm{Tr}\biggl((\boldsymbol{\Lambda}^{(2)})^{1/2}\Bigl((\boldsymbol{\Lambda}^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(1)})^{1/2} \\ &+(\boldsymbol{\Lambda}^{(2)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}^{(2)})^{1/2} +\lambda\boldsymbol{I}\Bigr)^{-1}(\boldsymbol{\Lambda}^{(2)})^{1/2}\biggr) \\ &= \frac{\sigma^2\lambda}{n}\sum_{i \in \mathcal{I}_0} \lambda_i^{(2)}G_{ii}(-\lambda). \end{align}\tag{84}\] Since 36 reduces to the first two equations in ?? at \(z=-\lambda\), then by ?? , with high probability, \[\begin{align} \check V(\lambda) &= \frac{\sigma^2\lambda}{n} \sum_{i \in \mathcal{I}_0}\mathfrak{G}_{ii}(-\lambda) + O(p^{-1}\lambda^{-6})\\ &= \sigma^2\gamma \int \frac{\lambda\lambda^{(2)}}{a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}) + O(p^{-1}\lambda^{-6}) \\ &:=\overline{V}(\lambda)+O(p^{-1}\lambda^{-6}) \end{align}\] where \((a_1,a_2)\) is defined in ?? .

Fix \(t>0\). Let \(\boldsymbol{A}:= \hat{\boldsymbol{\Sigma}}+ \lambda\boldsymbol{I}, \boldsymbol{B}:= \hat{\boldsymbol{\Sigma}}+(\lambda+t\lambda)\boldsymbol{I}\). Then from 26 , \[\begin{align} &\left|V(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) - \frac{1}{t\lambda}(\check V(\lambda+t\lambda)-\check V(\lambda))\right|\\ =& \left|\frac{\sigma^2}{n} \textrm{Tr}\left(\boldsymbol{\Sigma}^{(2)} \hat{\boldsymbol{\Sigma}}\left( \boldsymbol{A}^{-2} - \frac{1}{t\lambda}(\boldsymbol{A}^{-1} - \boldsymbol{B}^{-1})\right)\right)\right|\\ \lesssim& \left\| \boldsymbol{\Sigma}^{(2)} \hat{\boldsymbol{\Sigma}}\left( \boldsymbol{A}^{-2} - \frac{1}{t\lambda}(\boldsymbol{B}^{-1}(\boldsymbol{B}-\boldsymbol{A})\boldsymbol{A}^{-1})\right)\right\|_{\textrm{op}}\\ \lesssim& \left\| \hat{\boldsymbol{\Sigma}}\left( \boldsymbol{A}^{-2} - \boldsymbol{B}^{-1}\boldsymbol{A}^{-1}\right)\right\|_{\textrm{op}}\\ = & t\lambda \|\hat{\boldsymbol{\Sigma}}\boldsymbol{B}^{-1}\boldsymbol{A}^{-2}\|_{\textrm{op}}\\ =& O(t\lambda^{-2}) \end{align}\] with high probability, where in the last line we used Corollary 18.

Similarly (but without randomness), we have \[\begin{align} &\left|\sigma^2\gamma\int \frac{\lambda^{(1)}\lambda^{(2)}(a_1-a_3\lambda)+(\lambda^{(2)})^2(a_2-a_4\lambda)}{(a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}) - \frac{1}{t\lambda}(\overline{V}(\lambda+t\lambda)-\overline{V}(\lambda))\right|\\ &=O(t\lambda^{-2}), \end{align}\] where \((a_3,a_4):=\frac{\partial}{\partial\lambda}(a_1,a_2)\) satisfies the last two equations in ?? . Combining the previous two displays and setting \(t=p^{-1/2}\lambda^{-5/2}\), we get \[\begin{align} &\left|V(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)})-\sigma^2\gamma\int \frac{\lambda^{(1)}\lambda^{(2)}(a_1-a_3\lambda)+(\lambda^{(2)})^2(a_2-a_4\lambda)}{(a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}) \right|\\ \leq & \frac{1}{t\lambda} \left( |\check V(\lambda)-\overline{V}(\lambda)| + |\check V(\lambda+t\lambda)-\overline{V}(\lambda+t\lambda)|\right) + O(t\lambda^{-2})\\ =& O(t^{-1}p^{-1}\lambda^{-7}) + O(t\lambda^{-2})\\ =& O(p^{-1/2}\lambda^{-9/2}) \prec p^{-1/2+c}\lambda^{-9/2} \end{align}\] for any small constant \(c>0\), thereby obtaining ?? .

Similarly, for the bias term, define the auxiliary variables \[\label{def:check95B} \begin{align} \check B(\eta) &=\lambda\boldsymbol{\beta}_\eta^{(2)\top}\Bigl((\boldsymbol{\Lambda}_\eta^{(1)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(1)}\boldsymbol{V}(\boldsymbol{\Lambda}_\eta^{(1)})^{1/2}\\ &+(\boldsymbol{\Lambda}_\eta^{(2)})^{1/2}\boldsymbol{V}^\top\hat{\boldsymbol{\Sigma}}_{\boldsymbol{Z}}^{(2)}\boldsymbol{V}(\boldsymbol{\Lambda}_\eta^{(2)})^{1/2} +\lambda\boldsymbol{I}\Bigr)^{-1}\boldsymbol{\beta}_\eta^{(2)}, \\ \overline{B}(\eta) &= \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \int \frac{\lambda}{b_1\lambda^{(1)}+b_2\lambda^{(2)}+\lambda(1+\eta\lambda^{(2)})} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)}), \end{align}\tag{85}\] where \((b_1,b_2)\) is defined in ?? , then we know by ?? that with high probability, \[\check B(\eta) = \overline{B}(\eta) + O(p^{-1/2}\lambda^{-3})\]

Let \(\eta > 0\) and denote \(\boldsymbol{A}:= \hat{\boldsymbol{\Sigma}}+ \lambda\boldsymbol{I}, \boldsymbol{B}:= \hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I}+\lambda\eta\boldsymbol{\Sigma}^{(2)}\). Then from 27 , \[\begin{align} &\left| B(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) - \frac{1}{\eta}(\check B(0)-\check B(\eta))\right|\\ =& \left|\lambda\boldsymbol{\beta}^{(2)\top}\left(\lambda\boldsymbol{A}^{-1}\boldsymbol{\Sigma}^{(2)}\boldsymbol{A}^{-1}-\frac{1}{\eta}(\boldsymbol{A}^{-1}-\boldsymbol{B}^{-1})\right)\boldsymbol{\beta}^{(2)} \right|\\ \lesssim & \lambda \left\| \left(\lambda\boldsymbol{A}^{-1}\boldsymbol{\Sigma}^{(2)}\boldsymbol{A}^{-1}-\frac{1}{\eta}(\boldsymbol{B}^{-1}(\boldsymbol{B}-\boldsymbol{A})\boldsymbol{A}^{-1})\right) \right\|_{\textrm{op}}\\ =& \lambda^2 \|(\boldsymbol{A}^{-1}-\boldsymbol{B}^{-1})\boldsymbol{\Sigma}^{(2)}\boldsymbol{A}^{-1}\|_{\textrm{op}}\\ =& \lambda^3\eta \|\boldsymbol{B}^{-1}\boldsymbol{\Sigma}^{(2)}\boldsymbol{A}^{-1}\boldsymbol{\Sigma}^{(2)}\boldsymbol{A}^{-1}\|_{\textrm{op}}\\ =& O(\eta). \end{align}\] with high probability. Similarly, we have \[\left| \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \int \frac{b_3\lambda\lambda^{(1)}+(b_4+\lambda)\lambda\lambda^{(2)}}{(b_1\lambda^{(1)}+b_2\lambda^{(2)}+\lambda)^2} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)}) - \frac{1}{\eta}(\overline{B}(0) - \overline{B}(\eta))\right| + O(\eta),\] where \((b_3,b_4):=\frac{\partial}{\partial \eta}(b_1,b_2)\) satisties the last two equations in ?? . Combining the previous two displays and setting \(\eta = p^{-1/4}\lambda^{-3/2}\), we get \[\begin{align} & \left|B(\hat{\boldsymbol{\beta}}_{\lambda};\boldsymbol{\beta}^{(2)}) - \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \int \frac{b_3\lambda\lambda^{(1)}+(b_4+\lambda)\lambda\lambda^{(2)}}{(b_1\lambda^{(1)}+b_2\lambda^{(2)}+\lambda)^2} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)})\right|\\ \leq & \frac{1}{\eta} \left( |\check B(0)-\overline{B}(0)| + |\check B(\eta) - \overline{B}(\eta)| \right) + O(\eta)\\ =& O(\eta^{-1}p^{-1/2}\lambda^{-3}) + O(\eta)\\ =& O(p^{-1/4}\lambda^{-3/2}) \prec p^{-1/4+c}\lambda^{-3/2} \end{align}\] for any small constant \(c>0\), thereby obtaining ?? .

13.3 Proof of Theorem 7↩︎

The proof involves the following three components.

1. First, we prove that the limiting risk of the ridge estimator is close to the limiting risk of the interpolator.

   We first consider the limits of the variances, which are defined respectively as \[\label{eqn:def:design95shift95limits95V} \begin{align} \mathcal{V}^{(d)}(\hat{\boldsymbol{\beta}}) &:= -\sigma^2\gamma\int \frac{\lambda^{(2)}(\tilde{a}_3\lambda^{(1)}+\tilde{a}_4\lambda^{(2)})}{(\tilde{a}_1\lambda^{(1)}+\tilde{a}_2\lambda^{(2)}+1)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)})\\ \mathcal{V}^{(d)}(\hat{\boldsymbol{\beta}}_\lambda)&:= \sigma^2\gamma\int \frac{\lambda^{(1)}\lambda^{(2)}(a_1-a_3\lambda)+(\lambda^{(2)})^2(a_2-a_4\lambda)}{(a_1\lambda^{(1)}+a_2\lambda^{(2)}+\lambda)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}) \end{align}\tag{86}\]

   To this end, we first show that \[\label{eqn:a95tildea95close} \check a_k := a_k / \lambda = \tilde{a}_k + O(\lambda),\;\;k=1,2.\tag{87}\] From 87 , we know the first two equations of ?? becomes \[\label{eqn:design95shift95ridge95alpha95eqns95rescaled} \begin{align} \check f(\check a_1,\check a_2) &:= r_1 - \gamma \int \frac{\check a_1 \lambda^{(1)}}{\check a_1 \lambda^{(1)} + \check a_2 \lambda^{(2)} + 1} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)})= \check a_1 \lambda,\\ \check g(\check a_1,\check a_2) &:= r_2 - \gamma \int \frac{\check a_2 \lambda^{(1)}}{\check a_1 \lambda^{(1)} + \check a_2 \lambda^{(2)} + 1} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}) = \check a_2 \lambda.\\ \end{align}\tag{88}\] Let \(\check \boldsymbol{F}(\check \boldsymbol{a}) :=(\check f(\check \boldsymbol{a}),\check g(\check \boldsymbol{a}))\) where \(\check \boldsymbol{a}:=(\check a_1,\check a_2)\). From 87 and 35 , we know there exists some constants \(c,C>0\) such that \(c \leq \check a_1,\check a_2 \leq C\). This yields \(\check \boldsymbol{F}(\check \boldsymbol{a}) = (\check a_1\lambda,\check a_2\lambda) \sim \lambda\). Also, similar to how we derived 35 , we can also show that there exists constants \(c,C>0\) such that \[\label{eqn:tilde95a95bounded} c \leq \tilde{a}_1,\tilde{a}_2 \leq C\tag{89}\] . On the other hand, similar to how we obtained 37 and 40 , let \(\check \boldsymbol{J}\) be the Jacobian of \(\check \boldsymbol{F}\), we can similarly show that \(\|\boldsymbol{J}(\boldsymbol{a})\|_{\textrm{op}} = O(1)\) for all \(\boldsymbol{a}\in [c,C]^2\). Together with the fact that \(\check \boldsymbol{F}(\tilde{\boldsymbol{a}}) = 0\) from ?? , we have obtained 87 .

   Now taking the derivative of 88 with respect to \(\lambda\), we get \[\label{eqn:design95shift95ridge95alpha95eqns95der} \begin{align} \check a_1 + \check a_1'\lambda &= -\gamma\int \frac{\check a_1'\lambda^{(1)}+\lambda^{(1)}\lambda^{(2)}(\check a_1'\check a_2-\check a_2'\check a_1)}{(\check a_1\lambda^{(1)}+\check a_2\lambda^{(2)}+1)^2}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}),\\ \check a_2 + \check a_2'\lambda &= -\gamma\int \frac{\check a_2'\lambda^{(2)}+\lambda^{(1)}\lambda^{(2)}(\check a_2'\check a_1-\check a_1'\check a_2)}{(\check a_1\lambda^{(1)}+\check a_2\lambda^{(2)}+1)^2}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)}), \end{align}\tag{90}\]

   where \((a_1',a_2')\) represents respective derivatives with respect to \(\lambda\). Since \(c \leq \check a_1,\check a_2 \leq C\), we know \((\check a_1',\check a_2') = O(\lambda^{-1})\), which means the RHS of 90 is \(O(1)\). Thus the LHS is also \(O(1)\), which yields \((\check a_1',\check a_2') = O(1) \Rightarrow (\check a_1'\lambda,\check a_2'\lambda)=O(\lambda)\). Now combining 90 and the last two equations of ?? , we can use similar analysis to conclude that \[\label{eqn:tildea395checka1p95close} \check a_1' = \tilde{a}_3 + O(\lambda),\;\;\check a_2' = \tilde{a}_4 + O(\lambda)\tag{91}\]

   Finally, from the second equation of 86 and the fact that \(a_k = \check a_k \lambda \Rightarrow a_{k+2}:=a_k' = \check a_k+\check a_k' \lambda\), we know \[\mathcal{V}^{(d)}(\hat{\boldsymbol{\beta}}_\lambda):= -\sigma^2\gamma\int \frac{\lambda^{(2)}(\check a_1'\lambda^{(1)}+\check a_2'\lambda^{(2)})}{(\check a_1\lambda^{(1)}+\check a_2\lambda^{(2)}+1)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}),\] which, when combined with 87 and 91 , yields \[\label{eqn:design95shift95V95limits95close} \mathcal{V}^{(d)}(\hat{\boldsymbol{\beta}}_\lambda) = \mathcal{V}^{(d)}(\hat{\boldsymbol{\beta}}) + O(\lambda).\tag{92}\]

   Using similar analysis we can also show that the bias limits, defined respectively as \[\label{eqn:def:design95shift95limits95B} \begin{align} \mathcal{B}^{(d)}(\hat{\boldsymbol{\beta}}) &:= \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot\int \frac{\tilde{b}_3\lambda^{(1)}+(\tilde{b}_4+1)\lambda^{(2)}}{(\tilde{b}_1\lambda^{(1)}+\tilde{b}_2\lambda^{(2)}+1)^2} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)}) \\ \mathcal{B}^{(d)}(\hat{\boldsymbol{\beta}}_\lambda)&:=\|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \int \frac{b_3\lambda\lambda^{(1)}+(b_4+\lambda)\lambda\lambda^{(2)}}{(b_1\lambda^{(1)}+b_2\lambda^{(2)}+\lambda)^2} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)}), \end{align}\tag{93}\] are close: \[\label{eqn:design95shift95B95limits95close} \mathcal{B}^{(d)}(\hat{\boldsymbol{\beta}}_\lambda) = \mathcal{B}^{(d)}(\hat{\boldsymbol{\beta}}) + O(\lambda).\tag{94}\]

2. Next, we show that the ridge risk is close to the interpolator risk with high probability.

   We begin by showing \(V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})\) and \(V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\) are close, where the quantities are defined in ?? and ?? respectively. To this end, denote the singular value decomposition of \(\hat{\boldsymbol{\Sigma}}\) by \(\hat{\boldsymbol{\Sigma}}= \boldsymbol{U}\boldsymbol{D}\boldsymbol{U}^\top\). Then the quantities ?? and ?? can be rewritten as \[\begin{align} V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})&= \frac{\sigma^2}{n} \textrm{Tr}(\boldsymbol{U}\boldsymbol{D}^{-1} \boldsymbol{1}_{\boldsymbol{D}>0}\boldsymbol{U}^\top\boldsymbol{\Sigma}^{(2)})\\ V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})&= \frac{\sigma^2}{n} \textrm{Tr}(\boldsymbol{U}(\boldsymbol{D}+\lambda \boldsymbol{I})^{-2} \boldsymbol{D}\boldsymbol{U}^\top \boldsymbol{\Sigma}^{(2)}), \end{align}\] where \(\boldsymbol{1}_{\boldsymbol{D}>0}\) is the diagonal matrix with \((i,i)\)-th entry equals \(1\) if \(D_{ii}>0\) and \(0\) otherwise. Therefore, we have,

   \[\begin{align}\label{eqn:design95shift95V95originals95close} &|V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)})-V(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})|\\ \le& \frac{\sigma^2}{n} \|\boldsymbol{U}^\top \boldsymbol{\Sigma}^{(2)} \boldsymbol{U}\|_{\textrm{op}} \textrm{Tr}(\boldsymbol{D}^{-1} \boldsymbol{1}_{\boldsymbol{D}>0}-(\boldsymbol{D}+\lambda \boldsymbol{I})^{-2} \boldsymbol{D})\\ \lesssim& \frac{\sigma^2}{n}\textrm{Tr}(\boldsymbol{D}^{-1} \boldsymbol{1}_{\boldsymbol{D}>0}-(\boldsymbol{D}+\lambda \boldsymbol{I})^{-2} \boldsymbol{D}) \\ \lesssim& \frac{\lambda \sigma^2}{\lambda^2_{\min}(\hat{\Sigma})}= O(\lambda), \end{align}\tag{95}\] where \(\lambda_{\min}\) denotes the minimum nonzero eigenvalue. Here, the second inequality follows from the fact that \(\|\boldsymbol{\Sigma}^{(2)}\|_{\textrm{op}} \lesssim 1\), the third inequality holds because \(x^{-1}-(x+\lambda)^{-2}x\le 2\lambda/x^2\) for \(x>0\), and the final inequality is given by the following lemma.

   Lemma 31. Consider the same setup as Theorem 7. Recall \(\hat{\boldsymbol{\Sigma}}= \frac{1}{n}(\boldsymbol{X}^{(1)\top}\boldsymbol{X}^{(1)} + \boldsymbol{X}^{(2)\top}\boldsymbol{X}^{(2)})\). Let \(\lambda_{\min}(\hat{\boldsymbol{\Sigma}})\) denote the smallest nonzero eigenvalue of \(\hat{\boldsymbol{\Sigma}}\), then there exists a positive constant \(c\) such that with high probability, \[\label{eqn:min95eval95heterogeneous} \lambda_{\min}^2(\hat{\boldsymbol{\Sigma}}) \ge c\qquad{(41)}\]

   Proof. We know \(\lambda_{\min}(\hat{\boldsymbol{\Sigma}})= \lambda_{\min}\left(\frac{1}{n}\boldsymbol{X}\boldsymbol{X}^\top\right)\) where \[\boldsymbol{X}^\top = [(\boldsymbol{\Sigma}^{(1)})^{1/2}\boldsymbol{Z}^{(1)\top},(\boldsymbol{\Sigma}^{(2)})^{1/2}\boldsymbol{Z}^{(2)\top}]\] and \(\boldsymbol{Z}^{(1)},\boldsymbol{Z}^{(2)}\) satisfies Assumption 4. From [66], we know that with high probability, \[\label{eqn:X95free95eval95close} \lambda_{\min}\left(\frac{1}{n}\boldsymbol{X}\boldsymbol{X}^\top\right) \geq \lambda_{\min}\left(\frac{1}{n}\boldsymbol{X}_{G}\boldsymbol{X}_{G}^\top\right) - o(1),\tag{96}\] where \(\boldsymbol{X}_G^\top = [(\boldsymbol{\Sigma}^{(1)})^{1/2}\boldsymbol{Z}_G^{(1)\top},(\boldsymbol{\Sigma}^{(2)})^{1/2}\boldsymbol{Z}_G^{(2)\top}]\) and \(\boldsymbol{Z}_{G}^{(1)},\boldsymbol{Z}_G^{(2)}\) have i.i.d. standard Gaussian entries instead. Since we assumed \(\boldsymbol{\Sigma}^{(1)}\) and \(\boldsymbol{\Sigma}^{(2)}\) are simultaneously diagonalizable, we further have, by rotational invariance, \[\lambda_{\min}\left(\frac{1}{n}\boldsymbol{X}_G\boldsymbol{X}_G^\top\right) = \lambda_{\min} \left(\frac{1}{n}\check \boldsymbol{Z}\check\boldsymbol{Z}^\top\right)\] where \(\check\boldsymbol{Z}^\top = [(\boldsymbol{\Lambda}^{(1)})\boldsymbol{Z}^{(1)\top},(\boldsymbol{\Lambda}^{(2)})\boldsymbol{Z}^{(2)\top}]\) has (blockwise) diagonal covariances. From [67], we obtain that with high probability, \(\lambda_{\min}^2\left(\frac{1}{n}\check \boldsymbol{Z}\check\boldsymbol{Z}^\top\right) = \check \lambda + o(1)\) where \(\check \lambda\) is the solution to the following variational problem: \[\label{eqn:variational} \begin{align} \check \lambda &:= \sup_{x_1,...,x_n < 0} \min_{1\leq i \leq n} \left\{ \frac{1}{x_i} + \sum_{j=1}^p \frac{\frac{1}{n}\lambda_j^{(1)}}{1 - \sum_{k=1}^{n_1} \frac{1}{n}\lambda_j^{(1)}x_k + \sum_{k=n_1+1}^{n}\frac{1}{n}\lambda_{j}^{(2)}x_k}\right\} \\ &=\sup_{\alpha_1,\alpha_2>0} \min_{k=1,2} \left\{-\frac{r_k}{\alpha_k}+\gamma \int \frac{\lambda^{(k)}}{\alpha_1\lambda^{(1)}+\alpha_2\lambda^{(2)}+1} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})\right\}, \end{align}\tag{97}\] where recall that \(r_k=n_k/n\), \(k=1,2\) and \(d \hat{H}_p\) is defined by ?? . The second equality above the fact that the first line is maximized at \(x_1=...=x_{n_1},x_{n_1+1}=...=x_{n}\), and \(\alpha_1:=-r_1x_1,\alpha_2:=-r_2x_{n_1+1}\).

   Recall the definition of \((\tilde{a}_1,\tilde{a}_2)\) in ?? . For \(k=1,2\), we set \(\alpha_k = 2\tilde{a}_k\), we have \[\begin{align} &-\frac{r_k}{\alpha_k}+\gamma \int \frac{\lambda^{(k)}}{\alpha_1\lambda^{(1)}+\alpha_2\lambda^{(2)}+1} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})\\ =&-\frac{1}{\alpha_k}\left( r_k - \gamma \int \frac{\tilde{a}_k\lambda^{(k)}}{\tilde{a}_1\lambda^{(1)}+\tilde{a}_2\lambda^{(2)}+\frac{1}{2}} d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})\right)\\ =& \frac{\gamma}{\tilde{a}_k} \int \frac{\frac{1}{2}\tilde{a}_k \lambda^{(k)}}{(\tilde{a}_1\lambda^{(1)}+\tilde{a}_2\lambda^{(2)}+\frac{1}{2})(\tilde{a}_1\lambda^{(1)}+\tilde{a}_2\lambda^{(2)}+1)}d\hat{H}_p(\lambda^{(1)},\lambda^{(2)})\\ \geq & c \end{align}\] for some constant \(c>0\), where the last line follows from 89 . Now combining with 97 and 96 , we obtain ?? , thereby finishing the proof. ◻

   Now we turn to show \(B(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\) and \(B(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\) are close, where the quantities are defined via ?? and ?? respectively. Since \(\tilde{\boldsymbol{\beta}}=0\), we only need to compare \(B_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\) and \(B_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})\). Note that, \[\begin{align} B_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) &= \boldsymbol{\beta}^{(2)^\top} (\hat{\boldsymbol{\Sigma}}^\dagger\hat{\boldsymbol{\Sigma}}- \boldsymbol{I})\boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}^\dagger\hat{\boldsymbol{\Sigma}}- \boldsymbol{I}) \boldsymbol{\beta}^{(2)}\\ &= \boldsymbol{\beta}^{(2)^\top} \boldsymbol{U}\boldsymbol{1}_{\boldsymbol{D}=0} \boldsymbol{U}^\top \boldsymbol{\Sigma}^{(2)} \boldsymbol{U}\boldsymbol{1}_{\boldsymbol{D}=0} \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}\\ &:= \boldsymbol{\beta}^{(2)^\top} \boldsymbol{U}\boldsymbol{1}_{\boldsymbol{D}=0} \boldsymbol{A}\boldsymbol{1}_{\boldsymbol{D}=0} \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}= \|\boldsymbol{A}^{1/2} \boldsymbol{1}_{\boldsymbol{D}=0} \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}\|^2_2, \end{align}\] where we defined \(\boldsymbol{A}= \boldsymbol{U}^\top \boldsymbol{\Sigma}^{(2)} \boldsymbol{U}\). Also, \[\begin{align} B_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})&= \lambda^2\boldsymbol{\beta}^{(2)^\top} (\hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I})^{-1}\boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+\lambda\boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}\\ &=\lambda^2\boldsymbol{\beta}^{(2)^\top} \boldsymbol{U}(\boldsymbol{D}+\lambda \boldsymbol{I})^{-1}\boldsymbol{A}(\boldsymbol{D}+\lambda \boldsymbol{I})^{-1} \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}\\ &= \|\boldsymbol{A}^{1/2} \lambda(\boldsymbol{D}+\lambda \boldsymbol{I})^{-1} \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}\|^2_2. \end{align}\] Therefore, we have \[\begin{align} |B^{1/2}_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) -B^{1/2}_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})| &\le \|\boldsymbol{A}^{1/2}(\boldsymbol{1}_{\boldsymbol{D}=0}- \lambda(\boldsymbol{D}+\lambda \boldsymbol{I})^{-1}) \boldsymbol{U}^\top \boldsymbol{\beta}^{(2)}\|_2\\ & \lesssim \|\boldsymbol{A}\|^{1/2}_{\textrm{op}}\|\lambda(\boldsymbol{D}+\lambda \boldsymbol{I})^{-1})\boldsymbol{1}_{\boldsymbol{D}>0}\|_2\\ &\lesssim \frac{\lambda}{\lambda_{\min}(\hat{\Sigma})} =O(\lambda), \end{align}\] where the second inequality holds as \(\|\boldsymbol{A}\|_{\textrm{op}}=\|\boldsymbol{\Sigma}^{(2)}\|_{\textrm{op}}\lesssim 1\) and the last inequality follows from Lemma [lemma:min95eigenvalue95heterogeneous]. Moreover, since \(B_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)}), B_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) \lesssim 1\), we have \[\label{eqn:design95shift95B95originals95close} |B_1(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) -B_1(\hat{\boldsymbol{\beta}}_\lambda;\boldsymbol{\beta}^{(2)})| = O(\lambda).\tag{98}\]

3. Now we combine previous displays to obtain the desired results. Combining ?? , 92 and 95 , we get \[\begin{align} V(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) &= -\sigma^2\gamma\int \frac{\lambda^{(2)}(\tilde{a}_3\lambda^{(1)}+\tilde{a}_4\lambda^{(2)})}{(\tilde{a}_1\lambda^{(1)}+\tilde{a}_2\lambda^{(2)}+1)^2} d \hat{H}_p(\lambda^{(1)},\lambda^{(2)}) \\ &\qquad + O(\lambda^{-9/2}p^{-1/2+c})+O(\lambda). \end{align}\] Combining ?? , 94 and 98 , we get \[\begin{align} B(\hat{\boldsymbol{\beta}};\boldsymbol{\beta}^{(2)}) &= \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot\int \frac{\tilde{b}_3\lambda^{(1)}+(\tilde{b}_4+1)\lambda^{(2)}}{(\tilde{b}_1\lambda^{(1)}+\tilde{b}_2\lambda^{(2)}+1)^2} d \hat{G}_p(\lambda^{(1)},\lambda^{(2)}) \\ &\qquad + O(\lambda^{-3/2}p^{-1/4+c}) + O(\lambda). \end{align}\] Taking \(\lambda=p^{-1/12}\) and choosing \(c\) small enough completes the proof.

13.4 Proof of Proposition 8↩︎

Assume throughout the example that \(n_1+n_2<p\). Here, we want to compare \(\hat{R} (\boldsymbol{I})\) with \(\hat{R} (\boldsymbol{M}), \boldsymbol{M}\in \mathcal{S}\). To this end, plugging in \(\boldsymbol{\Sigma}^{(1)}=\boldsymbol{\Sigma}^{(2)}=\boldsymbol{I}\) into Theorem 7 yields

\[\label{eq:r95i} \hat{R}(\boldsymbol{I})=\sigma^2\cdot\frac{n}{p-n} + \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \frac{p-n}{p} + o(1),\tag{99}\] with high probability, where the first term is the variance and the second term is the bias.

Now consider any \(\boldsymbol{\Sigma}^{(1)} =\boldsymbol{M}\in \mathcal{S}\). Plugging in \(\boldsymbol{\Sigma}^{(2)}=\boldsymbol{I}\) and combining ?? and the third equation of ?? , we obtain that the bias is also \(\|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot \frac{p-n}{p}\). Therefore we are left with comparing the variance.

Plugging in \(\boldsymbol{\Sigma}^{(2)}=\boldsymbol{I}\) and combining ?? and the third equation of ?? , we obtain that the variance is \(\sigma^2 \cdot (\tilde{a}_1+\tilde{a}_2)\). Moreover, the first and second equations of ?? yields \(\tilde{a}_2 = \frac{n_2}{p-n}\). Therefore, using 99 , the proof is complete upon showing \(\tilde{a}_1 \leq \frac{n_1}{p-n}\) if and only if \(n_1 \leq p/2\).

Since \(\tilde{a}_2 = \frac{n_2}{p-n}\), the second equation of ?? yields \[\label{eq:a195example} 1 = \sum_{i=1}^{p/2}\left[ \frac{1}{\tilde{a}_1(p-n)\lambda_i^{(1)}+p-n_1} + \frac{1}{\tilde{a}_1(p-n)/\lambda_i^{(1)}+p-n_1}\right].\tag{100}\] We know the right hand side is a decreasing function of \(\tilde{a}_1\). Hence \(\tilde{a}_1 \leq \frac{n_1}{p-n}\) if and only if we have \[1 \geq \sum_{i=1}^{p/2}\left[ \frac{1}{n_1\lambda_i^{(1)}+p-n_1} + \frac{1}{n_1/\lambda_i^{(1)}+p-n_1}\right]:=\psi(n_1).\] Note by straightforward calculation, \(\psi(0)=\psi(p/2) = 1\), \(\psi'(p/2)>0\), and \(\psi''(x)>0\) for all \(0 \leq x \leq p\). Therefore, we have \(\tilde{a}_1 \leq \frac{n_1}{p-n}\) if and only if \(n_1 \leq p/2\).

13.5 Proof of Proposition 9↩︎

Here, we want to study the effect of \(\kappa\) in the quantity \(\hat{R}(\boldsymbol{\Sigma}(\kappa))\).

To analyze the quantity \(\hat{R}(\boldsymbol{\Sigma}(\kappa))\), first note that, by the proof of Proposition 8, the bias stays identical for any \(\kappa\) and the variance is \(\sigma^2(\tilde{a}_1+ \tilde{a}_2)\), where \(\tilde{a}_1,\tilde{a}_2\) are defined by ?? . Moreover, again by proof of Proposition 8, one has \(\tilde{a}_2=\frac{n_2}{p-n_2}\). Hence, we are left with analyzing \(\tilde{a}_1\) given by 100 which simplifies here as

\[\begin{align} &\frac{2}{p}= \left[ \frac{1}{\tilde{a}_1(p-n)\kappa+p-n_1} + \frac{1}{\tilde{a}_1(p-n)/\kappa+p-n_1}\right]\\ \implies& \tilde{a}^2_1 \frac{2(p-n)^2}{p}+ \frac{2(p-n_1)^2}{p}+\tilde{a}_1 \frac{2(p-n)(p-n_1)}{p} (\kappa+\frac{1}{\kappa}) \\ &= \tilde{a}_1 (p-n) (\kappa+\frac{1}{\kappa}) +2(p-n_1)\\ \implies& 2\tilde{a}_1^2 (p-n)^2+ \tilde{a}_1 (p-n)(p-2n_1)(\kappa+\frac{1}{\kappa}) - 2n_1(p-n_1) \\ &= 0 \end{align}\] If \(n_1 <p/2\), the linear term in the above quadratic equation is positive, implying \(\tilde{a}_1\) is smaller for larger values of \((\kappa+\frac{1}{\kappa})\). Hence, \(\tilde{a}_1\) is a decreasing function in \(\kappa\), implying \(\hat{R}(\boldsymbol{\Sigma}(\kappa))\) decreases as \(\kappa\) increases.

Similarly, if \(n_1 <p/2\), the linear term in the above quadratic equation is negative, and hence \(\hat{R}(\boldsymbol{\Sigma}(\kappa))\) increases as \(\kappa\) increases.

Finally, if \(n_1=p/2\), the linear term vanishes leading to the fact that \(\hat{R}(\boldsymbol{\Sigma}(\kappa))\) is agnostic of \(\kappa\).

14 Proof for Universality↩︎

In this theorem, we prove Theorem 11.

Continuing from the notation introduced in Theorem 11, we write the bias components as functions of the underlying whitened covariates. Specifically, for \(j = 1,2,3\), we denote by \(B_{\lambda,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) denote the respective term in ?? computed from \(\boldsymbol{X}^{(k)}=\boldsymbol{Z}^{(k)}(\boldsymbol{\Sigma}^{(k)})^{1/2}\), \(k=1,2\). More generally, for \(\boldsymbol{W}^{(1)} \in \{\boldsymbol{Z}^{(1)}, \tilde{\boldsymbol{Z}}^{(1)}\}\) and \(\boldsymbol{W}^{(2)} \in \{\boldsymbol{Z}^{(2)}, \tilde{\boldsymbol{Z}}^{(2)}\}\), we let \(B_{\lambda, j}(\boldsymbol{W}^{(1)},\boldsymbol{W}^{(2)})\) denote the same bias term evaluated by replacing \(\boldsymbol{Z}^{(k)}\) by \(\boldsymbol{W}^{(k)}\) for \(k = 1,2\). We suppress the dependence of these quantities on the ridge parameter \(\lambda\) whenever it is clear from context. Additionally, with slight abuse of notation, we denote by \(B_{0,j}(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)})\) the analogous ridgeless quantity in ?? . Finally, the variance terms \(\boldsymbol{V}_{\lambda}(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)})\) and \(\boldsymbol{V}_{0}(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)})\) are defined analogously, using ?? and ?? .

Our proof of universality uses the Lindeberg swapping method, where we individually swap each row of \(\boldsymbol{Z}\) with the corresponding row of \(\tilde{\boldsymbol{Z}}\) and bound the resulting change in the out-of-sample prediction risk. To bound this difference, we bound the derivative of the matrix functional along a Gaussian interpolation path.

Using the arguments from [23], [35], it suffices to show that, for \(j = 1,2,3\), \[|\mathbb{E}[\varphi(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \to 0\] for any bounded function \(\varphi\) with bounded first and second derivatives.

We begin by proving the following result, which we later use to bound the analogous ridgeless quantities.

We begin by proving the result for the bias terms and will later conclude with a similar proof for the variance term.

Let \(\boldsymbol{X}^{(i),k} \in \mathbb{R}^{n_i \times p}\) be the matrix whose first \(k\) rows match the first \(k\) rows of \(\tilde{\boldsymbol{X}}^{(i)}\) and whose last \(n_i - k\) rows match those of \(\boldsymbol{X}^{(i)}\). Then, let \(\boldsymbol{X}^{(i),\setminus k} \in \mathbb{R}^{(n_i - 1) \times p}\) be equal to the matrix \(\boldsymbol{X}^{(i),k}\) with the \(k\)th row removed. Note that \(\boldsymbol{X}^{(i),\setminus k}\) is also equal to the matrix \(\boldsymbol{X}^{(i),k-1}\) with the \(k\)th row removed. Then, we can define \[\begin{gather} \hat{\boldsymbol{\Sigma}}_k = \frac{1}{n} \boldsymbol{X}^{(1),k\top} \boldsymbol{X}^{(1),k} + \frac{1}{n} \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)} \\ \hat{\boldsymbol{\Sigma}}_{\setminus k} = \frac{1}{n} \boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k} + \frac{1}{n} \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)} \\ \tilde{\boldsymbol{\Sigma}}_{k} = \frac{1}{n} \tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} + \frac{1}{n} \boldsymbol{X}^{(2),k\top} \boldsymbol{X}^{(2),k} \\ \tilde{\boldsymbol{\Sigma}}_{\setminus k} = \frac{1}{n} \tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} + \frac{1}{n} \boldsymbol{X}^{(2),\setminus k\top} \boldsymbol{X}^{(2),\setminus k} \end{gather}\] For \(k = 1, \dots, n_1\), an application of Taylor’s theorem yields \[\begin{align} \varphi(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})) &= \varphi(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) \\ &\qquad + \varphi'(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) (B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) \\ &\qquad + \frac{1}{2}\varphi''(\tilde{B}^{(1),k})(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2. \end{align}\] where \(\tilde{B}^{(1),k}\) is some value between \(B_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})\) and \(B_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\). An analogous application yields \[\begin{align} \varphi(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})) &= \varphi(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) \\ &\quad + \varphi'(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) (B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})) \\ &\quad + \frac{1}{2}\varphi''(\dot{B}^{(1),k})(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2. \end{align}\] for some value \(\dot{B}^{(1),k}\) between \(B_1(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})\) and \(B_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\). Subtracting the first expression from the latter expression then yields \[\begin{align} \varphi(B_j(\boldsymbol{Z}^{(1),k},& \boldsymbol{Z}^{(2)})) - \varphi(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})) \\ &= \varphi'(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})) \\ &\qquad + \frac{1}{2}\varphi''(\tilde{B}^{(1),k})(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2 \\ &\qquad - \frac{1}{2}\varphi''(\dot{B}^{(1),k})(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2. \end{align}\] The analogous procedure yields \[\begin{align} \varphi(B_j(\tilde{\boldsymbol{Z}},& \boldsymbol{Z}^{(2),k})) - \varphi(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),k-1})) \\ &= \varphi'(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),k-1})) \\ &\qquad + \frac{1}{2}\varphi''(\tilde{B}^{(2),k})(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2 \\ &\qquad - \frac{1}{2}\varphi''(\dot{B}^{(2),k})(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2. \end{align}\] for each \(k = 1,\dots, n_2\). Importantly, for each \(k = 1, \dots, n_1\), the quantity \(B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\) is independent of \(\boldsymbol{Z}^{(1),k}, \tilde{\boldsymbol{Z}}^{(1),k}\). Similarly, for each \(k = 1, \dots, n_2\), the quantity \(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k})\) is independent of \(\boldsymbol{Z}^{(2),k}, \tilde{\boldsymbol{Z}}^{(2),k}\). Therefore, we have \[\begin{align} &\varphi(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})) - \varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})) \\&= \sum_{k=1}^{n_1} [\varphi(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})) - \varphi(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}))] \\ &\varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})) - \varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})) \\ &= \sum_{k=1}^{n_2} [\varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k})) - \varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}))] \end{align}\] Therefore, letting \(\mathbb{E}_{1,k}\) be expectation with respect to \(\boldsymbol{X}^{(1),k}, \tilde{\boldsymbol{X}}^{(1),k}\) and letting \(\mathbb{E}_{2,k}\) be expectation with respect to \(\boldsymbol{X}^{(2),k}, \tilde{\boldsymbol{X}}^{(2),k}\), we can apply the triangle inequality and the above reasoning to yield \[\begin{align} &\bigl|\mathbb{E}\bigl[\varphi\bigl(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\bigr)\bigr] - \mathbb{E}\bigl[\varphi\bigl(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})\bigr)\bigr] \bigr| \\ &\le \sum_{k=1}^{n_1} \bigl| \mathbb{E}\bigl[\varphi\bigl(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})\bigr)\bigr] - \mathbb{E}\bigl[\varphi\bigl(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})\bigr)\bigr] \bigr| \\ &\le \|\varphi'\|_\infty \sum_{k=1}^{n_1} \mathbb{E}\Bigl[\bigl|\mathbb{E}_{1,k}\bigl[B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})-B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})\bigr] \bigr|\Bigr] \\ &\quad + \frac{\|\varphi''\|_\infty}{2} \sum_{k=1}^{n_1} \mathbb{E}\Bigl[\mathbb{E}_{1,k}\bigl[\bigl(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)})-B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\bigr)^2\bigr]\Bigr] \\ &\quad + \frac{\|\varphi''\|_\infty}{2} \sum_{k=1}^{n_1} \mathbb{E}\Bigl[ \mathbb{E}_{1,k}\bigl[ \bigl( B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \bigr)^2 \bigr] \Bigr]. \end{align}\] and similarly \[\begin{align} &\bigl| \mathbb{E}\bigl[\varphi\bigl(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})\bigr)\bigr] - \mathbb{E}\bigl[\varphi\bigl(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})\bigr)\bigr] \bigr| \\ &\le \sum_{k=1}^{n_2} \bigl| \mathbb{E}\bigl[\varphi\bigl(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k})\bigr)\bigr] - \mathbb{E}\bigl[\varphi\bigl(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1})\bigr)\bigr] \bigr| \\ &\le \|\varphi'\|_\infty \sum_{k=1}^{n_2} \mathbb{E}\Bigl[ \bigl| \mathbb{E}_{2,k}\bigl[ B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) \bigr] \bigr| \Bigr] \\ &\quad + \frac{\|\varphi''\|_\infty}{2} \sum_{k=1}^{n_2} \mathbb{E}\Bigl[ \mathbb{E}_{2,k}\bigl[ \bigl( B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \bigr)^2 \bigr] \Bigr] \\ &\quad + \frac{\|\varphi''\|_\infty}{2} \sum_{k=1}^{n_2} \mathbb{E}\Bigl[ \mathbb{E}_{2,k}\bigl[ \bigl( B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \bigr)^2 \bigr] \Bigr]. \end{align}\] If both of these terms converge to zero, then the triangle inequality immediately implies that the term \(|\mathbb{E}[\varphi(B_j(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]|\) converges to zero, which yields the desired universality result.

14.1 Reduction via Sherman-Morrison↩︎

Now, we can expand each of the terms from above. We will show that bounding these differences reduces to controlling the behavior of eight relatively simple terms.

By Sherman-Morrison, we can express \[\begin{align} (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} &= (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \tfrac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} + \lambda \boldsymbol{I})^{-1} \\ &= (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}. \end{align}\]

14.1.1 Expanding \(B_1\) terms↩︎

Recall that \[B_1(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}) = \lambda^2 \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}\] and therefore we can express \[\begin{align} &B_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) \\&=\lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr)^{-1} \\ &\qquad \boldsymbol{\Sigma}^{(2)}\biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\beta}^{(2)} \end{align}\] from which we get \[\begin{align} & B_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\\ &= -2\lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k} \biggr) \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ &\qquad + \lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k} \biggr) \boldsymbol{\Sigma}^{(2)} \\ &\qquad \qquad \cdot \biggl( \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k} \biggr) \boldsymbol{\beta}^{(2)} \\ &:= \frac{\boldsymbol{u}^{(a,1)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(a,1)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(a,1)} \tilde{\boldsymbol{Z}}_k^{(1)}} + \frac{\boldsymbol{u}^{(a,2)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(a,2)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(a,2)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(a,2)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] and similarly \[\begin{align} & B_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)})\\ &:= \frac{\boldsymbol{u}^{(a,1)\top} \boldsymbol{Z}_k^{(1)} \boldsymbol{Z}_k^{(1)\top} \boldsymbol{v}^{(a,1)}}{1 + \boldsymbol{Z}_k^{(1)\top} \boldsymbol{A}^{(a,1)} \boldsymbol{Z}_k^{(1)}} + \frac{\boldsymbol{u}^{(a,2)\top} \boldsymbol{Z}_k^{(1)} (\boldsymbol{Z}_k^{(1)\top} \boldsymbol{A}^{(a,2)} \boldsymbol{Z}_k^{(1)}) \boldsymbol{Z}_k^{(1)\top} \boldsymbol{v}^{(a,2)}}{(1 + \boldsymbol{Z}_k^{(1)\top} \boldsymbol{B}^{(a,2)} \boldsymbol{Z}_k^{(1)})^2} \end{align}\] for vectors \(\boldsymbol{u}^{(a,1)}\), \(\boldsymbol{v}^{(a,1)}\), \(\boldsymbol{u}^{(a,2)}\), \(\boldsymbol{v}^{(a,2)}\) and matrices \(\boldsymbol{A}^{(a,1)}\), \(\boldsymbol{A}^{(a,2)}\), \(\boldsymbol{A}^{(a,3)}\) that are all independent of \(\boldsymbol{Z}_k^{(1)}, \tilde{\boldsymbol{Z}}_k^{(1)}\). Specifically, we have \[\begin{gather} \boldsymbol{u}^{(a,1)} = -\frac{2\lambda^2}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(a,1)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{A}^{(a,1)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(a,2)} = \frac{\lambda^2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(a,2)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{A}^{(a,2)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(a,2)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \end{gather}\] We similarly have \[\begin{align} &B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) \\&=\lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl((\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr)^{-1} \boldsymbol{\Sigma}^{(2)} \\ &\qquad \biggl((\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\beta}^{(2)} \end{align}\] from which we get \[\begin{align} &B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \\ &= - 2 \lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl( \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ &\qquad + \lambda^2 \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\Sigma}^{(2)} \\ &\qquad \qquad \biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\beta}^{(2)} \\ &:= \frac{\boldsymbol{u}^{(A,1)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(A,1)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(A,1)} \tilde{\boldsymbol{Z}}_k^{(2)}} + \frac{\boldsymbol{u}^{(A,2)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(A,2)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(A,2)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(A,2)} \tilde{\boldsymbol{Z}}_k^{(2)})^2} \\ \end{align}\] and similarly \[\begin{align} &B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k})\\ &:= \frac{\boldsymbol{u}^{(A,1)\top} \boldsymbol{Z}_k^{(2)} \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(A,1)}}{1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(A,1)} \boldsymbol{Z}_k^{(2)}} + \frac{\boldsymbol{u}^{(A,2)\top} \boldsymbol{Z}_k^{(2)} (\boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(A,2)} \boldsymbol{Z}_k^{(2)}) \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(A,2)}}{(1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{B}^{(A,2)} \boldsymbol{Z}_k^{(2)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(A,1)} = -\frac{2\lambda^2}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\beta}^{(2)}, \\ \boldsymbol{v}^{(A,1)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{A}^{(A,1)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(A,2)} = \frac{\lambda^2}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(A,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{A}^{(A,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(A,2)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \end{gather}\]

14.1.2 Expanding \(B_3\) terms↩︎

Now, recall that \[B_3(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}) = - 2\lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}}\] Therefore, using the result from earlier, \[\begin{align} &B_3(\boldsymbol{Z}^{(1), k}, \boldsymbol{Z}^{(2)}) \\&= -2 \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}_{k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{k} + \lambda \boldsymbol{I})^{-1}\biggl(\frac{\boldsymbol{X}^{(1), k\top}\boldsymbol{X}^{(1), k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &=-2\lambda \boldsymbol{\beta}^{(2)\top} \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \\ &\qquad \cdot \boldsymbol{\Sigma}^{(2)} \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \\ &\qquad \cdot \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k}{n} \biggr)\tilde{\boldsymbol{\beta}} \end{align}\] and therefore \[\begin{align} &B_3(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_3( \boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &= -2 \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2 \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr)\biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2 \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} \\ &\biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr)\biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2 \lambda \boldsymbol{\beta}^{(2)\top}\biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ &(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2 \lambda \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2 \lambda \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2 \lambda \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &:= \boldsymbol{u}^{(b,1)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,1)} + \frac{\boldsymbol{u}^{(b,2)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,2)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,2)} \tilde{\boldsymbol{Z}}_k^{(1)}} ]] \\ &\qquad + \frac{\boldsymbol{u}^{(b,3)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,3)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,3)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,3)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(b,4)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,4)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,4)} \tilde{\boldsymbol{Z}}_k^{(1)}} + \frac{\boldsymbol{u}^{(b,5)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,5)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,5)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,5)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(b,6)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,6)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,6)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,6)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(b,7)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,7)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,7)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,7)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(b,7)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] and similarly \[\begin{align} &B_3(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_3( \boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &:= \boldsymbol{u}^{(b,1)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,1)} + \frac{\boldsymbol{u}^{(b,2)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,2)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,2)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(b,3)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,3)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,3)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,3)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(b,4)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,4)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,4)} {\boldsymbol{Z}}_k^{(1)}} + \frac{\boldsymbol{u}^{(b,5)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,5)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,5)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,5)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(b,6)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,6)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,6)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,6)} {\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(b,7)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(b,7)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(b,7)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(b,7)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(b,7)} {\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(b,1)} = -\frac{2\lambda}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(b,1)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{u}^{(b,2)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}\\ \boldsymbol{v}^{(b,2)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1), \setminus k\top} \boldsymbol{X}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,2)}= \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2}(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(b,3)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(b,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}, \quad \boldsymbol{B}^{(b,3)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(b,4)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(b,4)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,4)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(b,5)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(b,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(b,5)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(b,6)} = -\frac{2\lambda}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(b,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(b,6)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(b,7)} = -\frac{2\lambda}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}, \quad \boldsymbol{v}^{(b,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(b,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(b,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{C}^{(b,7)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \end{gather}\] are all independent of \(\boldsymbol{Z}_k^{(1)}, \tilde{\boldsymbol{Z}}_k^{(1)}\).

Similarly, we can express \[\begin{align} &B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) \\&= -2 \lambda \boldsymbol{\beta}^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{k} + \lambda \boldsymbol{I})^{-1}\biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top}\tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &=-2\lambda \boldsymbol{\beta}^{(2)\top} \biggl((\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}} \biggr) \boldsymbol{\Sigma}^{(2)} \\ &\qquad \cdot \biggl((\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}} \biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr)\tilde{\boldsymbol{\beta}} \end{align}\] and therefore \[\begin{align} &B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \\ &= 2\lambda \boldsymbol{\beta}^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} \\ &\biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}} \biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2 \lambda \boldsymbol{\beta}^{(2)\top} \biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}} \biggr) \boldsymbol{\Sigma}^{(2)} \\ &(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2 \lambda \boldsymbol{\beta}^{(2)\top} \biggl( \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr) \\ &\boldsymbol{\Sigma}^{(2)} \biggl( \frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &:= \frac{\boldsymbol{u}^{(B,1)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(B,1)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(B,1)} \tilde{\boldsymbol{Z}}_k^{(2)}} + \frac{\boldsymbol{u}^{(B,2)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(B,2)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(B,2)} \tilde{\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(B,3)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(B,3)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(B,3)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(B,3)} \tilde{\boldsymbol{Z}}_k^{(2)})^2} \end{align}\] and similarly \[\begin{align} &B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \\ &= \frac{\boldsymbol{u}^{(B,1)\top} \boldsymbol{Z}_k^{(2)} \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(B,1)}}{1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(B,1)} \boldsymbol{Z}_k^{(2)}} + \frac{\boldsymbol{u}^{(B,2)\top} \boldsymbol{Z}_k^{(2)} \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(B,2)}}{1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(B,2)} \boldsymbol{Z}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(B,3)\top} \boldsymbol{Z}_k^{(2)} (\boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(B,3)} \boldsymbol{Z}_k^{(2)}) \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(B,3)}}{(1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{B}^{(B,3)} \boldsymbol{Z}_k^{(2)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(B,1)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(B,1)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(B,1)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(B,2)} = \frac{2\lambda}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(B,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2}(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(B,2)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(B,3)} = -\frac{2\lambda}{n^3} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ \boldsymbol{v}^{(B,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2}(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(B,1)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1}(\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(B,1)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda\boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \end{gather}\]

14.1.3 Expanding \(B_2\) terms↩︎

Now, recall that \[B_2(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}) = \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n} \biggr) \tilde{\boldsymbol{\beta}}\] an using the result from earlier, we have \[\begin{align} &B_2(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) \\&= \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\boldsymbol{X}^{(1),k\top} \boldsymbol{X}^{(1),k}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(1),k\top} \boldsymbol{X}^{(1),k}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &= \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \\ &\qquad \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \boldsymbol{\Sigma}^{(2)} \\ &\qquad \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \\ &\qquad \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}} \end{align}\] Therefore, we finally have \[\begin{align} &B_2(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_2( \boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &= 2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} \\ &\qquad \qquad \cdot \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr)\biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} \\ &\qquad \qquad \cdot \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \\ &\qquad \qquad \cdot \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + 2\tilde{\boldsymbol{\beta}}^\top\biggl(\frac{\tilde{\boldsymbol{X}}^{(1),\setminus k\top } \tilde{\boldsymbol{X}}^{(1), \setminus k}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n} \biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad - 2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)}\\ & (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + \tilde{\boldsymbol{\beta}}^\top\biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \boldsymbol{\Sigma}^{(2)} \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n}\tilde{\boldsymbol{X}}^{(1)\top}_k (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}^{(1)}_k}\biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)}_k \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &:= \boldsymbol{u}^{(c,1)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,1)} + \frac{\boldsymbol{u}^{(c,2)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,2)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,2)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(c,3)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,3)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,3)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,3)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(c,4)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,4)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,4)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,4)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad+ \frac{\boldsymbol{u}^{(c,5)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,5)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,5)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,5)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(c,6)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,6)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,6)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,6)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,6)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \boldsymbol{u}^{(c,7)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,7)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,7)} \\ &\qquad + \frac{(\boldsymbol{u}^{(c,8)\top} \tilde{\boldsymbol{Z}}_k^{(1)})^2 (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,8)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,6)} \tilde{\boldsymbol{Z}}_k^{(1)})}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,8)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{(\boldsymbol{u}^{(c,9)\top} \tilde{\boldsymbol{Z}}_k^{(1)})^2 (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,9)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,9)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,9)} \tilde{\boldsymbol{Z}}_k^{(1)})}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{D}^{(c,9)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] and similarly \[\begin{align} &B_2(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_2( \boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &:= \boldsymbol{u}^{(c,1)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,1)} + \frac{\boldsymbol{u}^{(c,2)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,2)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,2)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(c,3)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,3)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,3)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,3)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(c,4)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,4)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,4)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,4)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad+ \frac{\boldsymbol{u}^{(c,5)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,5)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,5)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,5)} {\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(c,6)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,6)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,6)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,6)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,6)} {\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \boldsymbol{u}^{(c,7)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,7)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(c,7)} \\ &\qquad + \frac{(\boldsymbol{u}^{(c,8)\top} {\boldsymbol{Z}}_k^{(1)})^2 ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,8)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,6)} {\boldsymbol{Z}}_k^{(1)})}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,8)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{(\boldsymbol{u}^{(c,9)\top} {\boldsymbol{Z}}_k^{(1)})^2 ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(c,9)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(c,9)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(c,9)} {\boldsymbol{Z}}_k^{(1)})}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{D}^{(c,9)} {\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(c,1)} = \frac{2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{v}^{(c,1)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{u}^{(c,2)} = -\frac{2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{v}^{(c,2)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1), \setminus k\top} \boldsymbol{X}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,2)}= \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2}(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}, \boldsymbol{v}^{(c,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{u}^{(c,3)} = -\frac{2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}, \quad \boldsymbol{B}^{(c,3)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,4)} = -\frac{2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}, \quad \boldsymbol{v}^{(c,4)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,4)} = (\boldsymbol{\Sigma}^{(1)})^{1/2}(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(c,4)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,5)} = \frac{1}{n^4} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{v}^{(c,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(c,5)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,6)} = \frac{2}{n^4} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1),\setminus k\top} \tilde{\boldsymbol{X}}^{(1),\setminus k}) \tilde{\boldsymbol{\beta}}, \quad \boldsymbol{v}^{(c,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(c,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{C}^{(c,6)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,7)} = \frac{1}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2}\tilde{\boldsymbol{\beta}}, \quad \boldsymbol{v}^{(c,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2}\tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(c,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,8)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}, \quad \boldsymbol{A}^{(c,8)} = -\frac{2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(c,8)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{C}^{(c,8)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(c,9)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \tilde{\boldsymbol{\beta}}, \quad \boldsymbol{A}^{(c,9)} = \frac{1}{n^4} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(c,9)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{C}^{(c,9)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{D}^{(c,9)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \end{gather}\] Similarly, we have \[\begin{align} &B_2(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) \\ &= \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &= \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \boldsymbol{\Sigma}^{(2)} \\ &\qquad \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr) \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}} \end{align}\] and therefore \[\begin{align} &B_2(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_2(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) \\ &= -2\tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr) \\ &\boldsymbol{\Sigma}^{(2)} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &\qquad + \tilde{\boldsymbol{\beta}}^\top \biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr) \boldsymbol{\Sigma}^{(2)} \\ &\biggl(\frac{1}{n} \frac{(\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(2)\top} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(2)}}\biggr)\biggl(\frac{\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}}{n}\biggr) \tilde{\boldsymbol{\beta}}\\ &:= \frac{\boldsymbol{u}^{(C,1)\top} \boldsymbol{Z}_k^{(2)} \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(C,1)}}{1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(C,1)} \boldsymbol{Z}_k^{(2)}} + \frac{\boldsymbol{u}^{(C,2)\top} \boldsymbol{Z}_k^{(2)} (\boldsymbol{Z}_k^{(2)\top} \boldsymbol{A}^{(C,2)} \boldsymbol{Z}_k^{(2)}) \boldsymbol{Z}_k^{(2)\top} \boldsymbol{v}^{(C,2)}}{(1 + \boldsymbol{Z}_k^{(2)\top} \boldsymbol{B}^{(C,2)} \boldsymbol{Z}_k^{(2)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(C,1)} = -\frac{2}{n^3} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{v}^{(C,1)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}\boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} )\tilde{\boldsymbol{\beta}}\\ \boldsymbol{A}^{(C,1)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(C,2)} = \frac{1}{n^4} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \tilde{\boldsymbol{\beta}}\\ \boldsymbol{v}^{(C,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)}) \boldsymbol{\beta}^{(2)} \\ \boldsymbol{A}^{(C,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(C,2)} = \frac{1}{n}(\boldsymbol{\Sigma}^{(2)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \end{gather}\]

14.1.4 Common terms↩︎

For fixed vectors \(\boldsymbol{u}\), \(\boldsymbol{v}\) and fixed PSD matrices \(\boldsymbol{A}\), \(\boldsymbol{B}\), \(\boldsymbol{C}\), and \(\boldsymbol{D}\), let \(\boldsymbol{S} = \{\boldsymbol{u}, \boldsymbol{v}, \boldsymbol{A}, \boldsymbol{B}, \boldsymbol{C}, \boldsymbol{D}\}\) and consider the scalar-valued functions \[\begin{align} f_1(\boldsymbol{w}; \boldsymbol{S}) &= \boldsymbol{u}^\top \boldsymbol{w} \boldsymbol{w}^\top \boldsymbol{v}, \\ f_2(\boldsymbol{w}; \boldsymbol{S}) &= \frac{\boldsymbol{u}^\top \boldsymbol{w} \boldsymbol{w}^\top \boldsymbol{v}}{1 + \boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}}, \\ f_3(\boldsymbol{w}; \boldsymbol{S}) &= \boldsymbol{u}^\top \boldsymbol{w}(\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) \boldsymbol{w}^\top \boldsymbol{v}, \\ f_4(\boldsymbol{w}; \boldsymbol{S}) &= \frac{\boldsymbol{u}^\top \boldsymbol{w} (\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) \boldsymbol{w}^\top \boldsymbol{v}}{1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}} \\ f_5(\boldsymbol{w}; \boldsymbol{S}) &= \frac{\boldsymbol{u}^\top \boldsymbol{w} (\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) \boldsymbol{w}^\top \boldsymbol{v}}{(1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2}, \\ f_6(\boldsymbol{w}; \boldsymbol{S})&= \frac{(\boldsymbol{u}^\top \boldsymbol{w})^2 (\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) (\boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})}{1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}} \\ f_7(\boldsymbol{w}; \boldsymbol{S}) &= \frac{\boldsymbol{u}^\top \boldsymbol{w} (\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) (\boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{w}^\top \boldsymbol{v}}{(1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2}, \\ f_8(\boldsymbol{w}; \boldsymbol{S}) &= \frac{(\boldsymbol{u}^\top \boldsymbol{w})^2 (\boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}) (\boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})}{(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2}. \end{align}\]

From the above expansions, we see that, for each \(j = 1,2,3\):

• There are some \(m_j^{(1)} \in \mathbb{Z}_{> 0}\), some \(i_{j,1}^{(1)}, \dots, i_{j,m_j^{(1)}}^{(1)} \in \mathbb{Z}_{> 0}\), and some collections of vectors and matrices \(\boldsymbol{S}_{j,1}^{(1)}, \dots, \boldsymbol{S}_{j,m_j^{(1)}}^{(1)}\) that are independent of \(\boldsymbol{Z}_k^{(1)}, \tilde{\boldsymbol{Z}}_k^{(1)}\) such that \[\begin{align} B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) &= \sum_{\ell=1}^{m_j^{(1)}} f_{i_{j,\ell}^{(1)}}(\tilde{\boldsymbol{Z}}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)}) \\ B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) &= \sum_{\ell=1}^{m_j^{(1)}} f_{i_{j,\ell}^{(1)}}(\boldsymbol{Z}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)}) \end{align}\]

• There are some \(m_j^{(2)} \in \mathbb{Z}_{> 0}\), some \(i_{j,1}^{(2)}, \dots, i_{j,m_j^{(2)}}^{(2)} \in \mathbb{Z}_{> 0}\), and some collections of vectors and matrices \(\boldsymbol{S}_{j,1}^{(2)}, \dots, \boldsymbol{S}_{j,m_j^{(2)}}^{(2)}\) that are independent of \(\boldsymbol{Z}_k^{(2)}, \tilde{\boldsymbol{Z}}_k^{(2)}\) such that \[\begin{align} B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) &= \sum_{\ell=1}^{m_j^{(2)}} f_{i_{j,\ell}^{(2)}}(\tilde{\boldsymbol{Z}}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)}) \\ B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}) &= \sum_{\ell=1}^{m_j^{(2)}} f_{i_{j,\ell}^{(2)}}(\boldsymbol{Z}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)}) \end{align}\]

Therefore, by Cauchy-Schwarz, for each \(j = 1,2,3\):

• We can bound \[\begin{align} \mathbb{E}_{1,k}[(B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2] &\leq m_j^{(1)} \sum_{\ell=1}^{m_j^{(1)}} \mathbb{E}_{1,k}[f_{i_{j,\ell}^{(1)}}(\tilde{\boldsymbol{Z}}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)})^2] \\ \mathbb{E}_{1,k}[(B_j(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2] &\leq m_j^{(1)} \sum_{\ell=1}^{m_j^{(1)}} \mathbb{E}_{1,k}[f_{i_{j,\ell}^{(1)}}(\boldsymbol{Z}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)})^2] \\ \mathbb{E}_{2,k}[(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2] &\leq m_j^{(2)} \sum_{\ell=1}^{m_j^{(2)}} \mathbb{E}_{2,k}[f_{i_{j,\ell}^{(2)}}(\tilde{\boldsymbol{Z}}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)})^2] \\ \mathbb{E}_{2,k}[(B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2] &\leq m_j^{(2)} \sum_{\ell=1}^{m_j^{(2)}} f_{i_{j,\ell}^{(2)}}\mathbb{E}_{2,k}[(\boldsymbol{Z}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)})^2] \end{align}\]

• We can bound \[\begin{align} &|\mathbb{E}_{1,k}[B_j(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - B_j(\boldsymbol{Z}^{(1), k-1}, \boldsymbol{Z}^{(2)})]| \\&\leq \sum_{\ell=1}^{m_j^{(1)}} |\mathbb{E}_{1,k}[f_{i_{j,\ell}^{(1)}}(\tilde{\boldsymbol{Z}}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)}) - f_{i_{j,\ell}^{(1)}}(\boldsymbol{Z}^{(1)}_k; \boldsymbol{S}_{j,\ell}^{(1)})]| \\ &|\mathbb{E}_{2,k}[B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - B_j(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2), k-1})]| \\&\leq \sum_{\ell=1}^{m_j^{(2)}} |\mathbb{E}_{2,k}[f_{i_{j,\ell}^{(2)}}(\tilde{\boldsymbol{Z}}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)}) - f_{i_{j,\ell}^{(2)}}(\boldsymbol{Z}^{(2)}_k; \boldsymbol{S}_{j,\ell}^{(2)})]| \end{align}\]

Therefore, proving our universality result reduces to bounding terms of the form \[\begin{gather} \mathbb{E}_{i,k}[f_\ell(\tilde{\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S})^2], \quad \mathbb{E}_{i,k}[f_\ell({\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S})^2], \quad |\mathbb{E}_{i,k}[f_\ell(\tilde{\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S}) - f_\ell(\boldsymbol{Z}_k^{(i)}; \boldsymbol{S})]| \end{gather}\] for \(\ell = 1,\dots, 8\), \(i = 1,2\), and any \(\boldsymbol{S}\) (with the bounds depending on \(\boldsymbol{S}\)).

14.2 Second moment bounds↩︎

In this section, we bound the second moment terms \(\mathbb{E}_{i,k}[f_\ell(\tilde{\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S})^2]\) and \(\mathbb{E}_{i,k}[f_\ell({\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S})^2]\) for \(\ell = 1, \dots, 8\), \(i = 1,2\), and any \(\boldsymbol{S}\).

We repeatedly apply Hölder’s inequality to yield the desired bounds. In bounding many of the terms, we also make use of the fact that \(\boldsymbol{A}, \boldsymbol{B}, \boldsymbol{C}, \boldsymbol{D}\) are PSD by assumption and thus \[0 < \frac{1}{1 + \boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}}, \frac{1}{1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}}, \frac{1}{1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}}, \frac{1}{1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}} \leq 1\] for all \(\boldsymbol{w} \in \mathbb{R}^p\).

Note that the bounds in this lemma directly translate to bounds on \(\mathbb{E}_{i,k}[f_\ell(\tilde{\boldsymbol{Z}}_k^{(i)}; \boldsymbol{S})^2]\) and \(\mathbb{E}_{i,k}[f_\ell(\boldsymbol{Z}_k^{(i)}; \boldsymbol{S})^2]\) for \(\ell = 1, \dots, 8\) and \(i = 1,2\) and any \(\boldsymbol{S}\) because we only make use of (a) matching first and second moments and (b) the mere existence of all other moments.

14.3 First moment bounds↩︎

In this section, we bound \(|\mathbb{E}_{i,k}[f_\ell(\tilde{\boldsymbol{Z}}^{(i)}_k; \boldsymbol{S}) - f_\ell(\boldsymbol{Z}^{(i)}_k; \boldsymbol{S}))]|\) for \(\ell = 1, \dots, 8\), \(i = 1,2\), and any \(\boldsymbol{S}\). As long as our proof only makes use of (a) matching first and second moments between \(\tilde{\boldsymbol{W}}\) and \(\boldsymbol{W}\), (b) the mere existence of all other moments of \(\boldsymbol{W}\), and (c) the assumption \(\tilde{\boldsymbol{W}} \sim \mathcal{N}(0, \boldsymbol{I})\), it suffices to consider bounding \[|\mathbb{E}_k[f_\ell(\tilde{\boldsymbol{W}}; \boldsymbol{S}) - f_\ell(\boldsymbol{W}; \boldsymbol{S})]|\] for \(\ell = 1, \dots, 8\) and any fixed \(\boldsymbol{S}\), where \(\tilde{\boldsymbol{W}} \sim \mathcal{N}(0, \boldsymbol{I})\) and where \(\boldsymbol{W}\) is a random \(\mathbb{R}^p\)-valued random variables consisting of i.i.d. entries with \(\mathbb{E}[W_1] = \mathbb{E}[\tilde{W}_1] = 0\) and \(\textrm{Var}(W_1) = \textrm{Var}(\tilde{W}_1) = 1\). Importantly, we do not assume that \(W_1 \overset{d}{=} \tilde{W}_1\) or that \(\mathbb{E}[W_1^m] = \mathbb{E}[\tilde{W}_1^m]\) for any \(m > 2\). To further simplify our notation, we omit \(\boldsymbol{S}\) when the dependence is clear. Therefore, the goal of this section is to bound \[|\mathbb{E}_k[f_\ell(\tilde{\boldsymbol{W}}) - f_\ell(\boldsymbol{W})]|\] in terms of \(\boldsymbol{S}\), for each \(\ell = 1, \dots, 8\).

At a high level, our strategy for bounding these terms involves interpolating between \(\tilde{\boldsymbol{W}}\) and \(\boldsymbol{W}\) and bounding the change in the value of our expression along the curve. To bound the value of the expression, we perform multiple Taylor expansions and invoke Stein’s identity to simplify the resulting expansions.

The following statement is the basis of our interpolation strategy, which we use bound the terms resulting from the Sherman-Morrison decompositions:

Therefore, with this lemma, it suffices to bound \[\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \cdot \sup_{u \in (0,1)} |\partial_i^3 f_\ell(\boldsymbol{W}_{t,u,i})| \biggr]\] for \(r = 1,3\), \(\ell = 1, \dots, 8\), and \(t \in (0,1)\). We now embark on these bounds.

In bounding terms of the first kind, we prove analogous bounds to Lemma 8 for \(\boldsymbol{W}_t\) and \(\boldsymbol{W}_{t,u,i}\):

We now begin bounding each \(f_1, \dots, f_8\). We prove the following bounds:

Note that the first function \(f_1\) can be handled directly because its expectation depends only on the first and second moments of \(\boldsymbol{W}\) and \(\tilde{\boldsymbol{W}}\). That is, \[\begin{align} \mathbb{E}[f_1(\tilde{\boldsymbol{W}}) - f_1(\boldsymbol{W})] = \boldsymbol{u}^\top (\mathbb{E}[\tilde{\boldsymbol{W}} \tilde{\boldsymbol{W}}^\top] - \mathbb{E}[\boldsymbol{W}\boldsymbol{W}^\top]) \boldsymbol{v} = 0 \end{align}\] because \(\mathbb{E}[\tilde{\boldsymbol{W}} \tilde{\boldsymbol{W}}^\top] = \mathbb{E}[\boldsymbol{W} \boldsymbol{W}^\top] = \boldsymbol{I}\). The rest of this section is dedicated to proving the remaining bounds.

14.4 Third derivative bounds↩︎

To use Lemma 34, we must obtain bounds on the third derivative of each term \(f_i(\boldsymbol{W})\). Here, we obtain pointwise bounds and will later further bound these terms in expectation.

14.4.1 First bound for \(f_2\)↩︎

Note that we can express \[f_2(\boldsymbol{w}) = \frac{g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w})}{g_{1, \boldsymbol{A}} (\boldsymbol{w})}\] Noting that \(g_{1, \boldsymbol{A}} \neq 0\), we can write \(f_2(\boldsymbol{w}) g_{1, \boldsymbol{A}}(\boldsymbol{w}) = g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w})\) and differentiating each side yields \[f_2'g_{1, \boldsymbol{A}} + f_2g_{1, \boldsymbol{A}}' = g_{\boldsymbol{u}}' g_{\boldsymbol{v}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'\] where \(f_2' = \partial_i f_2\), etc. This yields, as usual, \[f_2' = \frac{g_{\boldsymbol{u}}' g_{\boldsymbol{v}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' - f_2g_{1, \boldsymbol{A}}'}{g_{1, \boldsymbol{A}}} = \frac{u_i g_{\boldsymbol{v}} + g_{\boldsymbol{u}} v_i - 2f_2\cdot (\boldsymbol{A} \boldsymbol{w})_i}{g_{1, \boldsymbol{A}}}\] Differentiating each side again yields \[f_2'' g_{1, \boldsymbol{A}} + 2f_2' g_{1, \boldsymbol{A}}'+f_2 g_{1, \boldsymbol{A}}'' = g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} + 2 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}''\] and rearranging yields \[\begin{align} f_2'' &= \frac{g_{\boldsymbol{u}}''g_{\boldsymbol{v}} + 2g_{\boldsymbol{u}}'g_{\boldsymbol{v}}' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}'' - 2f_2'g_{1, \boldsymbol{A}}' - f_2g_{1, \boldsymbol{A}}''}{g_{1, \boldsymbol{A}}} = \frac{2u_i v_i - 4f_2'\cdot (\boldsymbol{A} \boldsymbol{w})_i - 2f_2\cdot\boldsymbol{A}_{ii}}{g_{1, \boldsymbol{A}}} \end{align}\] Again, we can differentiate each side again to yield \[f_2''' g_{1, \boldsymbol{A}} + 3f_2'' g_{1, \boldsymbol{A}}' + 3f_2'g_{1, \boldsymbol{A}}'' + f_2g_{1, \boldsymbol{A}}''' = g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} + 3g_{\boldsymbol{u}}''g_{\boldsymbol{v}}' + 3g_{\boldsymbol{u}}'g_{\boldsymbol{v}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}'''\] which yields \[\begin{align} f_2''' &= \frac{g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} + 3g_{\boldsymbol{u}}''g_{\boldsymbol{v}}' + 3g_{\boldsymbol{u}}'g_{\boldsymbol{v}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}''' - 3f_2'' g_{1, \boldsymbol{A}}' - 3f_2'g_{1, \boldsymbol{A}}'' - f_2g_{1, \boldsymbol{A}}'''}{g_{1, \boldsymbol{A}}} \\ &=\frac{ - 6f_2'' \cdot (\boldsymbol{A} \boldsymbol{w})_i - 6f_2'\cdot \boldsymbol{A}_{ii}}{g_{1, \boldsymbol{A}}} \end{align}\] For each \(i\), noting that \(g_{1, \boldsymbol{A}} \geq 1\), we have \[|f_2'| \lesssim|u_i g_{\boldsymbol{v}}| + |v_i g_{\boldsymbol{u}}| + |f_2 \cdot (\boldsymbol{A} \boldsymbol{w})_i|\] and \[\begin{align} |f_2''| &\lesssim|u_i v_i| + |f_2' \cdot (\boldsymbol{A} \boldsymbol{w})_i| + |f_2 \cdot \boldsymbol{A}_{ii}| \\ &\lesssim|u_i v_i| + (|u_i g_{\boldsymbol{v}} (\boldsymbol{A} \boldsymbol{w})_i| + |v_i g_{\boldsymbol{u}}(\boldsymbol{A} \boldsymbol{w})_i| + |f_2 \cdot (\boldsymbol{A} \boldsymbol{w})_i^2|) + |f_2 \cdot \boldsymbol{A}_{ii}| \\ \end{align}\] and finally \[\begin{align} \biggl|\partial_i^3 f_2(\boldsymbol{w}) \biggr| &\lesssim|f_2'' \cdot (\boldsymbol{A} \boldsymbol{w})_i| + |f_2' \cdot \boldsymbol{A}_{ii}| \\ &\lesssim[|u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| + (|u_i g_{\boldsymbol{v}} (\boldsymbol{A} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}}(\boldsymbol{A} \boldsymbol{w})_i^2| + |f_2 \cdot (\boldsymbol{A} \boldsymbol{w})_i^3|) \\ &+ |f_2 \cdot \boldsymbol{A}_{ii} (\boldsymbol{A} \boldsymbol{w})_i|] \\ &\qquad + (|u_i g_{\boldsymbol{v}} \boldsymbol{A}_{ii}| + |v_i g_{\boldsymbol{u}} \boldsymbol{A}_{ii}| + |f_2 \cdot (\boldsymbol{A} \boldsymbol{w})_i \boldsymbol{A}_{ii}|) \\ &\lesssim[|u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| + (|g_{\boldsymbol{v}} \cdot u_i (\boldsymbol{A} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} \cdot v_i (\boldsymbol{A} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i^3|) \\ &+ |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i|] \\ &\qquad + (|\boldsymbol{A}_{ii} g_{\boldsymbol{v}} \cdot u_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} \cdot v_i | + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i |) \end{align}\]

14.4.2 First bound for \(f_3\)↩︎

We can express \[f_3(\boldsymbol{w}; \boldsymbol{S}) = g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w})\] where \(g_{\boldsymbol{u}}(\boldsymbol{w}) = \boldsymbol{u}^\top \boldsymbol{w}\), \(g_{\boldsymbol{v}}(\boldsymbol{w}) = \boldsymbol{v}^\top \boldsymbol{w}\), and \(g_{\boldsymbol{A}}(\boldsymbol{w}) = \boldsymbol{w}^\top \boldsymbol{A} \boldsymbol{w}\). By Lemma 34, we see that, for \(m \geq 2\), \[\begin{gather} \sup_i \mathbb{E}\biggl[ \sup_{u \in (0,1)} g_{\boldsymbol{u}}^m\biggr] \lesssim\|\boldsymbol{u}\|_2^m, \quad \sup_i \mathbb{E}\biggl[ \sup_{u \in (0,1)} g_{\boldsymbol{v}}^m\biggr] \lesssim\|\boldsymbol{v}\|_2^m \end{gather}\] By the product rule, \[\begin{align} f_3''' &=g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}''' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}''' \\ &\qquad + 3 (g_{\boldsymbol{u}}'' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') \\ &\qquad + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' \\ &= 6 (u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}} + v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}} ) + 12 u_i v_i (\boldsymbol{A} \boldsymbol{w})_i \end{align}\] That is, \[\begin{align} \biggl|\partial_i^3 f_3(\boldsymbol{w})\biggr| &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}}| + |u_i v_i \cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ &= |\boldsymbol{A}_{ii} g_{\boldsymbol{v}} \cdot u_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} \cdot v_i| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| \end{align}\]

14.4.3 First bound for \(f_4\)↩︎

We can express \[f_4(\boldsymbol{w}; \boldsymbol{S}) = \frac{g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w})}{g_{1,\boldsymbol{B}}(\boldsymbol{w})}\] Noting that \(g_{1,\boldsymbol{B}} \neq 0\), we can write \(f_4(\boldsymbol{w}) g_{1,\boldsymbol{B}}(\boldsymbol{w}) = g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w})\) and differentiating each side yields \[f_4'g_{1,\boldsymbol{B}} + f_4 g_{1,\boldsymbol{B}}' = g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'\] where \(f_4' = \partial_i f_4\), etc. This yields, as usual, \[\begin{align} f_4' &= \frac{g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' - f_4 g_{1,\boldsymbol{B}}'}{g_{1,\boldsymbol{B}}} \\ &= \frac{u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} + v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} + 2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} - 2 f_4 \cdot (\boldsymbol{B} \boldsymbol{w})_i}{g_{1,\boldsymbol{B}}}. \end{align}\] Differentiating each side again yields \[\begin{align} f_4'' g_{1,\boldsymbol{B}} + 2f_4' g_{1,\boldsymbol{B}}'+f_4 g_{1,\boldsymbol{B}}'' &= g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' \\ &\qquad + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') \\ &= g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') \end{align}\] and rearranging yields \[\begin{align} f_4'' &= \frac{g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') - 2f_{4}' g_{1,\boldsymbol{B}}' - f_4 g_{1,\boldsymbol{B}}''}{g_{1,\boldsymbol{B}}} \\ &= \frac{2\boldsymbol{A}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} + 2 u_i v_i g_{\boldsymbol{A}} + 4u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} + 4v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} - 4f_4' \cdot (\boldsymbol{B} \boldsymbol{w})_i - 2 f_4 \cdot \boldsymbol{B}_{ii}}{g_{1,\boldsymbol{B}}} \end{align}\] Again, we can differentiate each side again to yield \[\begin{align} f_4''' g_{1,\boldsymbol{B}} + 3f_4'' g_{1,\boldsymbol{B}}' + 3f_4'g_{1,\boldsymbol{B}}'' + f_4 g_{1,\boldsymbol{B}}''' &= g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}''' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}''' \\ &\qquad + 3(g_{\boldsymbol{u}}'' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' \\ &\qquad \qquad + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') \\ &\qquad + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' \\ &= 3(g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' \end{align}\] which yields \[\begin{align} f_4''' &= \frac{3(g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' - 3f_{4}'' g_{1,\boldsymbol{B}}' - 3f_4' g_{1,\boldsymbol{B}}''}{g_{1,\boldsymbol{B}}} \\ &= \frac{6u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}} + 6 v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}} + 12 u_i v_i (\boldsymbol{A} \boldsymbol{w})_i - 6f_4'' \cdot (\boldsymbol{B} \boldsymbol{w})_i - 6 f_4' \cdot \boldsymbol{B}_{ii}}{g_{1,\boldsymbol{B}}} \end{align}\] For each \(i\), noting that \(g_{1,\boldsymbol{B}} \geq 1\), we have \[|f_4'| \lesssim|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| + |f_4 \cdot (\boldsymbol{B} \boldsymbol{w})_i|\] and \[\begin{align} |f_4''| &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} |+ |u_i v_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}}| + |f_4' \cdot (\boldsymbol{B} \boldsymbol{w})_i| + |f_4 \cdot \boldsymbol{B}_{ii}| \\ &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} |+ |u_i v_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}}| + |f_4 \cdot \boldsymbol{B}_{ii}|\\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (\boldsymbol{B} \boldsymbol{w})_i| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}(\boldsymbol{B} \boldsymbol{w})_i| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}(\boldsymbol{B} \boldsymbol{w})_i| + |f_4 \cdot (\boldsymbol{B} \boldsymbol{w})_i^2|) \end{align}\] and thus finally \[\begin{align} &|\partial_i^3 f_4(\boldsymbol{w}) | \\ &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}}| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| + |f_4'' \cdot (\boldsymbol{B} \boldsymbol{w})_i| + |f_4' \cdot \boldsymbol{B}_{ii}| \\ &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}}| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| \\ &\qquad + [|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} (\boldsymbol{B} \boldsymbol{w})_i|+ |u_i v_i g_{\boldsymbol{A}} (\boldsymbol{B} \boldsymbol{w})_i| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} (\boldsymbol{B} \boldsymbol{w})_i| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad\quad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (\boldsymbol{B} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}(\boldsymbol{B} \boldsymbol{w})_i^2| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}(\boldsymbol{B} \boldsymbol{w})_i^2| \\ &\qquad \quad +|f_4 \cdot (\boldsymbol{B} \boldsymbol{w})_i^3|) + |f_4 \cdot \boldsymbol{B}_{ii}(\boldsymbol{B} \boldsymbol{w})_i|] \\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} \boldsymbol{B}_{ii}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} \boldsymbol{B}_{ii}| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}\boldsymbol{B}_{ii}| + |f_4 \cdot (\boldsymbol{B} \boldsymbol{w})_i\boldsymbol{B}_{ii}|) \\ &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{v}} \cdot u_i | + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} \cdot v_i | + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| \\ &\qquad + [|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{B} \boldsymbol{w})_i|+ |g_{\boldsymbol{A}} \cdot u_i v_i (\boldsymbol{B} \boldsymbol{w})_i| + |g_{\boldsymbol{v}} \cdot u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + |g_{\boldsymbol{u}} \cdot v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ & \qquad+ (|g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot u_i (\boldsymbol{B} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{A}} \cdot v_i (\boldsymbol{B} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i^2| \\ &\qquad+ |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i^3|) + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i|] \\ & \qquad+ (|\boldsymbol{B}_{ii}g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot u_i | + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{A}} \cdot v_i | + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i | + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i|) \end{align}\]

14.4.4 First bound for \(f_5\)↩︎

We can express \[f_5(\boldsymbol{w}; \boldsymbol{S}) = \frac{g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w})}{h_{1,\boldsymbol{B}}(\boldsymbol{w})}\] Noting that \(h_{1,\boldsymbol{B}} \neq 0\), we can write \(f_5(\boldsymbol{w}) h_{1,\boldsymbol{B}}(\boldsymbol{w}) = g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w})\) and differentiating each side yields \[f_5'h_{1,\boldsymbol{B}} + f_5 h_{1,\boldsymbol{B}}' = g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'\] where \(f_5' = \partial_i f_5\), etc. This yields, as usual, \[\begin{align} f_5' &= \frac{g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' - f_5 h_{1,\boldsymbol{B}}'}{h_{1,\boldsymbol{B}}} \\ &= \frac{u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} + v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} + 2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} -4 f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i }{h_{1,\boldsymbol{B}}} \end{align}\] Differentiating each side again yields \[\begin{align} f_5'' h_{1,\boldsymbol{B}} + 2f_5' h_{1,\boldsymbol{B}}'+f_5 h_{1,\boldsymbol{B}}'' &= g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' \\ &\qquad + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') \\ &= g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') \end{align}\] and rearranging yields \[\begin{align} f_5'' &= \frac{g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}') - 2f_5' h_{1,\boldsymbol{B}}' - f_5 h_{1,\boldsymbol{B}}''}{h_{1,\boldsymbol{B}}} \\ &= \frac{2\boldsymbol{A}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} + 2 u_i v_i g_{\boldsymbol{A}} + 4u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} + 4v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} }{h_{1,\boldsymbol{B}}} \\ &\qquad + \frac{- 8f_5' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B}\boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i - 8 f_5 \cdot (\boldsymbol{B} \boldsymbol{w})_i^2 - 4 f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}}{h_{1,\boldsymbol{B}}} \end{align}\] Again, we can differentiate each side again to yield \[\begin{align} f_5''' h_{1,\boldsymbol{B}} + 3f_5'' h_{1,\boldsymbol{B}}' + 3f_5'h_{1,\boldsymbol{B}}'' + f_5 h_{1,\boldsymbol{B}}''' &= g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}''' g_{\boldsymbol{A}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}''' \\ &\qquad + 3(g_{\boldsymbol{u}}'' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} + g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' \\ &\qquad \qquad + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') \\ &\qquad + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' \\ &= 3(g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' \end{align}\] which yields \[\begin{align} f_5''' &= \frac{3(g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' + g_{\boldsymbol{u}}g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'') + 6 g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' - 3f_5'' h_{1,\boldsymbol{B}}' - 3f_5' h_{1,\boldsymbol{B}}'' - f_5 h_{1,\boldsymbol{B}}'''}{h_{1,\boldsymbol{B}}} \\ &= \frac{6u_i g_{\boldsymbol{v}} \boldsymbol{A}_{ii} + 6 v_i g_{\boldsymbol{u}} \boldsymbol{A}_{ii} + 12 u_i v_i (\boldsymbol{A} \boldsymbol{w})_i}{h_{1,\boldsymbol{B}}} \\ &\qquad - \frac{12f_5'' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i + 24f_5' \cdot (\boldsymbol{B} \boldsymbol{w})_i^2 }{h_{1,\boldsymbol{B}}} \\ &\qquad - \frac{12 f_5' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii} + 24 f_5 \cdot (\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{B}_{ii}}{h_{1,\boldsymbol{B}}} \end{align}\] For each \(i\), noting that \(h_{1,\boldsymbol{B}} \geq 1\), we have \[|f_5'| \lesssim|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i|\] and \[\begin{align} |f_5''| &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} |+ |u_i v_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}}| \\ &\qquad + |f_5' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i| + |f_5 \cdot \boldsymbol{(}\boldsymbol{B} \boldsymbol{w})_i^2| + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}| \\ &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} |+ |u_i v_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}}| \\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}(1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})(\boldsymbol{B} \boldsymbol{w})_i| + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2 (\boldsymbol{B} \boldsymbol{w})_i^2|) \\ &\qquad + |f_5 \cdot \boldsymbol{(}\boldsymbol{B} \boldsymbol{w})_i^2| + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}| \end{align}\] and thus finally \[\begin{align} &\biggl|\partial_i^3 f_5(\boldsymbol{w}) \biggr| \lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}}| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| + |f_5'' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + |f_5' \cdot (\boldsymbol{B} \boldsymbol{w})_i^2| + |f_5' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})\boldsymbol{B}_{ii}| + |f_5 \cdot (\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{B}_{ii}| \\ &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}}| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| \\ &+ [|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i|+ |u_i v_i g_{\boldsymbol{A}}(1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i| \\ &+ |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i| \\ & + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2(\boldsymbol{B} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}(1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2(\boldsymbol{B} \boldsymbol{w})_i^2| \\ & + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2(\boldsymbol{B} \boldsymbol{w})_i^2| + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^3 (\boldsymbol{B} \boldsymbol{w})_i^3|) \\ & + |f_5 \cdot \boldsymbol{(}\boldsymbol{B} \boldsymbol{w})_i^3 (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) | + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2 \boldsymbol{B}_{ii}(\boldsymbol{B} \boldsymbol{w})_i|] \\ & + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (\boldsymbol{B} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} (\boldsymbol{B} \boldsymbol{w})_i^2| + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} (\boldsymbol{B} \boldsymbol{w})_i^2| \\ &\qquad + |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i^3|) \\ & + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}(1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w}) \boldsymbol{B}_{ii}|+ |f_5 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{B} \boldsymbol{w})^2 (\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{B}_{ii}|) \\ &+ |f_5 \cdot (\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{B}_{ii}| \\ &\lesssim|\boldsymbol{A}_{ii} g_{\boldsymbol{v}} \cdot u_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} \cdot v_i| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| \\ & + [|\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{1, \boldsymbol{B}} \cdot (\boldsymbol{B} \boldsymbol{w})_i|+ |g_{\boldsymbol{A}}g_{1, \boldsymbol{B}} \cdot u_i v_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &+ | g_{\boldsymbol{v}} g_{1, \boldsymbol{B}} \cdot u_i (\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B} \boldsymbol{w})_i| + | g_{\boldsymbol{u}} g_{1, \boldsymbol{B}} \cdot v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &+ (| g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}}^2\cdot u_i(\boldsymbol{B} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{A}}g_{1, \boldsymbol{B}}^2\cdot v_i(\boldsymbol{B} \boldsymbol{w})_i^2| \\ & + | g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{1, \boldsymbol{B}}^2 \cdot (\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}}^3 \cdot (\boldsymbol{B} \boldsymbol{w})_i^3|) \\ & + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}} \cdot \boldsymbol{(}\boldsymbol{B} \boldsymbol{w})_i^3 | + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}}^2 \cdot (\boldsymbol{B} \boldsymbol{w})_i|] \\ & + (| g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot u_i (\boldsymbol{B} \boldsymbol{w})_i^2| + | g_{\boldsymbol{u}} g_{\boldsymbol{A}} \cdot v_i(\boldsymbol{B} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i^2| \\ &\qquad + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot g_{1, \boldsymbol{B}} \cdot (\boldsymbol{B} \boldsymbol{w})_i^3|) \\ & + (|\boldsymbol{B}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}} \cdot u_i| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}} \cdot v_i| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}}g_{1, \boldsymbol{B}} \cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ & + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1, \boldsymbol{B}}^2 \cdot (\boldsymbol{B} \boldsymbol{w})_i |) \\ & + | \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i| \end{align}\]

14.4.5 First bound for \(f_6\)↩︎

We can express \[f_6(\boldsymbol{w}; \boldsymbol{S}) = \frac{h_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w})}{g_{1,\boldsymbol{C}}(\boldsymbol{w})}.\]

Noting that \(g_{1,\boldsymbol{C}} \neq 0\), we can write \(f_6(\boldsymbol{w}) g_{1,\boldsymbol{C}}(\boldsymbol{w}) = h_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w})\) and differentiating each side yields \[f_6'g_{1,\boldsymbol{C}} + f_6 g_{1,\boldsymbol{C}}' = h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'\] where \(f_6' = \partial_i f_6\), etc. This yields, as usual, \[\begin{align} f_6' &= \frac{h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' - f_6 g_{1,C}'}{g_{1,\boldsymbol{C}}} \\ &= \frac{2u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} + 2 (\boldsymbol{A} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{B}} + 2 (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} - 2 f_6 \cdot (\boldsymbol{C} \boldsymbol{w})_i}{g_{1, \boldsymbol{C}}} \end{align}\] Differentiating each side again yields \[\begin{align} f_6'' g_{1, \boldsymbol{C}} + 2f_6' g_{1, \boldsymbol{C}}' + f_6 g_{1, \boldsymbol{C}}'' &= h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' \\ &\qquad + 2(h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') \end{align}\] and rearranging yields \[\begin{align} f_6'' &= \frac{h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' }{g_{1, \boldsymbol{C}}} \\ &\qquad +\frac{2(h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') - 2f_6' g_{1, \boldsymbol{C}}' - f_6 g_{1, \boldsymbol{C}}''}{g_{1, \boldsymbol{C}}} \\ &= \frac{2 u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} + 2 \boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}} + 2 \boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} + 8 u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}}{g_{1, \boldsymbol{C}}} \\ &\qquad + \frac{8 u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}} + 8 (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} }{g_{1, \boldsymbol{C}}} \\ &\qquad - \frac{4 f_6' \cdot ( \boldsymbol{C} \boldsymbol{w})_i + 2 f_6 \cdot \boldsymbol{C}_{ii}}{g_{1, \boldsymbol{C}}} \end{align}\] Again, we can differentiate each side again to yield \[\begin{align} f_6''' &g_{1, \boldsymbol{C}} + 3f_6'' g_{1, \boldsymbol{C}}' + 3f_6' g_{1,\boldsymbol{C}}'' + f_6 g_{1, \boldsymbol{C}}''' \\&= h_{\boldsymbol{u}}''' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}''' g_{\boldsymbol{B}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}''' \\ &\qquad + 3 (h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'') \\ &\qquad + 6 h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' \\ &= 3 (h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'') \\ &\qquad + 6 h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' \end{align}\] which yields \[\begin{align} f_6''' &= \frac{3 (h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'') + 6 h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'}{g_{1, \boldsymbol{C}}} \\ &\qquad - \frac{3f_6'' g_{1,\boldsymbol{C}}' + 3 f_6' g_{1, \boldsymbol{C}}''}{g_{1, \boldsymbol{C}}} \\ &= \frac{3 (4u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} + 4 u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}} + 4 u_i^2 (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}}}{g_{1, \boldsymbol{C}}} \\ &\qquad + \frac{4 u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}} + 4 (\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{A}_{ii} h_{\boldsymbol{u}} + 4 (\boldsymbol{A} \boldsymbol{w})_i \boldsymbol{B}_{ii} h_{\boldsymbol{u}}) }{g_{1, \boldsymbol{C}}} \\ &\qquad + \frac{48 u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i}{g_{1, \boldsymbol{C}}} - \frac{6f_6'' \cdot (\boldsymbol{C} \boldsymbol{w})_i + 6 f_6' \boldsymbol{C}_{ii}}{g_{1, \boldsymbol{C}}} \end{align}\] For each \(i\), noting that \(g_{1,\boldsymbol{C}} \geq 1\), we have \[|f_6'| \lesssim|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |(\boldsymbol{A} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |f_6 \cdot (\boldsymbol{C} \boldsymbol{w})_i|\] and \[\begin{align} |f_6''| &\lesssim|u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}| \\ &\qquad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}| + | (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}}| + |f_6'\cdot (\boldsymbol{C}\boldsymbol{w})_i| + |f_6 \boldsymbol{C}_{ii}| \\ &\lesssim|u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}|\\ &\qquad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}| + | (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}}| \\ &\qquad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C}\boldsymbol{w})_i| + |(\boldsymbol{A} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot (\boldsymbol{C}\boldsymbol{w})_i| \\ &+ |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}\cdot (\boldsymbol{C}\boldsymbol{w})_i| + |f_6 \cdot (\boldsymbol{C} \boldsymbol{w})_i^2|) \end{align}\] and thus finally \[\begin{align} \biggl|\partial_i^3 f_6(\boldsymbol{w})\biggr| &\lesssim|u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}}| + |u_i^2 (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}}| \\ &\qquad + |(\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{A}_{ii} h_{\boldsymbol{u}}| + |(\boldsymbol{A} \boldsymbol{w})_i \boldsymbol{B}_{ii} h_{\boldsymbol{u}}| + |u_i(\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + |f_6'' \cdot (\boldsymbol{C} \boldsymbol{w})_i | + |f_6' \boldsymbol{C}_{ii}| \\ &\lesssim|u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}}| + |u_i^2 (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}}| \\ &\qquad + |(\boldsymbol{B} \boldsymbol{w})_i \boldsymbol{A}_{ii} h_{\boldsymbol{u}}| + |(\boldsymbol{A} \boldsymbol{w})_i \boldsymbol{B}_{ii} h_{\boldsymbol{u}}| + |u_i(\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + [|u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}\cdot (\boldsymbol{C} \boldsymbol{w})_i|+ |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + | (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} \cdot (\boldsymbol{C} \boldsymbol{w})_i| + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C}\boldsymbol{w})_i^2| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot (\boldsymbol{C}\boldsymbol{w})_i^2| + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}\cdot (\boldsymbol{C}\boldsymbol{w})_i^2| + |f_6 \cdot (\boldsymbol{C} \boldsymbol{w})_i^3|)] \\ &\qquad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}}\boldsymbol{C}_{ii}| + |(\boldsymbol{A} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{B}}\boldsymbol{C}_{ii}| \\ &+ |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}\boldsymbol{C}_{ii}| + |f_6 \cdot (\boldsymbol{C} \boldsymbol{w})_i\boldsymbol{C}_{ii}|) \\ &\lesssim| g_{\boldsymbol{B}} \cdot u_i^2 (\boldsymbol{A} \boldsymbol{w})_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot u_i| + |g_{\boldsymbol{A}} \cdot u_i^2 (\boldsymbol{B}\boldsymbol{w})_i | + |\boldsymbol{B}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{A}} \cdot u_i | \\ &\qquad + | \boldsymbol{A}_{ii} g_{\boldsymbol{u}}^2 \cdot (\boldsymbol{B} \boldsymbol{w})_i| + | \boldsymbol{B}_{ii} g_{\boldsymbol{u}}^2 \cdot (\boldsymbol{A} \boldsymbol{w})_i| + |g_{\boldsymbol{u}} \cdot u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + [|g_{\boldsymbol{A}} g_{\boldsymbol{B}}\cdot u_i^2 (\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}}\cdot (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad + |g_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + | g_{\boldsymbol{u}} g_{\boldsymbol{A}}\cdot u_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i| + |g_{\boldsymbol{u}}^2 \cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + (|g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot u_i (\boldsymbol{C}\boldsymbol{w})_i^2| + | g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}}\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C}\boldsymbol{w})_i^2| \\ &\qquad \quad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}}\cdot (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C}\boldsymbol{w})_i^2| + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^3|)] \\ &\qquad + (|\boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot u_i| + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}} \cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ &+ |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i| + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i|) \end{align}\] We now bound each of the corresponding terms, as we did earlier.

14.4.6 First bound for \(f_7\)↩︎

We can express \[f_7(\boldsymbol{w}; \boldsymbol{S}) = \frac{g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w})}{h_{1,\boldsymbol{C}}(\boldsymbol{w})}.\] Noting that \(h_{1,\boldsymbol{C}} \neq 0\), we can write \(f_7(\boldsymbol{w}) h_{1,\boldsymbol{C}}(\boldsymbol{w}) = g_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{v}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w})\) and differentiating each side yields \[\begin{align} f_7'h_{1,\boldsymbol{C}} + f_7 h_{1,\boldsymbol{C}}' &= g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} ' \end{align}\] where \(f_7' = \partial_i f_7\), etc. This yields, as usual, \[\begin{align} f_7' &= \frac{g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} ' - f_7 h_{1,\boldsymbol{C}}'}{h_{1,\boldsymbol{C}}} \\ &= \frac{u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + 2 (\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} }{h_{1,\boldsymbol{C}}} \\ &\qquad+ \frac{2 (\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} - 4f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i}{h_{1,\boldsymbol{C}}} \end{align}\] Differentiating each side again yields \[\begin{align} f_7'' h_{1, \boldsymbol{C}} + 2f_7' h_{1, \boldsymbol{C}}' + f_7 h_{1, \boldsymbol{C}}'' &= g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' \\ &\qquad + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} \\ &\qquad \qquad + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') \\ &=g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' \\ &\qquad + 2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} \\ &\qquad \qquad + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') \end{align}\] and rearranging yields \[\begin{align} &f_7'' = \frac{2(g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}')}{h_{1, \boldsymbol{C}}} \\ &\qquad + \frac{g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' - 2f_7' h_{1,\boldsymbol{C}}' - f_7 h_{1,\boldsymbol{C}}''}{h_{1, \boldsymbol{C}}} \\ &=\frac{2(u_i v_i g_{\boldsymbol{A}} g_{\boldsymbol{B}} + 2 u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{B}} + 2 u_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{A}})}{h_{1, \boldsymbol{C}}} \\ &\qquad + \frac{2( 2 v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{B}} + 2 v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} + 4 (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}})}{h_{1, \boldsymbol{C}}} \\ &\qquad + \frac{2 \boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} + 2 \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} - 8 f_7' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i }{h_{1, \boldsymbol{C}}} \\ &\qquad + \frac{- 8 f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i^2 - 4 f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}}{h_{1, \boldsymbol{C}}} \end{align}\] Again, we can differentiate each side again to yield \[\begin{align} &f_7''' h_{1, \boldsymbol{C}} + 3f_7'' h_{1, \boldsymbol{C}}' + 3f_7' h_{1, \boldsymbol{C}}'' + f_7 h_{1, \boldsymbol{C}}''' \\&= g_{\boldsymbol{u}}''' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}''' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}''' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}''' \\ &\qquad + 3 (g_{\boldsymbol{u}}'' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} + g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}\\ &\qquad \qquad + g_{\boldsymbol{u}}'' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} \\ &\qquad \qquad + g_{\boldsymbol{u}} g_{\boldsymbol{v}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'') \\ &\qquad + 6 (g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') \\ &= 3 ( g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} \\ &\qquad \qquad + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'') \\ &\qquad + 6 (g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}') \end{align}\] which yields \[\begin{align} f_7''' &= \frac{3 ( g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'')}{h_{1,\boldsymbol{C}}} \\ &\quad + \frac{6 (g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} + g_{\boldsymbol{u}}' g_{\boldsymbol{v}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' + g_{\boldsymbol{u}}' g_{\boldsymbol{v}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' + g_{\boldsymbol{u}} g_{\boldsymbol{v}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}')}{h_{1, \boldsymbol{C}}} \\ & \quad- \frac{3f_7'' h_{1,\boldsymbol{C}}' + 3f_7' h_{1,\boldsymbol{C}}'' + f_7 h_{1,\boldsymbol{C}}'''}{h_{1, \boldsymbol{C}}} \\ &= \frac{ 6 (u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{B}}+ u_i \boldsymbol{B}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{A}} + v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}})}{h_{1, \boldsymbol{C}}} \\ &\quad + \frac{ 6 (v_i \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} + 2 \boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} + 2 \boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}})}{h_{1, \boldsymbol{C}}} \\ &\quad + \frac{12 (u_i v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} + u_i v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}+ 2 u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}} + 2 v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}})}{h_{1, \boldsymbol{C}}} \\ &\quad - \frac{12f_7'' (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i + 24f_7' \cdot (\boldsymbol{C} \boldsymbol{w})_i^2}{h_{1, \boldsymbol{C}}} \\ &\quad + \frac{12 f_7' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})\boldsymbol{C}_{ii} + 24 f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i \boldsymbol{C}_{ii}}{h_{1, \boldsymbol{C}}} \end{align}\] For each \(i\), noting that \(h_{1,\boldsymbol{C}} \geq 1\), we have \[\begin{align} |f_7'| &\lesssim|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |(\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}}| \\ &\qquad + |(\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i| \end{align}\] and \[\begin{align} |f_7''| &\lesssim|u_i v_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{B}}| \\ &\qquad + |v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}| \\ &\qquad + |f_7' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| + |f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| \\ &\lesssim|u_i v_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{B}}| \\ &\qquad + |v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}| \\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2 (\boldsymbol{C} \boldsymbol{w})_i^2|) \\ &\qquad + |f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| \end{align}\] and thus finally \[\begin{align} &\biggl|\partial_i^3 f_7(\boldsymbol{W}_{t,u,i})\biggr| \\ &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |u_i \boldsymbol{B}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |v_i \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |\boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| \\ &\qquad + |\boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| + |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}}| \\ &\qquad + | v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}}| +| f_7'' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| + |f_7' \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad + |f_7' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})\boldsymbol{C}_{ii}| + |f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i \boldsymbol{C}_{ii}| \\ &\lesssim|u_i \boldsymbol{A}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{B}}| + |u_i \boldsymbol{B}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{A}}| + |v_i \boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |v_i \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |\boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}| \\ &\qquad + |\boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}}|+ |u_i v_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}}| + |u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}}| \\ &\qquad + | v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}}| \\ &\qquad + [|u_i v_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}(1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| + |u_i (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{B}}(1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |u_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{v}} g_{\boldsymbol{A}}(1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |v_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{u}} g_{\boldsymbol{v}} (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2(\boldsymbol{C} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2(\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |(\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2(\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |(\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2(\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^3 (\boldsymbol{C} \boldsymbol{w})_i^3|) \\ &\qquad \quad + |f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i^3 (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2 \boldsymbol{C}_{ii} (\boldsymbol{C} \boldsymbol{w})_i|] \\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| + |(\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^2| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i^3|) \\ &\qquad + (|u_i g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| + |v_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_ig_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w}) \boldsymbol{C}_{ii}| + |f_7 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{C} \boldsymbol{w})^2 (\boldsymbol{C} \boldsymbol{w})_i \boldsymbol{C}_{ii}|) \\ &\qquad + |f_7 \cdot (\boldsymbol{C} \boldsymbol{w})_i \boldsymbol{C}_{ii}| \\ &\lesssim| \boldsymbol{A}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot u_i| + | \boldsymbol{B}_{ii} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot u_i| + | \boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}} \cdot v_i| + | \boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} \cdot v_i| + |\boldsymbol{A}_{ii} \cdot g_{\boldsymbol{u}} g_{\boldsymbol{v}} (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} \cdot (\boldsymbol{A} \boldsymbol{w})_i| + | g_{\boldsymbol{B}} \cdot u_i v_i (\boldsymbol{A} \boldsymbol{w})_i| + | g_{\boldsymbol{A}} \cdot u_i v_i (\boldsymbol{B} \boldsymbol{w})_i | + | g_{\boldsymbol{v}} \cdot u_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + | g_{\boldsymbol{u}} \cdot v_i (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i| \\ &\qquad + [|g_{\boldsymbol{A}} g_{\boldsymbol{B}}g_{1, \boldsymbol{C}}\cdot u_i v_i (\boldsymbol{C} \boldsymbol{w})_i| + | g_{\boldsymbol{v}} g_{\boldsymbol{B}}g_{1, \boldsymbol{C}}\cdot u_i (\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + | g_{\boldsymbol{v}} g_{\boldsymbol{A}}g_{1, \boldsymbol{C}}\cdot u_i (\boldsymbol{B} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + | g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{1, \boldsymbol{C}}\cdot v_i (\boldsymbol{B} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{1, \boldsymbol{C}}\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} g_{1, \boldsymbol{C}} \cdot (\boldsymbol{C} \boldsymbol{w})_i| + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}}g_{1, \boldsymbol{C}} \cdot (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + (|g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}^2 \cdot u_i (\boldsymbol{C} \boldsymbol{w})_i^2| + | g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}^2 \cdot v_i (\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}^2 \cdot (\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1,\boldsymbol{C}}^2\cdot (\boldsymbol{B} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}^3 \cdot (\boldsymbol{C} \boldsymbol{w})_i^3|) \\ &\qquad \quad + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}\cdot (\boldsymbol{C} \boldsymbol{w})_i^3 | + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}^2 \cdot (\boldsymbol{C} \boldsymbol{w})_i|] \\ &\qquad + (|g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot u_i (\boldsymbol{C} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot v_i (\boldsymbol{C} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}} \cdot(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i^2| \\ &\qquad \quad + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} \cdot (\boldsymbol{B} \boldsymbol{w})_i(\boldsymbol{C} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}} \cdot (\boldsymbol{C} \boldsymbol{w})_i^3|) \\ &\qquad + (|\boldsymbol{C}_{ii}g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}} \cdot u_i | + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}} \cdot v_i| \\ &\qquad \quad + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{B}}g_{1,\boldsymbol{C}} \cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ &\qquad \quad + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{1,\boldsymbol{C}} \cdot (\boldsymbol{B} \boldsymbol{w})_i| + |\boldsymbol{C}_{ii}g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{C}}^2 \cdot (\boldsymbol{C} \boldsymbol{w})_i|) \\ &\qquad + | \boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{v}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i| \end{align}\] We now bound each of the corresponding terms, as we did earlier.

14.4.7 First bound for \(f_8\)↩︎

We can express \[f_8(\boldsymbol{w}; \boldsymbol{S}) = \frac{h_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w}) g_{\boldsymbol{C}}(\boldsymbol{w})}{h_{1,\boldsymbol{D}}(\boldsymbol{w})}.\] Noting that \(h_{1,\boldsymbol{D}} \neq 0\), we can write \(f_8(\boldsymbol{w}) h_{1,\boldsymbol{D}}(\boldsymbol{w}) = h_{\boldsymbol{u}}(\boldsymbol{w}) g_{\boldsymbol{A}}(\boldsymbol{w}) g_{\boldsymbol{B}}(\boldsymbol{w}) g_{\boldsymbol{C}}(\boldsymbol{w})\) and differentiating each side yields \[f_8' h_{1,\boldsymbol{D}} + f_8 h_{1,\boldsymbol{D}}' = h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'\] where \(f_8' = \partial_i f_8\), etc. This yields, as usual, \[\begin{align} f_8' &= \frac{h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' - f_8 h_{1, \boldsymbol{D}}'}{h_{1, \boldsymbol{D}}} \\ &= \frac{2 u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + 2 (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + 2 (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{ 2 (\boldsymbol{C} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} - 4 f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i}{h_{1, \boldsymbol{D}}} \end{align}\] Differentiating each side again yields \[\begin{align} f_8'' h_{1, \boldsymbol{D}} + 2f_8' h_{1, \boldsymbol{D}}' + f_8 h_{1, \boldsymbol{D}}'' &= h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'' \\ &\qquad + 2( h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' \\ &\qquad \qquad + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}') \end{align}\] and rearranging yields \[\begin{align} f_{8}'' &= \frac{2( h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' ) }{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{2( h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}') }{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'' - 2f_8' h_{1, \boldsymbol{D}}' - f_8 h_{1,\boldsymbol{D}}''}{h_{1, \boldsymbol{D}}} \\ &= \frac{8( u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{8((\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B}\boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{C}} + (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} + (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} ) }{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{2u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + 2 \boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + 2 \boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} + 2 \boldsymbol{C}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}}{h_{1, \boldsymbol{D}}} \\ &\qquad - \frac{8f_8' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i + 8 f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i^2 + 4 f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}}{h_{1, \boldsymbol{D}}} \end{align}\] Again, we can differentiate each side again to yield \[\begin{align} &f_8''' h_{1, \boldsymbol{D}} + 3 f_8'' h_{1, \boldsymbol{D}}' + 3 f_8' h_{1, \boldsymbol{D}}'' + f_8 h_{1, \boldsymbol{D}}''' \\&= h_{\boldsymbol{u}}''' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}''' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}''' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}''' \\ &\qquad + 3(h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} \\ &\qquad \qquad + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} \\ &\qquad \qquad + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}''+ h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}'') \\ &\qquad + 6 (h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}}) \\ &= 3(h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} \\ &\qquad \qquad + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'' g_{\boldsymbol{C}} \\ &\qquad \qquad + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}''+ h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}'') \\ &\qquad + 6 (h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}}) \end{align}\] which yields \[\begin{align} f_8''' &= \frac{3(h_{\boldsymbol{u}}'' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}} + h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{3(h_{\boldsymbol{u}}'' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}'' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}}' g_{\boldsymbol{C}} + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}'' g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{3(h_{\boldsymbol{u}} g_{\boldsymbol{A}}'' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}''+ h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}'' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}'')}{h_{1,\boldsymbol{D}}} \\ &\qquad + \frac{6 (h_{\boldsymbol{u}} g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}} g_{\boldsymbol{B}}' g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}} g_{\boldsymbol{C}}' + h_{\boldsymbol{u}}' g_{\boldsymbol{A}}' g_{\boldsymbol{B}}' g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad - \frac{3f_8'' h_{1,\boldsymbol{D}}' + 3f_8' h_{1, \boldsymbol{D}}'' + f_8 h_{1, \boldsymbol{D}}'''}{h_{1, \boldsymbol{D}}} \\ &= \frac{12(u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}} g_{\boldsymbol{C}} + u_i^2 (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{12(u_i^2 (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{C}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{B}} + \boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}} + \boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{12( \boldsymbol{A}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} + \boldsymbol{C}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} + \boldsymbol{B}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} + \boldsymbol{C}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}})}{h_{1,\boldsymbol{D}}} \\ &\qquad + \frac{48 ( (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} + u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}})}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{48(u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{B}} + u_i (\boldsymbol{u}^\top \boldsymbol{w})(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{C}})}{h_{1, \boldsymbol{D}}} \\ &\qquad - \frac{12f_8'' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i + 24f_8' \cdot (\boldsymbol{D} \boldsymbol{w})_i^2}{h_{1, \boldsymbol{D}}} \\ &\qquad + \frac{12f_8' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii} + 24 f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i \boldsymbol{D}_{ii}}{h_{1, \boldsymbol{D}}} \end{align}\] For each \(i\), noting that \(h_{1,\boldsymbol{D}} \geq 1\), we have \[\begin{align} |f_8'| &\lesssim|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |(\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}| \\ &\qquad + |(\boldsymbol{C} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \end{align}\] and \[\begin{align} |f_8''| &\lesssim|u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B}\boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{C}}| + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |(\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| \\ &\qquad + |u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |\boldsymbol{C}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| \\ &\qquad + |f_8' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| + |f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i^2| + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| \\ &\lesssim|u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B}\boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{C}}| + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |(\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| \\ &\qquad + |u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |\boldsymbol{C}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| \\ &\qquad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{C} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i^2|) \\ &\qquad + |f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i^2| + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| \end{align}\] and thus finally \[\begin{align} &\biggl|\partial_i^3 f_8(\boldsymbol{w})\biggr | \\ &\lesssim|u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i^2 (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{C}}| \\ &\qquad + |u_i^2 (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{C}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |\boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}}| + |\boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}}| \\ &\qquad + |\boldsymbol{A}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{C}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |\boldsymbol{C}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}}| \\ &\qquad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w})(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{C}}| \\ &\qquad + |f_8'' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| + |f_8' \cdot (\boldsymbol{D} \boldsymbol{w})_i^2| + |f_8' \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}|\\ &\qquad + |f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i \boldsymbol{D}_{ii}| \\ &\lesssim|u_i^2 (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{A}_{ii} g_{\boldsymbol{B}} g_{\boldsymbol{C}}| + |u_i^2 (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{B}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{C}}| \\ &\qquad + |u_i^2 (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) \boldsymbol{C}_{ii} g_{\boldsymbol{A}} g_{\boldsymbol{B}}| + |\boldsymbol{A}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}}| + |\boldsymbol{B}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{C}}| \\ &\qquad + |\boldsymbol{A}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{C}_{ii} (\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}}| + |\boldsymbol{B}_{ii} (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| + |\boldsymbol{C}_{ii} (\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}}| \\ &\qquad + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}}| \\ &\qquad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{B}}| + |u_i (\boldsymbol{u}^\top \boldsymbol{w})(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i g_{\boldsymbol{C}}| \\ &\qquad + [|u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{A} \boldsymbol{w})_i g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{B}\boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |u_i (\boldsymbol{u}^\top \boldsymbol{w}) (\boldsymbol{C} \boldsymbol{w})_i g_{\boldsymbol{A}} g_{\boldsymbol{B} }(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B}\boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |u_i^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| + |\boldsymbol{A}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |\boldsymbol{B}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| + |\boldsymbol{C}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |(\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |(\boldsymbol{C} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^3(\boldsymbol{D} \boldsymbol{w})_i^3|) \\ &\qquad \quad + |f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i^3 (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})| + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2 \boldsymbol{D}_{ii} (\boldsymbol{D} \boldsymbol{w})_i|] \\ &\qquad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (\boldsymbol{D} \boldsymbol{w})_i^2| + |(\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}}(\boldsymbol{D} \boldsymbol{w})_i^2| + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \quad + |(\boldsymbol{C} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} (\boldsymbol{D} \boldsymbol{w})_i^2| + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})(\boldsymbol{D} \boldsymbol{w})_i^3|) \\ &\qquad + (|u_i (\boldsymbol{u}^\top \boldsymbol{w}) g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| + |(\boldsymbol{A} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| \\ &\qquad \quad + |(\boldsymbol{B} \boldsymbol{w})_i h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}}(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| + |(\boldsymbol{C} \boldsymbol{w})_ih_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}(1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w}) \boldsymbol{D}_{ii}| \\ &\qquad \quad + |f_8 \cdot (1 + \boldsymbol{w}^\top \boldsymbol{D} \boldsymbol{w})^2(\boldsymbol{D} \boldsymbol{w})_i \boldsymbol{D}_{ii}|) \\ &\qquad + |f_8 \cdot (\boldsymbol{D} \boldsymbol{w})_i \boldsymbol{D}_{ii}| \end{align}\] or rather \[\begin{align} &\biggl|\partial_i^3 f_8(\boldsymbol{w})\biggr| \\ &\lesssim|g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot u_i^2 (\boldsymbol{A} \boldsymbol{w})_i | + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot u_i| + |g_{\boldsymbol{A}} g_{\boldsymbol{C}} \cdot u_i^2 (\boldsymbol{B} \boldsymbol{w})_i | + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} \cdot u_i| \\ &\qquad + |g_{\boldsymbol{A}} g_{\boldsymbol{B}}\cdot u_i^2 (\boldsymbol{C} \boldsymbol{w})_i | + | \boldsymbol{C}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}\cdot u_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{C}}\cdot (\boldsymbol{B} \boldsymbol{w})_i | + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{C}}\cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ &\qquad + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}}\cdot (\boldsymbol{C} \boldsymbol{w})_i | + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}}\cdot (\boldsymbol{A} \boldsymbol{w})_i | + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}}\cdot (\boldsymbol{C} \boldsymbol{w})_i | \\ &\qquad + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}}\cdot (\boldsymbol{B} \boldsymbol{w})_i | \\ &\qquad + |g_{\boldsymbol{u}}^2\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i | + | g_{\boldsymbol{u}} g_{\boldsymbol{A}}\cdot u_i(\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i | \\ &\qquad + | g_{\boldsymbol{u}} g_{\boldsymbol{B}}\cdot u_i(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i | + | g_{\boldsymbol{u}}g_{\boldsymbol{C}}\cdot u_i(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{B} \boldsymbol{w})_i | \\ &\qquad + [| g_{\boldsymbol{u}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot u_i(\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i| + | g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}\cdot u_i(\boldsymbol{B}\boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + | g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B} } g_{1,\boldsymbol{D}} \cdot u_i(\boldsymbol{C} \boldsymbol{w})_i(\boldsymbol{D} \boldsymbol{w})_i|+ |g_{\boldsymbol{u}}^2 g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot (\boldsymbol{A} \boldsymbol{w})_i(\boldsymbol{B}\boldsymbol{w})_i(\boldsymbol{D} \boldsymbol{w})_i| \\ &\qquad \quad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}}g_{1,\boldsymbol{D}}\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i| + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{1,\boldsymbol{D}}\cdot (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{C} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i | \\ &\qquad \quad + |g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot u_i^2 (\boldsymbol{D} \boldsymbol{w})_i| + |\boldsymbol{A}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot (\boldsymbol{D} \boldsymbol{w})_i | \\ &\qquad \quad + |\boldsymbol{B}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}\cdot(\boldsymbol{D} \boldsymbol{w})_i | + |\boldsymbol{C}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{D}} \cdot (\boldsymbol{D} \boldsymbol{w})_i | \\ &\qquad \quad + (| g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}^2 \cdot u_i(\boldsymbol{D} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}} g_{\boldsymbol{C}}g_{1,\boldsymbol{D}}^2\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i^2 | \\ &\qquad \qquad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}^2 \cdot (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i^2| + | g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{1,\boldsymbol{D}}^2 \cdot (\boldsymbol{C} \boldsymbol{w})_i(\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \qquad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}^3 \cdot (\boldsymbol{D} \boldsymbol{w})_i^3|) \\ &\qquad \quad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}\cdot (\boldsymbol{D} \boldsymbol{w})_i^3 | + |\boldsymbol{D}_{ii}g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}^2 \cdot (\boldsymbol{D} \boldsymbol{w})_i|] \\ &\qquad + (|u_i g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot (\boldsymbol{D} \boldsymbol{w})_i^2| + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}} g_{\boldsymbol{C}}\cdot (\boldsymbol{A} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i^2| + | g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{C}} \cdot (\boldsymbol{B} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i^2| \\ &\qquad \quad + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} \cdot (\boldsymbol{C} \boldsymbol{w})_i (\boldsymbol{D} \boldsymbol{w})_i^2 | + |g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot (\boldsymbol{D} \boldsymbol{w})_i^3|) \\ &\qquad + (|\boldsymbol{D}_{ii} g_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot u_i| + |\boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}} \cdot (\boldsymbol{A} \boldsymbol{w})_i| \\ &\qquad \quad + |\boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{C}}g_{1,\boldsymbol{D}} \cdot (\boldsymbol{B} \boldsymbol{w})_i | + |\boldsymbol{D}_{ii} h_{\boldsymbol{u}} g_{\boldsymbol{A}} g_{\boldsymbol{B}}g_{1,\boldsymbol{D}} \cdot (\boldsymbol{C} \boldsymbol{w})_i| \\ &\qquad \quad + | \boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} g_{1,\boldsymbol{D}}^2 \cdot (\boldsymbol{D} \boldsymbol{w})_i|) \\ &\qquad + |\boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot (\boldsymbol{D} \boldsymbol{w})_i| \end{align}\]

14.4.8 Second stage bounding↩︎

We are now ready to further bound these terms in expectation. Note that every term \(|\partial_i^3 f_j(\boldsymbol{w})|\) is bounded by a sum of terms of the form \[|\text{(zero or one matrix diagonal element}) \cdot (\text{product of functions}) \cdot (\text{product of vector elements})|\] Therefore, to bound \(\sum_{i=1}^p\mathbb{E}[ |W_i|^r \cdot \sup_{u \in (0,1)} |\partial_i^3 f_j(\boldsymbol{W}_{t,u,i})|]\), we do the following to each term that appears:

1. Bound matrix diagonal elements by the operator norm of the matrix.

2. Use Cauchy-Schwarz to separate the expectation of \(|W_i|^r \cdot \text{product of functions}\) from the expectation of the product of vector elements.

3. Use Hölder’s inequality to split the expectation of \(|W_i|^r \cdot \text{product of functions}\) into multiple expectations.

4. Bound each of these expectations using Lemma 34.

5. Use Hölder’s inequality on the expectation of the product of vector elements.

6. Use Cauchy-Schwarz (potentially repeatedly) on the sum (over \(i = 1, \dots, p\)) of products of vector elements. If there is only one vector element, use Cauchy-Schwarz with said vector element and the constant \(1\) vector, thus picking up an additional \(\sqrt{p}\) factor.

7. For random vector elements involving \(\boldsymbol{W}_{t,u,i}\), use Lemma 34 to bound each resulting vector norm expectation.

This process of repeatedly applying Cauchy-Schwarz and Hölder’s inequality on various terms suffices for our universality proof because they ultimately maintain the scale of the resulting bounds (with respect to \(p\) and norms of elements of \(\boldsymbol{S}\)). As an example, we can bound \[\begin{align} \mathbb{E}\biggl[\sum_{i=1}^p &|W_i|^r \cdot \sup_{u \in (0,1)} |\boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot (\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i|\biggr] \\ &\leq \|\boldsymbol{D}\|_{\text{op}}\sum_{i=1}^p\mathbb{E}\biggl[ |W_i|^r \cdot \sup_{u \in (0,1)} |\boldsymbol{D}_{ii} g_{\boldsymbol{u}}^2 g_{\boldsymbol{A}} g_{\boldsymbol{B}} g_{\boldsymbol{C}} \cdot (\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i|\biggr] \\ &\leq \|\boldsymbol{D}\|_{\text{op}} \sup_i \mathbb{E}\biggl[|W_i|^{2r}\sup_{u \in (0,1)} |g_{\boldsymbol{u}}^4 g_{\boldsymbol{A}}^2 g_{\boldsymbol{B}}^2 g_{\boldsymbol{C}}^2| \biggr]^{1/2} \sum_{i=1}^p\mathbb{E}\biggl[ \sup_{u \in (0,1)} |(\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i^2|\biggr]^{1/2} \\ &\leq \|\boldsymbol{D}\|_{\text{op}} \sup_i \mathbb{E}[|W_i|^{2r}]^{1/2} \sup_i\mathbb{E}\biggl[\sup_{u \in (0,1)} |g_{\boldsymbol{u}}^4 | \biggr]^{1/2} \sup_i \mathbb{E}\biggl[\sup_{u \in (0,1)} |g_{\boldsymbol{A}}^2 | \biggr]^{1/2} \\ &\qquad \sup_i \mathbb{E}\biggl[\sup_{u \in (0,1)} | g_{\boldsymbol{B}}^2 | \biggr]^{1/2} \sup_i \mathbb{E}\biggl[\sup_{u \in (0,1)} |g_{\boldsymbol{C}}^2| \biggr]^{1/2} \cdot \sum_{i=1}^p\mathbb{E}\biggl[ \sup_{u \in (0,1)} |(\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i^2|\biggr]^{1/2} \\ &\lesssim\|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} p^3 \cdot \sum_{i=1}^p\mathbb{E}\biggl[ \sup_{u \in (0,1)} |(\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i^2|\biggr]^{1/2} \\ &\leq \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} p^3 \cdot \sqrt{p} \biggl(\sum_{i=1}^p\mathbb{E}\biggl[ \sup_{u \in (0,1)} |(\boldsymbol{D} \boldsymbol{W}_{t,u,i})_i^2|\biggr]\biggr)^{1/2} \\ &\lesssim\|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} p^3 \cdot \sqrt{p} \cdot \|\boldsymbol{D}\|_{\text{op}} \sqrt{p} \\ &= \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{D}\|_{\text{op}}^2 p^4 \end{align}\] More generally, for some matrix \(\boldsymbol{M}\), some function \(h\) and some vector \(\boldsymbol{z}\) depending on \(\boldsymbol{W}_{t,u,i}\), we have \[\begin{align} \mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \cdot \sup_{u \in (0,1)} |\boldsymbol{M}_{ii} h \cdot \boldsymbol{z}_i|\biggr] \lesssim\|\boldsymbol{M}\|_{\text{op}} \sup_i\mathbb{E}\biggl[ \sup_{u \in (0,1)} h^4\biggr]^{1/4}\cdot \sum_{i=1}^p \mathbb{E}\biggl[ \sup_{u \in (0,1)} z_i^2\biggr]^{1/2} \end{align}\] Because all such functions \(h\) for us consist of some product of the functions \(g_{\boldsymbol{u}}\), \(g_{\boldsymbol{v}}\), \(g_{\boldsymbol{A}}\), \(g_{\boldsymbol{B}}\), \(g_{\boldsymbol{C}}\), we immediately have that our bound for \(\sup_i \mathbb{E}[\sup_u h^4]^{1/4}\) is the same as the bound we would yield above from \(\sup_i \mathbb{E}[\sup_u |h|]\).

Now, we handle the last term \(\sum_{i=1}^p \mathbb{E}[\sup_u z_i^2]^{1/2}\) for all possible \(\boldsymbol{z}\) in our bounds:

• If there is only one element (either random or deterministic; e.g., \(z_i =v_i^m\) or \(z_i = (\boldsymbol{A} \boldsymbol{w})_i^m\)), we apply Cauchy-Schwarz with the \(1\) vector. For random quantities, we then apply Lemma 34. The resulting expression is of the form \(\sqrt{p} \cdot \|\boldsymbol{v}\|_{2m}^m \leq \sqrt{p} \|\boldsymbol{v}\|_2^m\) if the vector is deterministic or \(\sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2}\) if it is deterministic: i.e., \[\begin{align} \sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}| \biggr]^{1/2} &\leq \biggl(\sum_{i=1}^p 1^2 \biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}|\biggr] \biggr)^{1/2} \\ &\lesssim\sqrt{p} (\|\boldsymbol{A}\|_{\text{op}}^{2m} p^{m})^{1/2} = \sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \end{align}\] and similarly for deterministic vectors, where we crucially use the fact that \(\|\boldsymbol{v}\|_m \leq \|\boldsymbol{v}\|_q\) for \(m \geq q\).

• If there is one deterministic element and one random element (e.g., \(v_i^q (\boldsymbol{A} \boldsymbol{w})_i^m\)), then we can apply Cauchy-Schwarz and then bound the random quantity using Lemma 34. The resulting expression is of the form \(\|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2}\): i.e., \[\begin{align} \sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |v_i^{2q}(\boldsymbol{A} \boldsymbol{w})_i^{2m}| \biggr]^{1/2} &= \sum_{i=1}^p v_i^q \cdot \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}| \biggr]^{1/2}\\ &\leq \biggl(\sum_{i=1}^p v_i^{2q}\biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}|\biggr] \biggr)^{1/2} \\ &\lesssim\|\boldsymbol{v}\|_{2q}^{q}(\|\boldsymbol{A}\|_{\text{op}}^{2m} p^{m})^{1/2} \leq \|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \end{align}\]

• If there are two deterministic elements and one random element (e.g., \(u_i^t v_i^q (\boldsymbol{A} \boldsymbol{w})_i^m\)), then a second application of Cauchy-Schwarz on the deterministic element yields the bound \(\|\boldsymbol{u}\|_2^t \|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2}\): i.e., \[\begin{align} &\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |u_i^{2t} v_i^{2q}(\boldsymbol{A} \boldsymbol{w})_i^{2m}| \biggr]^{1/2} \\&= \sum_{i=1}^p u_i^t v_i^q \cdot \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}| \biggr]^{1/2}\\ &\leq \biggl(\sum_{i=1}^p u_i^{2t}v_i^{2q}\biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}|\biggr] \biggr)^{1/2} \\ &\leq \biggl(\sum_{i=1}^p u_i^{4t}\biggr)^{1/4}\biggl(\sum_{i=1}^p v_i^{4q}\biggr)^{1/4}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m}|\biggr] \biggr)^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_{4t}^t\|\boldsymbol{v}\|_{4q}^{q}(\|\boldsymbol{A}\|_{\text{op}}^{2m} p^{m})^{1/2} \leq \|\boldsymbol{u}\|_2^t \|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \end{align}\]

• If there is one deterministic element and two random elements (e.g., \(v_i^q (\boldsymbol{A} \boldsymbol{w})_i^m (\boldsymbol{B} \boldsymbol{w})_i^t\)), then (1) a second application of Cauchy-Schwarz on the shared expectation and (2) a third application of Cauchy-Schwarz on the sum yields the bound \(\|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \|\boldsymbol{B}\|_{\text{op}}^t p^{t/2}\): i.e., \[\begin{align} \sum_{i=1}^p &\mathbb{E}\biggl[\sup_{u \in (0,1)} |v_i^{2q}(\boldsymbol{A} \boldsymbol{w})_i^{2m} (\boldsymbol{B} \boldsymbol{w})_i^{2t}| \biggr]^{1/2} \\&= \sum_{i=1}^p v_i^q \cdot \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m} (\boldsymbol{B} \boldsymbol{w})_i^{2t}| \biggr]^{1/2}\\ &\leq \biggl(\sum_{i=1}^p v_i^{2q}\biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2m} (\boldsymbol{B} \boldsymbol{w})_i^{2t}|\biggr] \biggr)^{1/2} \\ &\leq \biggl(\sum_{i=1}^p v_i^{2q}\biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{4m}|\biggr]^{1/2} \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{B} \boldsymbol{w})_i^{4t}|\biggr]^{1/2} \biggr)^{1/2} \\ &\leq \biggl(\sum_{i=1}^p v_i^{2q}\biggr)^{1/2}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{4m}|\biggr]\biggr)^{1/4} \biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{B} \boldsymbol{w})_i^{4t}|\biggr] \biggr)^{1/4} \\ &\lesssim\|\boldsymbol{v}\|_{2q}^{q}(\|\boldsymbol{A}\|_{\text{op}}^{4m} p^{2m})^{1/4} (\|\boldsymbol{B}\|_{\text{op}}^{4t} p^{2t})^{1/4} \leq \|\boldsymbol{v}\|_2^q \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \|\boldsymbol{B}\|_{\text{op}}^t p^{t/2} \end{align}\]

• Similarly, using the same techniques (i.e., Cauchy-Schwarz with the 1 vector, Cauchy-Schwarz in the expectation, Cauchy-Schwarz in the sum), \[\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A}\boldsymbol{w})_i^{2m} (\boldsymbol{B} \boldsymbol{w})_i^{2t}|\biggr]^{1/2} \leq \sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^m p^{m/2} \|\boldsymbol{B}\|_{\text{op}}^t p^{t/2}\]

• For no deterministic elements but three random elements (e.g., \((\boldsymbol{A} \boldsymbol{w})_i^q (\boldsymbol{B} \boldsymbol{w})_i^m (\boldsymbol{C} \boldsymbol{w})_i^t\)), we can apply Cauchy-Schwarz once and apply Hölder’s inequality twice to yield the bound \(\sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^q p^{q/2} \|\boldsymbol{B}\|_{\text{op}}^m p^{m/2} \|\boldsymbol{C}\|_{\text{op}}^t p^{t/2}\): i.e., \[\begin{align} &\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2q} (\boldsymbol{B} \boldsymbol{w})_i^{2m}(\boldsymbol{C} \boldsymbol{w})_i^{2t}| \biggr]^{1/2} \\ &\leq\sqrt{p}\biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} |(\boldsymbol{A} \boldsymbol{w})_i^{2q} (\boldsymbol{B} \boldsymbol{w})_i^{2m} (\boldsymbol{C} \boldsymbol{w})_i^{2t}|\biggr] \biggr)^{1/2} \\ &\leq \sqrt{p} \biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{A} \boldsymbol{w})_i^{6q}\biggr]^{1/3} \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{B} \boldsymbol{w})_i^{6m}\biggr]^{1/3} \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{C} \boldsymbol{w})_i^{6t}\biggr]^{1/3}\biggr)^{1/2} \\ &\leq \sqrt{p} \biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{A} \boldsymbol{w})_i^{6q}\biggr]\biggr)^{1/6} \biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{B} \boldsymbol{w})_i^{6m}\biggr]\biggr)^{1/6} \\ &\qquad \biggl(\sum_{i=1}^p \mathbb{E}\biggl[\sup_{u \in (0,1)} (\boldsymbol{C} \boldsymbol{w})_i^{6t}\biggr]\biggr)^{1/6} \\ &\leq \sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^q p^{q/2} \|\boldsymbol{B}\|_{\text{op}}^m p^{m/2} \|\boldsymbol{C}\|_{\text{op}}^t p^{t/2}. \end{align}\]

Therefore, we have \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r\sup_{u \in (0,1)}|\partial_i^3 f_2(\boldsymbol{W}_{t,u,i}) | \\ &\lesssim\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{v}\|_2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2\cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}}^3 p^{3/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2\|\boldsymbol{v}\|_2 \cdot \sqrt{p}\|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{v}\|_2 \cdot \sqrt{p}\|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}}\|\boldsymbol{u}\|_2 \cdot \sqrt{p} \|\boldsymbol{v}\|_2 \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2\|\boldsymbol{v}\|_2 \cdot \sqrt{p}\|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 (\|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}}^2 p + \|\boldsymbol{A}\|_{\text{op}}^3 p^2) \end{align}\] Similarly, \[\begin{align} & \mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_3(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \cdot \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2\|\boldsymbol{v}\|_2\|\boldsymbol{A}||_{\text{op}} p^{1/2} \end{align}\] and \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_4(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \cdot \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p}\|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} +\\ &\qquad \|\boldsymbol{v}\|_2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\| \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{B} \|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{v}\|_{2} \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 (\|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} p^{3/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}}^2 p^2 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}}^3 p^3) \end{align}\] and \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_5(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \cdot \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}}^2 p^2 \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}}^2 p^2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p^2 \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}}^3 p^3 \sqrt{p} \|\boldsymbol{B}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{B}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}}^2 p^2 \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}}^2 p \\ &\qquad + \| \boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{B}\|_{\text{op}}^3 p^{3/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{u}\|_2 \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}}^2 p^2 \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 (\|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}}^4 p^4 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}}^6 p^6) \end{align}\] and \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_6(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \cdot \sqrt{p}\|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\| \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\|_2^2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}}^3 p^{3/2} + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2^2 (\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} p^{3/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} p^{5/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}}^2 p^3 \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}}^3 p^{7/2}) \end{align}\] and \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_7(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2\|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{v}\|_2 + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{v}\|_2 \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\|_2 \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}}p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^3 p^3 \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}}\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{C}\|_{\text{op}}^2 p + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{v}\|_2 \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}}^2 p \\ &\qquad + \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}}^3 p^{3/2} \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{v}\|_{\text{op}} \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_{\text{op}} \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{v}\|_2 \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &\qquad +\|\boldsymbol{C}\|_{\text{op}}\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\qquad + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2 \|\boldsymbol{v}\|_2 (\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} p^{3/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}}^2 p^{7/2} \\ &\qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}}^4 p^5 + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}}^6 p^7 \end{align}\] and finally \[\begin{align} &\mathbb{E}\biggl[\sum_{i=1}^p |W_i|^r \sup_{u \in (0,1)}|\partial_i^3 f_3(\boldsymbol{w})|\biggr] \\ &\lesssim\|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ & + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{u}\|_2 \\ & + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ & + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{B}\|_{\text{op}} p^{1/2} + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p}\|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{u}\|_2^2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} + \|\boldsymbol{u}\|_2 \|\boldsymbol{C}\|_{\text{op}}p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2^2 \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{D}\|_{\text{op}}^2 p \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p \\ & + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p\\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^3 p^3 \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}}^3 p^{3/2} \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}}^3 p^{3/2} \\ &+ \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \|\boldsymbol{u}\|_2 \|\boldsymbol{D}\|_{\text{op}}^2 p \\ & + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p \\ & + \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \|\boldsymbol{D}\|_{\text{op}}^2 p \\ &+ \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}}^3 p^{3/2} \\ & + \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{u}\|_2 \\ &+ \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{A}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{B}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{C}\|_{\text{op}} p^{1/2} \\ & + \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \|\boldsymbol{D}\|_{\text{op}}^2 p^2 \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &+ \|\boldsymbol{D}\|_{\text{op}} \|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} p \|\boldsymbol{B}\|_{\text{op}} p \|\boldsymbol{C}\|_{\text{op}} p \cdot \sqrt{p} \|\boldsymbol{D}\|_{\text{op}} p^{1/2} \\ &\lesssim\|\boldsymbol{u}\|_2^2(\|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} p^{5/2} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{D}\|_{\text{op}}^2 p^{9/2} \\ &\qquad \qquad + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{D}\|_{\text{op}}^4 p^{6} + \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} \|\boldsymbol{D}\|_{\text{op}}^6 p^{8}) \end{align}\]

We are now ready to bound each term from the Taylor expansions.

14.5 Bounding \(B_1\) terms↩︎

14.5.1 Swapping \(\boldsymbol{Z}^{(1)}\)↩︎

Note that \[\begin{gather} \|\boldsymbol{u}^{(a,1)}\|_2 \lesssim\lambda n^{-1} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(a,1)}\|_2 \lesssim\lambda^{-2} \|\boldsymbol{\beta}^{(2)}\|_{\text{op}}, \quad \|\boldsymbol{A}^{(a,1)}\|_{\text{op}} \lesssim n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(a,2)}\|_2 \lesssim\lambda n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(a,2)}\|_2 \lesssim\lambda^{-1} \|\boldsymbol{\beta}^{(2)}\|_2 \\ \|\boldsymbol{A}^{(a,2)}\|_{\text{op}} \lesssim\lambda^{-2}, \quad \|\boldsymbol{B}^{(a,2)}\|_{\text{op}} \lesssim n^{-1} \lambda^{-1} \end{gather}\]

14.5.1.1 First term

From the first moment bounds for \(f_2\), we have \[\begin{align} &|\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(a,1)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(a,1)})]| \\ &\lesssim\|\boldsymbol{u}^{(a,1)}\|_2 \|\boldsymbol{v}^{(a,1)}\|_2 (\|\boldsymbol{A}^{(a,1)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(a,1)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(a,1)}\|_{\text{op}}^3 p^2) \\ &\lesssim\lambda^{-2} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2^2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \end{align}\] and similarly, from the second moment bounds for \(f_2\), \[\begin{align} \max\{\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(a,1)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(a,1)})^2]\} &\lesssim\|\boldsymbol{u}^{(a,1)}\|_2^2 \|\boldsymbol{v}^{(a,1)}\|_2^2 \\ &\lesssim\lambda^{-2} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2^4 \end{align}\]

14.5.1.2 Second term

From the first moment bounds for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(a,2)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(a,2)})]| \\ &\lesssim\|\boldsymbol{u}^{(a,2)}\|_2 \|\boldsymbol{v}^{(a,2)}\|_2 \|\boldsymbol{A}^{(a,2)}\|_{\text{op}} ( p^{1/2} + \|\boldsymbol{B}^{(a,2)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(a,2)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(a,2)}\|_{\text{op}}^6 p^6) \\ &\lesssim\lambda n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \cdot \lambda^{-1}\|\boldsymbol{\beta}^{(2)}\|_2 \lambda^{-2}( p^{1/2} + n^{-2} \lambda^{-2} p^{5/2} + n^{-4} \lambda^{-4} p^4 + n^{-6}\lambda^{-6} p^6) \\ &\lesssim\lambda^{-2} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2^2 (1 + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \end{align}\] and similarly, from the second moment bounds for \(f_5\), we have \[\begin{align} \max\{\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(a,2)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(a,2)})^2]\} &\lesssim\|\boldsymbol{u}^{(a,2)}\|_2^2 \|\boldsymbol{v}^{(a,2)}\|_2^2 \|\boldsymbol{A}^{(a,2)}\|_{\text{op}}^2 p^2 \\ &\lesssim\lambda^{-4} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2^4 \end{align}\]

14.5.1.3 Combined

Combining all of these bounds with the bounds above demonstrate that \[\begin{align} |&\mathbb{E}[\varphi(B_1(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))]| \\ &\lesssim\lambda^{-2} n^{-1/2}\|\boldsymbol{\beta}^{(2)}\|_2^2( (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &+ n^{-1/2}(1 + \lambda^{-2}) \|\boldsymbol{\beta}^{(2)}\|_2^2) \end{align}\]

14.5.2 Swapping \(\boldsymbol{Z}^{(2)}\)↩︎

The same bounds hold. That is, \[\begin{align} |&\mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim\lambda^{-2} n^{-1/2}\|\boldsymbol{\beta}^{(2)}\|_2^2( (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &+ n^{-1/2}(1 + \lambda^{-2}) \|\boldsymbol{\beta}^{(2)}\|_2^2) \end{align}\]

14.5.3 Universality for \(B_1\)↩︎

Therefore, for any bounded function \(\varphi\) with bounded first and second derivatives, we have \[\begin{align} |&\mathbb{E}[\varphi(B_1(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim\lambda^{-2} n^{-1/2}\|\boldsymbol{\beta}^{(2)}\|_2^2( (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &+ n^{-1/2}(1 + \lambda^{-2}) \|\boldsymbol{\beta}^{(2)}\|_2^2) \end{align}\]

14.6 Bounding \(B_3\) terms↩︎

14.6.1 Swapping \(\boldsymbol{Z}^{(1)}\)↩︎

We first bound \[\begin{gather} \|\boldsymbol{u}^{(b,1)}\|_2 \lesssim\lambda^{-1} n^{-1} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,1)}\|_2 \lesssim\|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{u}^{(b,2)}\|_2 \lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,2)}\|_2 \lesssim\lambda^{-1} \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \\ \|\boldsymbol{A}^{(b,2)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1}, \quad \|\boldsymbol{u}^{(b,3)}\|_2 \lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,3)}\|_2 \lesssim\|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(b,3)}\|_{\text{op}} \lesssim\lambda^{-1}, \quad \|\boldsymbol{B}^{(b,3)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(b,4)}\|_2 \lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,4)}\|_2 \lesssim\lambda^{-2} \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{A}^{(b,4)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(b,5)}\|_2 \lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,5)}\|_2 \lesssim\|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(b,5)}\|_{\text{op}} \lesssim\lambda^{-2}, \quad \|\boldsymbol{B}^{(b,5)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(b,6)}\|_2 \lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,6)}\|_2 \lesssim\lambda^{-1} \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_{2}\\ \|\boldsymbol{A}^{(b,6)}\|_{\text{op}} \lesssim\lambda^{-2}, \quad \|\boldsymbol{B}^{(b,6)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(b,7)}\|_2 \lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(b,7)}\|_2 \lesssim\|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(b,7)}\|_{\text{op}} \lesssim\lambda^{-2}, \quad \|\boldsymbol{B}^{(b,7)}\|_{\text{op}} \lesssim\lambda^{-1} \\ \|\boldsymbol{C}^{(b,7)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \end{gather}\]

14.6.1.1 First term

Now, using the first moment bound for \(f_1\), we have \[|\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,1)}) - f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,1)})]| = 0\] and similarly, using the second moment bound for \(f_1\), we have \[\begin{align} \max&\{\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,1)})^2], \mathbb{E}_{1,k}[f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,1)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,1)}\|_2^2 \|\boldsymbol{v}^{(b,1)}\|_2^2 \\ &\leq \lambda^{-2} n^{-2}\|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \end{align}\]

14.6.1.2 Second term

Using the first moment bound for \(f_2\), we have \[\begin{align} &|\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,2)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,2)})]| \\&\lesssim\|\boldsymbol{u}^{(b,2)}\|_2 \|\boldsymbol{v}^{(b,2)}\|_2 ( \|\boldsymbol{A}^{(b,2)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(b,2)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(b,2)}\|_{\text{op}}^3 p^2) \\ &\lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \cdot \lambda^{-1} \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ &\qquad \cdot (\lambda^{-1} n^{-1} p^{1/2} + \lambda^{-2} n^{-2} p + \lambda^{-3} n^{-3} p^2) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \\ &\qquad \cdot (1+ \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,2)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,2)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,2)}\|_2^2 \|\boldsymbol{v}^{(b,2)}\|_2^2 \\ &\lesssim\lambda^{-4} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \cdot \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4. \end{align}\]

14.6.1.3 Third term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,3)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,3)})]| \\ &\lesssim\|\boldsymbol{u}^{(b,3)}\|_2 \|\boldsymbol{v}^{(b,3)}\|_2 \|\boldsymbol{A}^{(b,3)}\|_{\text{op}}( p^{1/2} + \|\boldsymbol{B}^{(b,3)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(b,3)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(b,3)}\|_{\text{op}}^3 p^3) \\ &\lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-1} (p^{1/2} + \lambda^{-1} n^{-1} p^{3/2} + \lambda^{-2} n^{-2} p^2 + \lambda^{-3} n^{-3} p^3) \\ &\lesssim\lambda^{-2} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} + \lambda^{-2} n^{-1/2} + \lambda^{-3} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,3)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,3)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,3)}\|_2^2 \|\boldsymbol{v}^{(b,3)}\|_2^2 \|\boldsymbol{A}^{(b,3)}\|_{\text{op}}^2 p^2 \\ &\lesssim(\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2)^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \lambda^{-2} p^2 \\ &= \lambda^{-4} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \end{align}\]

14.6.1.4 Fourth term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,4)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,4)})]| \\ &\lesssim\|\boldsymbol{u}^{(b,4)}\|_2 \|\boldsymbol{v}^{(b,4)}\|_2 ( \|\boldsymbol{A}^{(b,4)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(b,4)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(b,4)}\|_{\text{op}}^3 p^2) \\ &\lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \lambda^{-2} \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 (\lambda^{-1} n^{-1} p^{1/2} + \lambda^{-2} n^{-2} p + \lambda^{-3} n^{-3} p^2) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,4)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,4)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,4)}\|_2^2 \|\boldsymbol{v}^{(b,4)}\|_2^2 \\ &\lesssim\lambda^{-4} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 \end{align}\]

14.6.1.5 Fifth term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,5)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,5)})]| \\ &\lesssim\|\boldsymbol{u}^{(b,5)}\|_2 \|\boldsymbol{v}^{(b,5)}\|_2 \|\boldsymbol{A}^{(b,5)}\|_{\text{op}}( p^{1/2} + \|\boldsymbol{B}^{(b,5)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(b,5)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(b,5)}\|_{\text{op}}^3 p^3) \\ &\lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} ( p^{1/2} + \lambda^{-1} n^{-1} p^{3/2} + \lambda^{-2} n^{-2} p^2 + \lambda^{-3} n^{-3} p^3) \\ &\lesssim\lambda^{-2} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 ( 1 + \lambda^{-1} + \lambda^{-2} n^{-1/2} + \lambda^{-3} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,5)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,5)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,5)}\|_2^2 \|\boldsymbol{v}^{(b,5)}\|_2^2 \|\boldsymbol{A}^{(b,5)}\|_{\text{op}}^2 p^2 \\ &\lesssim\lambda^{-4} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}^{(2)}\|_2^2 \end{align}\]

14.6.1.6 Sixth term

Using the first moment bound for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,6)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,6)})]| \\ &\lesssim\|\boldsymbol{u}^{(b,6)}\|_2 \|\boldsymbol{v}^{(b,6)}\|_2 \|\boldsymbol{A}^{(b,6)}\|_{\text{op}}(p^{1/2} + \|\boldsymbol{B}^{(b,6)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(b,6)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(b,6)}\|_{\text{op}}^6 p^6) \\ &\lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2 \lambda^{-1} \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_{2} \lambda^{-2}(p^{1/2} + \lambda^{-2} n^{-2} p^{5/2} + \lambda^{-4} n^{-4} p^4 + \lambda^{-6} n^{-6} p^6) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_{2} \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^2 (1 + \lambda^{-2} + \lambda^{-4} n^{-1/2}+ \lambda^{-6} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_5\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,6)})^2], \mathbb{E}_{1,k}[f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,6)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,6)}\|_2^2 \|\boldsymbol{v}^{(b,6)}\|_2^2 \|\boldsymbol{A}^{(b,6)}\|_{\text{op}}^2 p^2 \\ &\lesssim n^{-6} \|\boldsymbol{\beta}^{(2)}\|_2^2 \cdot (\lambda^{-1} \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_{2})^2 \lambda^{-4} p^2 \\ &\lesssim\lambda^{-6} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_{2}^2 \cdot \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^4 \end{align}\]

14.6.1.7 Seventh term

Using the first moment bound for \(f_7\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,7)}) - f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,7)})]| \\ &\lesssim\|\boldsymbol{u}^{(b,7)}\|_2 \|\boldsymbol{v}^{(b,7)}\|_2 \|\boldsymbol{A}^{(b,7)}\|_{\text{op}} \|\boldsymbol{B}^{(b,7)}\|_{\text{op}} \\ &\qquad ( p^{3/2} + \|\boldsymbol{C}^{(b,7)}\|_{\text{op}}^2 p^{7/2} + \|\boldsymbol{C}^{(b,7)}\|_{\text{op}}^4 p^5 + \|\boldsymbol{C}^{(b,7)}\|_{\text{op}}^6 p^7) \\ &\lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} \lambda^{-1} ( p^{3/2} + \lambda^{-2} n^{-2} p^{7/2} + \lambda^{-4} n^{-4} p^5 + \lambda^{-6} n^{-6} p^7) \\ &\lesssim\lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 ( 1 + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_7\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(b,7)})^2], \mathbb{E}_{1,k}[f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(b,7)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(b,7)}\|_2^2 \|\boldsymbol{v}^{(b,7)}\|_2^2 \|\boldsymbol{A}^{(b,7)}\|_{\text{op}}^2 \|\boldsymbol{B}^{(b,7)}\|_{\text{op}}^2 p^4 \\ &\lesssim\lambda^{-6} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \end{align}\]

14.6.1.8 Combined

It follows from the above bounds that \[\begin{align} |&\mathbb{E}[\varphi(B_3(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))]| \\ &\lesssim\mathbb{E}[\lambda^{-2} n^{-1}\|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 \cdot (1+ \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \cdot \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 \\ &\qquad + \lambda^{-2} n^{-1/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} + \lambda^{-2} n^{-1/2} + \lambda^{-3} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-1} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 \\ &\qquad + \lambda^{-2} n^{-1/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 ( 1 + \lambda^{-1} + \lambda^{-2} n^{-1/2} + \lambda^{-3} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-1} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}^{(2)}\|_2^2 \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_{2} \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^2 (1 + \lambda^{-2} + \lambda^{-4} n^{-1/2}+ \lambda^{-6} n^{-1/2}) \\ &\qquad + \lambda^{-6} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_{2}^2 \cdot \|\boldsymbol{X}^{(1)\setminus k}\|_{\text{op}}^4 \\ &\qquad + \lambda^{-3} n^{-1/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 ( 1 + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &\qquad + \lambda^{-6} n^{-1} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2] \\ &\lesssim\lambda^{-2} n^{-1}\|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + \lambda^{-4} + (\lambda^{-2} + \lambda^{-4}) n^{-2} \mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4]) \\ &\qquad + \lambda^{-3} n^{-1/2} \|\boldsymbol{\beta}^{(2)}\|_2\|\tilde{\boldsymbol{\beta}}\|_2 ((1 + \lambda^{-1} + \lambda^{-2} + (\lambda^{-3} + \lambda^{-4}+ \lambda^{-6}) n^{-1/2}) \\ &\qquad \qquad + (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) n^{-1} \mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2]) \\ &\lesssim\lambda^{-2} n^{-1} \log^2 n \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + \lambda^{-4}) \\ &\qquad + \lambda^{-3} n^{-1/2} \log n \|\boldsymbol{\beta}^{(2)}\|_2\|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} + \lambda^{-2} + (\lambda^{-3} + \lambda^{-4}+ \lambda^{-6}) n^{-1/2}) \end{align}\]

14.6.2 Swapping \(\boldsymbol{Z}^{(2)}\)↩︎

Note that \[\begin{gather} \|\boldsymbol{u}^{(B,1)}\|_2 \lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(B,1)}\|_2 \lesssim\lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(B,1)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(B,2)}\|_2 \lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(B,2)}\|_2 \lesssim\lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(B,2)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \\ \|\boldsymbol{u}^{(B,3)}\|_2 \lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2, \quad \|\boldsymbol{v}^{(B,3)}\|_2 \lesssim\lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(B,3)}\|_{\text{op}} \lesssim\lambda^{-2}, \quad \|\boldsymbol{B}^{(B,3)}\|_{\text{op}} \lesssim\lambda^{-1} n^{-1} \end{gather}\]

14.6.2.1 First term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{2,k}[f_2(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,1)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,1)})]| \\ &\lesssim\|\boldsymbol{u}^{(B,1)}\|_2 \|\boldsymbol{v}^{(B,1)}\|_2 ( \|\boldsymbol{A}^{(B,1)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(B,1)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(B,1)}\|_{\text{op}}^3 p^2) \\ &\lesssim\lambda^{-1} n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2 \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 ( \lambda^{-1} n^{-1} p^{1/2} + \lambda^{-2} n^{-2} p + \lambda^{-3} n^{-3} p^2) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 ( 1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{2,k}[f_2(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,1)})^2], \mathbb{E}_{2,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,1)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(B,1)}\|_2^2 \|\boldsymbol{v}^{(B,1)}\|_2^2 \\ &\lesssim\lambda^{-4} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \end{align}\]

14.6.2.2 Second term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{2,k}[f_2(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,2)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,2)})]| \\ &\lesssim\|\boldsymbol{u}^{(B,2)}\|_2 \|\boldsymbol{v}^{(B,2)}\|_2 ( \|\boldsymbol{A}^{(B,2)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(B,2)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(B,2)}\|_{\text{op}}^3 p^2) \\ &\lesssim n^{-2} \|\boldsymbol{\beta}^{(2)}\|_2\lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 (\lambda^{-1} n^{-1} p^{1/2} + \lambda^{-2} n^{-2} p + \lambda^{-3} n^{-3} p^2) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{2,k}[f_2(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,2)})^2], \mathbb{E}_{2,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,2)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(B,2)}\|_2^2 \|\boldsymbol{v}^{(B,2)}\|_2^2 \\ &\lesssim\lambda^{-4} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \end{align}\]

14.6.2.3 Third term

Using the first moment bound for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{2,k}[f_5(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,3)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,3)})]| \\ &\lesssim\|\boldsymbol{u}^{(B,3)}\|_2 \|\boldsymbol{v}^{(B,3)}\|_2 \|\boldsymbol{A}^{(B,3)}\|_{\text{op}}(p^{1/2} + \|\boldsymbol{B}^{(B,3)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(B,3)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(B,3)}\|_{\text{op}}^6 p^6) \\ &\lesssim n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2 \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} (p^{1/2} + \lambda^{-2} n^{-2} p^{5/2} + \lambda^{-4} n^{-4} p^4 + \lambda^{-6} n^{-6} p^6) \\ &\lesssim\lambda^{-3} n^{-5/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 (1 + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \end{align}\] and using the second moment bound for \(f_5\) yields \[\begin{align} \max\{&\mathbb{E}_{2,k}[f_5(\boldsymbol{Z}_k^{(2)}; \boldsymbol{S}^{(B,3)})^2], \mathbb{E}_{2,k}[f_5(\tilde{\boldsymbol{Z}}_k^{(2)}; \boldsymbol{S}^{(B,3)})^2]\} \\ &\lesssim\lambda^{-6} n^{-4} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \end{align}\]

14.6.2.4 Combined

It follows from the above bounds that \[\begin{align} |&\mathbb{E}[\varphi(B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_3(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim\mathbb{E}[\lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 ( 1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} n^{-1/2}) \\ &\qquad + \lambda^{-4} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 (1 + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2})\\ &\qquad + \lambda^{-6} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 ] \\ &\lesssim\lambda^{-4} n^{-3} \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4] (1 + \lambda^{-2}) \\ &\qquad + \lambda^{-3} n^{-3/2} \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2] \\ &(1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &\lesssim\lambda^{-4} n^{-1} \log^2 n \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2}) \\ &\qquad + \lambda^{-3} n^{-1/2} \log n \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \end{align}\]

14.6.3 Universality for \(B_3\)↩︎

Therefore, for any bounded function \(\varphi\) with bounded first and second derivatives, we have \[\begin{align} |&\mathbb{E}[\varphi(B_3(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_3(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim\lambda^{-2} n^{-1} \log^2 n \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + \lambda^{-4}) \\ &\qquad + \lambda^{-3} n^{-1/2} \log n \|\boldsymbol{\beta}^{(2)}\|_2\|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} + \lambda^{-2} + (\lambda^{-3} + \lambda^{-4}+ \lambda^{-6}) n^{-1/2})\\ &\qquad + \lambda^{-4} n^{-1} \log^2 n \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2}) \\ &\qquad + \lambda^{-3} n^{-1/2} \log n \|\boldsymbol{\beta}^{(2)}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} n^{-1/2} + \lambda^{-2} + \lambda^{-4} n^{-1/2} + \lambda^{-6} n^{-1/2}) \\ &\lesssim\lambda^{-2} n^{-1} \log^2 n \cdot \|\boldsymbol{\beta}^{(2)}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + \lambda^{-4}) \\ &\qquad + \lambda^{-3} n^{-1/2} \log n \cdot \|\boldsymbol{\beta}^{(2)}\|_2\|\tilde{\boldsymbol{\beta}}\|_2 (1 + \lambda^{-1} + \lambda^{-2} + (\lambda^{-3} + \lambda^{-4}+ \lambda^{-6}) n^{-1/2}) \end{align}\]

14.7 Bounding \(B_2\) terms↩︎

14.7.1 Swapping \(\boldsymbol{Z}^{(1)}\)↩︎

We first bound \[\begin{gather} \|\boldsymbol{u}^{(c,1)}\|_2 \lesssim n^{-2} \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,1)}\|_2 \lesssim\|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{u}^{(c,2)}\|_2 \lesssim n^{-3} \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,2)}\|_2 = \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \\ \|\boldsymbol{A}^{(c,2)}\|_{\text{op}}= n^{-1} \lambda^{-1}, \quad \|\boldsymbol{u}^{(c,3)}\|_2 = n^{-3}\lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,3)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(c,3)}\|_{\text{op}} = \lambda^{-1} , \quad \|\boldsymbol{B}^{(c,3)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(c,4)}\|_2 = n^{-3} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,4)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(c,4)}\|_{\text{op}} = \lambda^{-2} , \quad \|\boldsymbol{B}^{(c,4)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(c,5)}\|_2 = n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,5)}\|_2 = \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(c,5)}\|_{\text{op}} = \lambda^{-2}, \quad \|\boldsymbol{B}^{(c,5)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(c,6)}\|_2 = n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,6)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(c,6)}\|_{\text{op}} = \lambda^{-2} , \quad \|\boldsymbol{B}^{(c,6)}\|_{\text{op}} = \lambda^{-1}, \quad \|\boldsymbol{C}^{(c,6)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(c,7)}\|_2 = n^{-2} \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(c,7)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2 ,\quad \|\boldsymbol{A}^{(c,7)}\|_{\text{op}} = \lambda^{-2} \\ \|\boldsymbol{u}^{(c,8)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{A}^{(c,8)}\|_{\text{op}} = n^{-3} \lambda^{-1}, \quad \|\boldsymbol{B}^{(c,8)}\|_{\text{op}} = \lambda^{-2}, \quad \|\boldsymbol{C}^{(c,8)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(c,9)}\|_2 = \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{A}^{(c,9)}\|_{\text{op}} = n^{-4} \lambda^{-1} \\ \|\boldsymbol{B}^{(c,9)}\|_{\text{op}} = \lambda^{-2}, \quad \|\boldsymbol{C}^{(c,9)}\|_{\text{op}} = \lambda^{-1} , \quad \|\boldsymbol{D}^{(c,9)}\|_{\text{op}} = n^{-1} \lambda^{-1} \end{gather}\]

14.7.1.1 First term

Now, using the first moment bound for \(f_1\), we have \[|\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,1)}) - f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,1)})]| = 0\] and similarly, using the second moment bound for \(f_1\), we have \[\begin{align} \max&\{\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,1)})^2], \mathbb{E}_{1,k}[f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,1)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,1)}\|_2^2 \|\boldsymbol{v}^{(c,1)}\|_2^2 \\ &\lesssim\lambda^{-4}n^{-4} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.2 Second term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,2)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,2)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,2)}\|_2 \|\boldsymbol{v}^{(c,2)}\|_2 ( \|\boldsymbol{A}^{(c,2)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(c,2)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(c,2)}\|_{\text{op}}^3 p^2) \\ &\lesssim n^{-3} \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 ( n^{-1} \lambda^{-1} p^{1/2} + n^{-2} \lambda^{-2} p + n^{-3} \lambda^{-3} p^2) \\ &\lesssim n^{-7/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 ( \lambda^{-1} + \lambda^{-2}n^{-1/2} + \lambda^{-3} n^{-1/2} ) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,2)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,2)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,2)}\|_2^2 \|\boldsymbol{v}^{(c,2)}\|_2^2 \\ &\lesssim( n^{-3} \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 \cdot (\lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 \\ &\lesssim\lambda^{-6} n^{-6} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^8 \end{align}\]

14.7.1.3 Third term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,3)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,3)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,3)}\|_2 \|\boldsymbol{v}^{(c,3)}\|_2 \|\boldsymbol{A}^{(c,3)}\|_{\text{op}}( p^{1/2} + \|\boldsymbol{B}^{(c,3)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(c,3)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(c,3)}\|_{\text{op}}^3 p^3) \\ &\lesssim n^{-3}\lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-1}( p^{1/2} + n^{-1} \lambda^{-1} p^{3/2} + n^{-2} \lambda^{-2} p^2 + n^{-3} \lambda^{-3} p^3) \\ &\lesssim n^{-5/2}\lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \\ \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,3)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,3)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,3)}\|_2^2 \|\boldsymbol{v}^{(c,3)}\|_2^2 \|\boldsymbol{A}^{(c,3)}\|_{\text{op}}^2 p^2 \\ &\lesssim(n^{-3}\lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 (\|\tilde{\boldsymbol{\beta}}\|_2)^2 \lambda^{-2} p^2 \\ &\lesssim n^{-4}\lambda^{-6} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \end{align}\]

14.7.1.4 Fourth term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,4)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,4)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,4)}\|_2 \|\boldsymbol{v}^{(c,4)}\|_2 \|\boldsymbol{A}^{(c,4)}\|_{\text{op}}( p^{1/2} + \|\boldsymbol{B}^{(c,4)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(c,4)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(c,4)}\|_{\text{op}}^3 p^3) \\ &\lesssim n^{-3} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} ( p^{1/2} + n^{-1} \lambda^{-1} p^{3/2} + n^{-2} \lambda^{-2} p^2 + n^{-3} \lambda^{-3} p^3) \\ &\lesssim n^{-5/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,4)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,4)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,4)}\|_2^2 \|\boldsymbol{v}^{(c,4)}\|_2^2 \|\boldsymbol{A}^{(c,4)}\|_{\text{op}}^2 p^2 \\ &\lesssim(n^{-3} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \lambda^{-4} p^2 \\ &\lesssim n^{-4} \lambda^{-6} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.5 Fifth term

Using the first moment bound for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,5)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,5)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,5)}\|_2 \|\boldsymbol{v}^{(c,5)}\|_2 \|\boldsymbol{A}^{(c,5)}\|_{\text{op}}(p^{1/2} + \|\boldsymbol{B}^{(c,5)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(c,5)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(c,5)}\|_{\text{op}}^6 p^6) \\ &\lesssim n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} \\ &\qquad (p^{1/2} + n^{-2} \lambda^{-2} p^{5/2} + n^{-4} \lambda^{-4} p^4 + n^{-6} \lambda^{-6} p^6) \\ &\lesssim n^{-7/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 (1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6}) \end{align}\] and using the second moment bound for \(f_5\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,5)})^2], \mathbb{E}_{1,k}[f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,5)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,5)}\|_2^2 \|\boldsymbol{v}^{(c,5)}\|_2^2 \|\boldsymbol{A}^{(c,5)}\|_{\text{op}}^2 p^2 \\ &\lesssim(n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 (\lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2\lambda^{-4} p^2 \\ &\lesssim n^{-6} \lambda^{-8} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^8 \end{align}\]

14.7.1.6 Sixth term

Using the first moment bound for \(f_7\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,6)}) - f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,6)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,6)}\|_2 \|\boldsymbol{v}^{(c,6)}\|_2 \|\boldsymbol{A}^{(c,6)}\|_{\text{op}} \|\boldsymbol{B}^{(c,6)}\|_{\text{op}} \\ &\qquad ( p^{3/2} + \|\boldsymbol{C}^{(c,6)}\|_{\text{op}}^2 p^{7/2} + \|\boldsymbol{C}^{(c,6)}\|_{\text{op}}^4 p^5 + \|\boldsymbol{C}^{(c,6)}\|_{\text{op}}^6 p^7) \\ &\lesssim n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} \lambda^{-1} ( p^{3/2} + n^{-2} \lambda^{-2} p^{7/2} + n^{-4} \lambda^{-4} p^5 + n^{-6} \lambda^{-6} p^7) \\ &\lesssim n^{-5/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \end{align}\] and using the second moment bound for \(f_7\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,6)})^2], \mathbb{E}_{1,k}[f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,6)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,6)}\|_2^2 \|\boldsymbol{v}^{(c,6)}\|_2^2 \|\boldsymbol{A}^{(c,6)}\|_{\text{op}}^2 \|\boldsymbol{B}^{(c,6)}\|_{\text{op}}^2 p^4 \\ &\lesssim(n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \lambda^{-4} \lambda^{-2} p^4 \\ &\lesssim n^{-4} \lambda^{-8} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.7 Seventh term

Using the first moment bound for \(f_3\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_3(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,7)}) - f_3(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,7)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,7)}\|_2 \|\boldsymbol{v}^{(c,7)}\|_2 \|\boldsymbol{A}^{(c,7)}\|_{\text{op}} p^{1/2} \\ &\lesssim n^{-2} \|\tilde{\boldsymbol{\beta}}\|_2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} p^{1/2} \\ &\lesssim n^{-3/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 \end{align}\] and using the second moment bound for \(f_3\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_3(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,7)})^2], \mathbb{E}_{1,k}[f_3(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,7)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,7)}\|_2^2 \|\boldsymbol{v}^{(c,7)}\|_2^2 \|\boldsymbol{A}^{(c,7)}\|_{\text{op}}^2 p^2 \\ &\lesssim n^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{\beta}}\|_2^2 \lambda^{-4} p^2 \\ &\lesssim n^{-2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.8 Eighth term

Using the first moment bound for \(f_6\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_6(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,8)}) - f_6(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,8)})]| \\ &\lesssim\|\boldsymbol{u}^{(c,8)}\|_2^2\|\boldsymbol{A}^{(c,8)}\|_{\text{op}} \|\boldsymbol{B}^{(c,8)}\|_{\text{op}} ( p^{3/2} + \|\boldsymbol{C}^{(c,8)}\|_{\text{op}} p^{5/2} + \|\boldsymbol{C}^{(c,8)}\|_{\text{op}}^2 p^3 + \|\boldsymbol{C}^{(c,8)}\|_{\text{op}}^3 p^{7/2}) \\ &\lesssim\|\tilde{\boldsymbol{\beta}}\|_2^2 n^{-3} \lambda^{-1} \lambda^{-2} ( p^{3/2} + n^{-1} \lambda^{-1} p^{5/2} + n^{-2} \lambda^{-2} p^3 + n^{-3} \lambda^{-3} p^{7/2}) \\ &\lesssim n^{-3/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1} \lambda^{-3}) \end{align}\] and using the second moment bound for \(f_6\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_6(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,8)})^2], \mathbb{E}_{1,k}[f_6(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,8)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(c,8)}\|_2^4 \|\boldsymbol{A}^{(c,8)}\|_{\text{op}}^2 \|\boldsymbol{B}^{(c,8)}\|_{\text{op}}^2 p^4 \\ &\lesssim\|\tilde{\boldsymbol{\beta}}\|_2^4 (n^{-3} \lambda^{-1})^2 \lambda^{-4} p^4 \\ &\lesssim n^{-2} \lambda^{-6} \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.9 Ninth term

Using the first moment bound for \(f_8\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_8(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,8)}) - f_8(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,8)})]| \\ &\lesssim\|\boldsymbol{u}\|_2^2 \|\boldsymbol{A}\|_{\text{op}} \|\boldsymbol{B}\|_{\text{op}} \|\boldsymbol{C}\|_{\text{op}} (p^{5/2} + \|\boldsymbol{D}\|_{\text{op}}^2 p^{9/2} + \|\boldsymbol{D}\|_{\text{op}}^4 p^{6} + \|\boldsymbol{D}\|_{\text{op}}^6 p^{8}) \\ &\lesssim\|\tilde{\boldsymbol{\beta}}\|_2^2 n^{-4} \lambda^{-1} \lambda^{-2} \lambda^{-1} (p^{5/2} + n^{-2} \lambda^{-2} p^{9/2} + n^{-4} \lambda^{-4} p^{6} + n^{-6} \lambda^{-6} p^{8}) \\ &\lesssim n^{-3/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \end{align}\] and using the second moment bound for \(f_8\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_8(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(c,8)})^2], \mathbb{E}_{1,k}[f_8(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(c,8)})^2]\} \\ &\lesssim\|\boldsymbol{u}\|_2^4 \|\boldsymbol{A}\|_{\text{op}}^2 \|\boldsymbol{B}\|_{\text{op}}^2 \|\boldsymbol{C}\|_{\text{op}}^2 p^6 \\ &\lesssim\|\tilde{\boldsymbol{\beta}}\|_2^4 n^{-8} \lambda^{-2} \lambda^{-4} \lambda^{-2} p^6 \\ &\lesssim n^{-2} \lambda^{-8} \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.1.10 Combined

It follows from the above bounds that \[\begin{align} |&\mathbb{E}[\varphi(B_3(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_3(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))]| \\ &\lesssim\mathbb{E}[\lambda^{-4}n^{-3} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-5/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 ( \lambda^{-1} + \lambda^{-2}n^{-1/2} + \lambda^{-3} n^{-1/2} ) \\ &\qquad + \lambda^{-6} n^{-5} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^8 \\ &\qquad + n^{-3/2}\lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \\ &\qquad + n^{-3}\lambda^{-6} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \\ &\qquad + n^{-3/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3})\\ &\qquad + n^{-3} \lambda^{-6} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-5/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 (1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6}) \\ &\qquad + n^{-5} \lambda^{-8} \|\tilde{\boldsymbol{\beta}}\|_2^4 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^8 \\ &\qquad + n^{-3/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2 ( 1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} \\ &\qquad + n^{-3} \lambda^{-8} \|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad +n^{-1/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 + n^{-1} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-1/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1} \lambda^{-3}) + n^{-1} \lambda^{-6} \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-1/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) + n^{-1} \lambda^{-8} \|\tilde{\boldsymbol{\beta}}\|_2^4 ] \\ &\lesssim n^{-1} (\lambda^{-4} + \lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 + n^{-3} (\lambda^{-4} + \lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^4] \\ &\qquad + n^{-5} (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^8] \\ &\qquad + n^{-1/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-1} + \lambda^{-2} + \lambda^{-4} + n^{-1/2}(\lambda^{-3} + \lambda^{-6} + \lambda^{-7} + \lambda^{-8})) \\ &\qquad + n^{-3/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-1} + \lambda^{-3} + n^{-1/2} (\lambda^{-2} + \lambda^{-5} + \lambda^{-7})) \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1),\setminus k}\|_{\text{op}}^2] \\ &\qquad + n^{-5/2} \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (\lambda^{-1} + \lambda^{-3} + n^{-1/2} (\lambda^{-2} + \lambda^{-5} + \lambda^{-7})) \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1),\setminus}\|_{\text{op}}^4] \\ &\lesssim n^{-1} \log^2 n \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^4 + n^{-1} \log^4 n (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-1/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + n^{-1/2} \lambda^{-7} ) \\ &\qquad + n^{-1/2} \log n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \\ &\qquad + n^{-1/2} \log^2 n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (\lambda^{-1} + \lambda^{-3} + n^{-1/2} (\lambda^{-2} + \lambda^{-5} + \lambda^{-7})) \end{align}\]

14.7.2 Swapping \(\boldsymbol{Z}^{(2)}\)↩︎

We first bound \[\begin{gather} \|\boldsymbol{u}^{(C,1)}\|_2 \lesssim n^{-3} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(C,1)}\|_2 = \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(C,1)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(C,2)}\|_2 = n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2, \quad \|\boldsymbol{v}^{(C,2)}\|_2 = \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \\ \|\boldsymbol{A}^{(C,2)}\|_{\text{op}} = \lambda^{-2}, \quad \|\boldsymbol{B}^{(C,2)}\|_{\text{op}} = n^{-1} \lambda^{-1} \end{gather}\]

14.7.2.1 First term

From the first moment bounds for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(C,1)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(C,1)})]| \\ &\lesssim\|\boldsymbol{u}^{(C,1)}\|_2 \|\boldsymbol{v}^{(C,1)}\|_2 (\|\boldsymbol{A}^{(C,1)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(C,1)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(C,1)}\|_{\text{op}}^3 p^2) \\ &\lesssim n^{-3} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 (n^{-1} \lambda^{-1} p^{1/2} + n^{-2} \lambda^{-2} p + n^{-3} \lambda^{-3} p^2) \\ &\lesssim n^{-7/2} \lambda^{-4} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + n^{-1/2} \lambda^{-1} + n^{-1/2} \lambda^{-2}) \end{align}\] and similarly, from the second moment bounds for \(f_2\), \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(C,1)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(C,1)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(C,1)}\|_2^2 \|\boldsymbol{v}^{(C,1)}\|_2^2 \\ &\lesssim n^{-6} \lambda^{-6} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^8 \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.2.2 Second term

From the first moment bounds for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(C,2)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(C,2)})]| \\ &\lesssim\|\boldsymbol{u}^{(C,2)}\|_2 \|\boldsymbol{v}^{(C,2)}\|_2 \|\boldsymbol{A}^{(C,2)}\|_{\text{op}} ( p^{1/2} + \|\boldsymbol{B}^{(C,2)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(C,2)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(C,2)}\|_{\text{op}}^6 p^6) \\ &\lesssim n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2 \lambda^{-2} \\ &\qquad ( p^{1/2} + n^{-2} \lambda^{-2} p^{5/2} + n^{-4} \lambda^{-4} p^4 + n^{-6} \lambda^{-6} p^6) \\ &\lesssim n^{-7/2} \lambda^{-4} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4 \|\tilde{\boldsymbol{\beta}}\|_2^2 ( 1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6}) \end{align}\] and similarly, from the second moment bounds for \(f_5\), we have \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(C,2)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(C,2)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(C,2)}\|_2^2 \|\boldsymbol{v}^{(C,2)}\|_2^2 \|\boldsymbol{A}^{(C,2)}\|_{\text{op}}^2 p^2 \\ &\lesssim(n^{-4} \lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2(\lambda^{-1} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^2 \|\tilde{\boldsymbol{\beta}}\|_2)^2 \lambda^{-4} p^2 \\ &\lesssim n^{-6} \lambda^{-8} \|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^8 \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.2.3 Combined

Combining all of these bounds with the bounds above demonstrate that \[\begin{align} |&\mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_1(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim n^{-5/2} \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^4] (1 + \lambda^{-2} n^{-1/2} (\lambda^{-1} + \lambda^{-4} + \lambda^{-6})) \\ &\qquad + n^{-5} (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \mathbb{E}[\|\tilde{\boldsymbol{X}}^{(1)}\|_{\text{op}}^8] \\ &\lesssim n^{-1/2} \log^2 n \cdot \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} n^{-1/2} (\lambda^{-1} + \lambda^{-4} + \lambda^{-6})) \\ &\qquad + n^{-1} \log^4 n \cdot (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \end{align}\]

14.7.3 Universality for \(B_2\)↩︎

Therefore, for any bounded function \(\varphi\) with bounded first and second derivatives, we have \[\begin{align} |&\mathbb{E}[\varphi(B_2(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_2(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\lesssim n^{-1} \log^2 n \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^4 + n^{-1} \log^4 n (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-1/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + n^{-1/2} \lambda^{-7} ) \\ &\qquad + n^{-1/2} \log n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \\ &\qquad + n^{-1/2} \log^2 n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (\lambda^{-1} + \lambda^{-3} + n^{-1/2} (\lambda^{-2} + \lambda^{-5} + \lambda^{-7})) \\ &\qquad + n^{-1/2} \log^2 n \cdot \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + \lambda^{-2} n^{-1/2} (\lambda^{-1} + \lambda^{-4} + \lambda^{-6})) \\ &\qquad + n^{-1} \log^4 n \cdot (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\lesssim n^{-1} \log^2 n \lambda^{-4} \|\tilde{\boldsymbol{\beta}}\|_2^4 + n^{-1} \log^4 n (\lambda^{-6} + \lambda^{-8}) \|\tilde{\boldsymbol{\beta}}\|_2^4 \\ &\qquad + n^{-1/2} \lambda^{-2} \|\tilde{\boldsymbol{\beta}}\|_2^2 (1 + n^{-1/2} \lambda^{-7} ) \\ &\qquad + n^{-1/2} \log n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 \\ &\qquad + n^{-1/2} \log^2 n \lambda^{-3} \|\tilde{\boldsymbol{\beta}}\|_2^2 (\lambda^{-1} + \lambda^{-3} + n^{-1/2} (\lambda^{-2} + \lambda^{-5} + \lambda^{-7})) \end{align}\]

14.8 Variance term↩︎

The proof of universality for the variance term follows by the same procedure as the bias terms; notably, the variance term is remarkably similar to \(B_3\) and thus its proof is also quite similar. We include it here for completeness.

Let \(\boldsymbol{U} = \tilde{\boldsymbol{U}} \cdot \frac{p}{n}\), where \(\tilde{\boldsymbol{U}} \sim \text{Unif}(\mathcal{S}^{d-1})\) is drawn independently of everything else. Define \(V(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)})\) analogously to the bias quantities \(B_j(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)})\) and then define \(\tilde{V}(\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)}) = \sigma^2 \boldsymbol{U}^\top (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-2} \hat{\boldsymbol{\Sigma}}\boldsymbol{\Sigma}^{(2)} \boldsymbol{U}\). With these quantities, we can express \[\begin{align} V(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}) &= \frac{\sigma^2}{n} \textrm{Tr}[(\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-2} \hat{\boldsymbol{\Sigma}}\boldsymbol{\Sigma}^{(2)}] \\ &= \mathbb{E}_U[\sigma^2 \boldsymbol{U}^\top (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-2} \hat{\boldsymbol{\Sigma}}\boldsymbol{\Sigma}^{(2)} \boldsymbol{U}] \\ &= \mathbb{E}_{U}[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})] \end{align}\] and similarly \(V(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}) = \mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})]\), where the expectation \(\mathbb{E}_U\) is taken with respect to \(\boldsymbol{U}\). Then, it suffices to show that, for each \(i = 1, \dots, n\) and bounded function \(\varphi\) with bounded first and second derivative, we have \[|\mathbb{E}[\varphi(\mathbb{E}_{U}[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})] - \mathbb{E}[\varphi(\mathbb{E}_{U}[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})])]| \to 0.\]

14.8.1 Lindeberg and Sherman-Morrison expansions↩︎

As before, we have \[\begin{align} &|\mathbb{E}[\varphi(\mathbb{E}_{U}[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})])] - \mathbb{E}[\varphi(\mathbb{E}_{U}[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})])]|\\ &\leq \|\varphi'\|_\infty \sum_{k=1}^{n_1} \mathbb{E}[|\mathbb{E}_U\mathbb{E}_{1,k}[\tilde{V}(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - \tilde{V}(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)})]|] \\ &\qquad + \|\varphi''\|_\infty \sum_{k=1}^{n_1} \mathbb{E}[\mathbb{E}_U\mathbb{E}_{1,k}[(\tilde{V}(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - \tilde{V}(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2]] \\ &\qquad + \|\varphi''\|_\infty \sum_{k=1}^{n_1} \mathbb{E}[\mathbb{E}_U\mathbb{E}_{1,k}[(\tilde{V}(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - \tilde{V}(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}))^2]] \\ &|\mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})])] - \mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})])]| \\ &\leq \|\varphi'\|_\infty \sum_{k=1}^{n_2} \mathbb{E}[|\mathbb{E}_U\mathbb{E}_{2,k}[(\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}))] - \mathbb{E}[(\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}))]|] \\ &\qquad + \|\varphi''\|_\infty \sum_{k=1}^{n_2} \mathbb{E}[\mathbb{E}_U\mathbb{E}_{2,k}[(\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - \tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2]] \\ &\qquad + \|\varphi''\|_\infty \sum_{k=1}^{n_2} \mathbb{E}[\mathbb{E}_U\mathbb{E}_{2,k}[(\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k-1}) - \tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),\setminus k}))^2]] \end{align}\] Recall that, by Sherman-Morrison, we can express \[\begin{align} (\hat{\boldsymbol{\Sigma}}_k + \lambda \boldsymbol{I})^{-1} &= (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \tfrac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} + \lambda \boldsymbol{I})^{-1} \\ &= (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}. \end{align}\] and therefore \[\begin{align} \tilde{V}(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) &= \sigma^2 \boldsymbol{U}^\top \biggl((\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} - \frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}} \biggr)^2 \\ &\qquad \cdot \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n} + \frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \end{align}\] and therefore \[\begin{align} &V_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - V_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &= \sigma^2 \boldsymbol{U}^\top (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad - \sigma^2 \boldsymbol{U}^\top (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ & \biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n} \biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad - \sigma^2 \boldsymbol{U}^\top (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ &\biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad - \sigma^2 \boldsymbol{U}^\top \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \\ &\biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n} \biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad - \sigma^2 \boldsymbol{U}^\top \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ & (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad + \sigma^2 \boldsymbol{U}^\top \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ &\biggl(\frac{\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k}}{n} + \frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n} \biggr) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ &\qquad + \sigma^2 \boldsymbol{U}^\top \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \\ & \biggl(\frac{1}{n} \frac{(\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1}}{1 + \frac{1}{n} \tilde{\boldsymbol{X}}_k^{(1)\top} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \tilde{\boldsymbol{X}}_k^{(1)}}\biggr) \boldsymbol{\Sigma}^{(2)} \biggl(\frac{\tilde{\boldsymbol{X}}_k^{(1)} \tilde{\boldsymbol{X}}_k^{(1)\top}}{n}\biggr) \boldsymbol{U} \\ &:= \boldsymbol{u}^{(d,1)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,1)} + \frac{\boldsymbol{u}^{(d,2)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,2)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,2)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(d,3)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,3)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,3)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,3)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(d,4)\top} \tilde{\boldsymbol{Z}}_k^{(1)} \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,4)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,4)} \tilde{\boldsymbol{Z}}_k^{(1)}} + \frac{\boldsymbol{u}^{(d,5)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,5)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,5)}}{1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,5)} \tilde{\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(d,6)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,6)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,6)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,6)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(d,7)\top} \tilde{\boldsymbol{Z}}_k^{(1)} (\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,7)} \tilde{\boldsymbol{Z}}_k^{(1)})(\tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,7)} \tilde{\boldsymbol{Z}}_k^{(1)}) \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,7)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(d,7)} \tilde{\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] and similarly \[\begin{align} &V_1(\boldsymbol{Z}^{(1),k-1}, \boldsymbol{Z}^{(2)}) - V_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &:= \boldsymbol{u}^{(d,1)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,1)} + \frac{\boldsymbol{u}^{(d,2)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,2)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,2)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad+ \frac{\boldsymbol{u}^{(d,3)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,3)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,3)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,3)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(d,4)\top} {\boldsymbol{Z}}_k^{(1)} {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,4)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,4)} {\boldsymbol{Z}}_k^{(1)}} + \frac{\boldsymbol{u}^{(d,5)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,5)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,5)}}{1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,5)} {\boldsymbol{Z}}_k^{(1)}} \\ &\qquad + \frac{\boldsymbol{u}^{(d,6)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,6)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,6)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,6)} {\boldsymbol{Z}}_k^{(1)})^2}\\ &\qquad+ \frac{\boldsymbol{u}^{(d,7)\top} {\boldsymbol{Z}}_k^{(1)} ({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{A}^{(d,7)} {\boldsymbol{Z}}_k^{(1)})({\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{B}^{(d,7)} {\boldsymbol{Z}}_k^{(1)}) {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{v}^{(d,7)}}{(1 + {\boldsymbol{Z}}_k^{(1)\top} \boldsymbol{C}^{(d,7)} {\boldsymbol{Z}}_k^{(1)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(d,1)} = \frac{\sigma^2}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U}, \quad \boldsymbol{v}^{(d,1)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{u}^{(d,2)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U} \\ \boldsymbol{v}^{(d,2)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k} + \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(d,2)}= \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(d,3)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U}, \quad \boldsymbol{v}^{(d,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(d,3)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(d,3)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(d,4)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U} \\ \boldsymbol{v}^{(d,4)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k} + \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(d,4)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(d,5)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U}, \quad \boldsymbol{v}^{(d,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(d,5)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(d,5)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(d,6)} = \frac{\sigma^2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U} \\ \boldsymbol{v}^{(d,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{X}^{(1),\setminus k\top} \boldsymbol{X}^{(1),\setminus k} + \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(d,6)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(d,6)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{u}^{(d,7)} = \frac{\sigma^2}{n^3} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U}, \quad \boldsymbol{v}^{(d,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{U} \\ \boldsymbol{A}^{(d,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{B}^{(d,7)} = (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\boldsymbol{\Sigma}^{(1)})^{1/2} \\ \boldsymbol{C}^{(d,7)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} \end{gather}\] By the symmetry of \(V_1\) in \(\boldsymbol{Z}^{(1)}\) and \(\boldsymbol{Z}^{(2)}\), we see that \[\begin{align} &V_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2),k}) - V_1(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2), \setminus k}) \\ &:= \boldsymbol{u}^{(D,1)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,1)} + \frac{\boldsymbol{u}^{(D,2)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,2)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,2)} \tilde{\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,3)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,3)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,3)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,3)} \tilde{\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,4)\top} \tilde{\boldsymbol{Z}}_k^{(2)} \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,4)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,4)} \tilde{\boldsymbol{Z}}_k^{(2)}} + \frac{\boldsymbol{u}^{(D,5)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,5)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,5)}}{1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,5)} \tilde{\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,6)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,6)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,6)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,6)} \tilde{\boldsymbol{Z}}_k^{(2)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(D,7)\top} \tilde{\boldsymbol{Z}}_k^{(2)} (\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,7)} \tilde{\boldsymbol{Z}}_k^{(2)})(\tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,7)} \tilde{\boldsymbol{Z}}_k^{(2)}) \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,7)}}{(1 + \tilde{\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{C}^{(D,7)} \tilde{\boldsymbol{Z}}_k^{(2)})^2} \end{align}\] and similarly \[\begin{align} &V_1(\boldsymbol{Z}^{(1),k}, \boldsymbol{Z}^{(2)}) - V_1(\boldsymbol{Z}^{(1),\setminus k}, \boldsymbol{Z}^{(2)}) \\ &:= \boldsymbol{u}^{(D,1)\top} {\boldsymbol{Z}}_k^{(2)} {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,1)} + \frac{\boldsymbol{u}^{(D,2)\top} {\boldsymbol{Z}}_k^{(2)} {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,2)}}{1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,2)} {\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,3)\top} {\boldsymbol{Z}}_k^{(2)} ({\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,3)} {\boldsymbol{Z}}_k^{(2)}) {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,3)}}{1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,3)} {\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,4)\top} {\boldsymbol{Z}}_k^{(2)} {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,4)}}{1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,4)} {\boldsymbol{Z}}_k^{(2)}} + \frac{\boldsymbol{u}^{(D,5)\top} {\boldsymbol{Z}}_k^{(2)} ({\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,5)} {\boldsymbol{Z}}_k^{(2)}) {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,5)}}{1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,5)} {\boldsymbol{Z}}_k^{(2)}} \\ &\qquad + \frac{\boldsymbol{u}^{(D,6)\top} {\boldsymbol{Z}}_k^{(2)} ({\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,6)} {\boldsymbol{Z}}_k^{(2)}) {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,6)}}{(1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,6)} {\boldsymbol{Z}}_k^{(2)})^2} \\ &\qquad + \frac{\boldsymbol{u}^{(D,7)\top} {\boldsymbol{Z}}_k^{(2)} ({\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{A}^{(D,7)} {\boldsymbol{Z}}_k^{(2)})({\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{B}^{(D,7)} {\boldsymbol{Z}}_k^{(2)}) {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{v}^{(D,7)}}{(1 + {\boldsymbol{Z}}_k^{(2)\top} \boldsymbol{C}^{(D,7)} {\boldsymbol{Z}}_k^{(2)})^2} \end{align}\] where \[\begin{gather} \boldsymbol{u}^{(D,1)} = \frac{\sigma^2}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U}, \quad \boldsymbol{v}^{(D,1)} = (\boldsymbol{\Sigma}^{(2)})^{3/2} \boldsymbol{U} \\ \boldsymbol{u}^{(D,2)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U} \\ \boldsymbol{v}^{(D,2)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} + \boldsymbol{X}^{(2), \setminus k\top} \boldsymbol{X}^{(2),\setminus k}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(D,2)}= \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(D,3)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} \boldsymbol{U}, \quad \boldsymbol{v}^{(D,3)} = (\boldsymbol{\Sigma}^{(2)})^{3/2} \boldsymbol{U} \\ \boldsymbol{A}^{(D,3)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(D,3)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(D,4)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U} \\ \boldsymbol{v}^{(D,4)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} + \boldsymbol{X}^{(2), \setminus k\top} \boldsymbol{X}^{(2),\setminus k}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(D,4)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(D,5)} = \frac{\sigma^2}{n^2} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U}, \quad \boldsymbol{v}^{(D,5)} = (\boldsymbol{\Sigma}^{(2)})^{3/2} \boldsymbol{U} \\ \boldsymbol{A}^{(D,5)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(D,5)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(D,6)} = \frac{\sigma^2}{n^3} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U} \\ \boldsymbol{v}^{(D,6)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\tilde{\boldsymbol{X}}^{(1)\top} \tilde{\boldsymbol{X}}^{(1)} + \boldsymbol{X}^{(2), \setminus k\top} \boldsymbol{X}^{(2),\setminus k}) \boldsymbol{\Sigma}^{(2)} \boldsymbol{U} \\ \boldsymbol{A}^{(D,6)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(D,6)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{u}^{(D,7)} = \frac{\sigma^2}{n^3} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{U}, \quad \boldsymbol{v}^{(D,7)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} \boldsymbol{U} \\ \boldsymbol{A}^{(D,7)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-2} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{B}^{(D,7)} = (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\boldsymbol{\Sigma}^{(2)})^{1/2} \\ \boldsymbol{C}^{(D,7)} = \frac{1}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} (\tilde{\boldsymbol{\Sigma}}_{\setminus k} + \lambda \boldsymbol{I})^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2} \end{gather}\]

14.8.2 Bounding first and second moments↩︎

Now, we can apply Lemmas 35 and 32. We first bound \[\begin{gather} \|\boldsymbol{u}^{(d,1)}\|_2 = \sigma^2 n^{-1} \lambda^{-2} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,1)}\|_2 = \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{u}^{(d,2)}\|_2 = \sigma^2 n^{-2} \lambda^{-2} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,2)}\|_2 = \lambda^{-1} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,2)}\|_{\text{op}}= n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(d,3)}\|_2 = \sigma^2 n^{-2} \lambda^{-2} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,3)}\|_2 = \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,3)}\|_{\text{op}} = \lambda^{-1} , \quad \|\boldsymbol{B}^{(d,3)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(d,4)}\|_2 = \sigma^2 n^{-2} \lambda^{-1} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,4)}\|_2 = \lambda^{-2} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,4)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(d,5)}\|_2 = \sigma^2 n^{-2} \lambda^{-1} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,5)}\|_2 = \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,5)}\|_{\text{op}} = \lambda^{-2} , \quad \|\boldsymbol{B}^{(d,5)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(d,6)}\|_2 = \sigma^2 n^{-3} \lambda^{-1} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,6)}\|_2 = \lambda^{-1} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,6)}\|_{\text{op}} = \lambda^{-2} , \quad \|\boldsymbol{B}^{(d,6)}\|_{\text{op}} = n^{-1} \lambda^{-1} \\ \|\boldsymbol{u}^{(d,7)}\|_2 = \sigma^2 n^{-3} \lambda^{-1} \|\boldsymbol{U}\|_2, \quad \|\boldsymbol{v}^{(d,7)}\|_2 = \|\boldsymbol{U}\|_2 \\ \|\boldsymbol{A}^{(d,7)}\|_{\text{op}} = \lambda^{-2} , \quad \|\boldsymbol{B}^{(d,7)}\|_{\text{op}} = \lambda^{-1} \\ \|\boldsymbol{C}^{(d,7)}\|_{\text{op}} = n^{-1} \lambda^{-1} \end{gather}\]

14.8.2.1 First term

Now, using the first moment bound for \(f_1\), we have \[|\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,1)}) - f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,1)})]| = 0\] and similarly, using the second moment bound for \(f_1\), we have \[\begin{align} \max&\{\mathbb{E}_{1,k}[f_1(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,1)})^2], \mathbb{E}_{1,k}[f_1(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,1)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(d,1)}\|_2^2 \|\boldsymbol{v}^{(d,1)}\|_2^2 \\ &\lesssim\sigma^4 n^{-2} \lambda^{-4} \|\boldsymbol{U}\|_2^4 \end{align}\]

14.8.2.2 Second term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,2)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,2)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,2)}\|_2 \|\boldsymbol{v}^{(d,2)}\|_2 ( \|\boldsymbol{A}^{(d,2)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(d,2)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(d,2)}\|_{\text{op}}^3 p^2) \\ &\lesssim\sigma^2 n^{-2} \lambda^{-2} \lambda^{-1} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2^2 ( n^{-1} \lambda^{-1} p^{1/2} + n^{-2} \lambda^{-2} p + n^{-3} \lambda^{-3} p^2) \\ &\lesssim\sigma^2 n^{-5/2} \lambda^{-4} \|\boldsymbol{U}\|_2^2(\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) (1 + n^{-1/2} \lambda^{-1} + n^{-1/2} \lambda^{-2}) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,2)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,2)})^2]\} \\ &\lesssim\sigma^4 n^{-4} \lambda^{-6} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^4) \|\boldsymbol{U}\|_2^4. \end{align}\]

14.8.2.3 Third term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,3)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,3)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,3)}\|_2 \|\boldsymbol{v}^{(d,3)}\|_2 \|\boldsymbol{A}^{(d,3)}\|_{\text{op}}( p^{1/2} + \|\boldsymbol{B}^{(d,3)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(d,3)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(d,3)}\|_{\text{op}}^3 p^3) \\ &\lesssim\sigma^2 n^{-2} \lambda^{-2} \lambda^{-1} \|\boldsymbol{U}\|_2^2 ( p^{1/2} + n^{-1} \lambda^{-1} p^{3/2} + n^{-2} \lambda^{-2} p^2 + n^{-3} \lambda^{-3} p^3) \\ &\lesssim\sigma^2 n^{-3/2} \|\boldsymbol{U}\|_2^2 \lambda^{-3} ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,3)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,3)})^2]\} \\ &\lesssim\sigma^4 n^{-2} \lambda^{-6} \|\boldsymbol{U}\|_2^4 \end{align}\]

14.8.2.4 Fourth term

Using the first moment bound for \(f_2\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,4)}) - f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,4)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,4)}\|_2 \|\boldsymbol{v}^{(d,4)}\|_2 ( \|\boldsymbol{A}^{(d,4)}\|_{\text{op}} p^{1/2} + \|\boldsymbol{A}^{(d,4)}\|_{\text{op}}^2 p + \|\boldsymbol{A}^{(d,4)}\|_{\text{op}}^3 p^2) \\ &\lesssim\sigma^2 n^{-2} \lambda^{-1} \lambda^{-2} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2^2 ( n^{-1} \lambda^{-1} p^{1/2} + n^{-2} \lambda^{-2} p + n^{-3} \lambda^{-3} p^2) \\ &\lesssim\sigma^2 n^{-5/2} \lambda^{-4} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \|\boldsymbol{U}\|_2^2 ( 1 + n^{-1/2} \lambda^{-1} + n^{-1/2} \lambda^{-2} ) \end{align}\] and using the second moment bound for \(f_2\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_2(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,4)})^2], \mathbb{E}_{1,k}[f_2(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,4)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(d,4)}\|_2^2 \|\boldsymbol{v}^{(d,4)}\|_2^2 \\ &\lesssim(\sigma^2 n^{-2} \lambda^{-1})^2 \|\boldsymbol{U}\|_2^4 (\lambda^{-2} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2))^2 \\ &\lesssim\sigma^4 n^{-4} \lambda^{-6} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^4) \|\boldsymbol{U}\|_2^4. \end{align}\]

14.8.2.5 Fifth term

Using the first moment bound for \(f_4\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,5)}) - f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,5)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,5)}\|_2 \|\boldsymbol{v}^{(d,5)}\|_2 \|\boldsymbol{A}^{(d,5)}\|_{\text{op}} \|\boldsymbol{U}\|_2^2 \\ &\qquad ( p^{1/2} + \|\boldsymbol{B}^{(d,5)}\|_{\text{op}} p^{3/2} + \|\boldsymbol{B}^{(d,5)}\|_{\text{op}}^2 p^2 + \|\boldsymbol{B}^{(d,5)}\|_{\text{op}}^3 p^3) \\ &\lesssim\sigma^2 n^{-3/2} \lambda^{-3} \|\boldsymbol{U}\|_2^2 ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \end{align}\] and using the second moment bound for \(f_4\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_4(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,5)})^2], \mathbb{E}_{1,k}[f_4(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,5)})^2]\} \\ &\lesssim\sigma^4 n^{-2} \lambda^{-6} \|\boldsymbol{U}\|_2^4 \end{align}\]

14.8.2.6 Sixth term

Using the first moment bound for \(f_5\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,6)}) - f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,6)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,6)}\|_2 \|\boldsymbol{v}^{(d,6)}\|_2 \|\boldsymbol{A}^{(d,6)}\|_{\text{op}}(p^{1/2} + \|\boldsymbol{B}^{(d,6)}\|_{\text{op}}^2 p^{5/2} + \|\boldsymbol{B}^{(d,6)}\|_{\text{op}}^4 p^4 + \|\boldsymbol{B}^{(d,6)}\|_{\text{op}}^6 p^6) \\ &\lesssim\sigma^2 n^{-3} \lambda^{-1} \lambda^{-1} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) \lambda^{-2} \|\boldsymbol{U}\|_2^2 \\ &\qquad (p^{1/2} + n^{-2} \lambda^{-2} p^{5/2} + n^{-4} \lambda^{-4} p^4 + n^{-6} \lambda^{-6} p^6) \\ &\lesssim\sigma^2 n^{-5/2} \lambda^{-4} \|\boldsymbol{U}\|_2^2 (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^2) (1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \end{align}\] and using the second moment bound for \(f_5\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_5(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,6)})^2], \mathbb{E}_{1,k}[f_5(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,6)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(d,6)}\|_2^2 \|\boldsymbol{v}^{(d,6)}\|_2^2 \|\boldsymbol{A}^{(d,6)}\|_{\text{op}}^2 p^2 \\ &\lesssim\sigma^4 n^{-4} \lambda^{-8} (\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4 + \|\boldsymbol{X}^{(2)}\|_{\text{op}}^4) \|\boldsymbol{U}\|_2^4 \end{align}\]

14.8.2.7 Seventh term

Using the first moment bound for \(f_7\), we have \[\begin{align} |&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,7)}) - f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,7)})]| \\ &\lesssim\|\boldsymbol{u}^{(d,7)}\|_2 \|\boldsymbol{v}^{(d,7)}\|_2 \|\boldsymbol{A}^{(d,7)}\|_{\text{op}} \|\boldsymbol{B}^{(d,7)}\|_{\text{op}} \\ &\qquad ( p^{3/2} + \|\boldsymbol{C}^{(d,7)}\|_{\text{op}}^2 p^{7/2} + \|\boldsymbol{C}^{(d,7)}\|_{\text{op}}^4 p^5 + \|\boldsymbol{C}^{(d,7)}\|_{\text{op}}^6 p^7) \\ &\lesssim\sigma^2 n^{-3/2} \lambda^{-4} \|\boldsymbol{U}\|_2^2 ( 1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \end{align}\] and using the second moment bound for \(f_7\) yields \[\begin{align} \max\{&\mathbb{E}_{1,k}[f_7(\boldsymbol{Z}_k^{(1)}; \boldsymbol{S}^{(d,7)})^2], \mathbb{E}_{1,k}[f_7(\tilde{\boldsymbol{Z}}_k^{(1)}; \boldsymbol{S}^{(d,7)})^2]\} \\ &\lesssim\|\boldsymbol{u}^{(d,7)}\|_2^2 \|\boldsymbol{v}^{(d,7)}\|_2^2 \|\boldsymbol{A}^{(d,7)}\|_{\text{op}}^2 \|\boldsymbol{B}^{(d,7)}\|_{\text{op}}^2 p^4 \\ &\lesssim\sigma^4 n^{-2} \lambda^{-8} \|\boldsymbol{U}\|_2^4 \end{align}\]

14.8.2.8 Combined

It thus follows from the above bounds that \[\begin{align} |&\mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})])] - \mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})])]| \\ &\lesssim\sigma^4 n^{-1} \lambda^{-4} \mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-3/2} \lambda^{-4} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^2]) \\ &(1 + n^{-1/2} \lambda^{-1} + n^{-1/2} \lambda^{-2}) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-3} \lambda^{-6} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^4])\mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-1/2} \lambda^{-3} ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-1} \lambda^{-6} \mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-3/2} \lambda^{-4} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^2]) \\ &( 1 + n^{-1/2} \lambda^{-1} + n^{-1/2} \lambda^{-2} ) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-3} \lambda^{-6} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^4]) \mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-1/2} \lambda^{-3} ( 1 + \lambda^{-1} + n^{-1/2} \lambda^{-2} + n^{-1/2} \lambda^{-3}) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-1} \lambda^{-6} \mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-3/2} \lambda^{-4} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^2] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^2]) \\ &(1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-3} \lambda^{-8} (\mathbb{E}[\|\boldsymbol{X}^{(1),\setminus k}\|_{\text{op}}^4] + \mathbb{E}[\|\boldsymbol{X}^{(2)}\|_{\text{op}}^4]) \mathbb{E}[\|\boldsymbol{U}\|_2^4] \\ &\qquad + \sigma^2 n^{-1/2} \lambda^{-4} ( 1 + \lambda^{-2} + n^{-1/2} \lambda^{-4} + n^{-1/2} \lambda^{-6} ) \mathbb{E}[\|\boldsymbol{U}\|_2^2] \\ &\qquad + \sigma^4 n^{-1} \lambda^{-8} \mathbb{E}[\|\boldsymbol{U}\|_2^4]\\ &\lesssim\sigma^2 n^{-1/2} \log n \cdot \lambda^{-4} (1 + \lambda^{-2} + n^{-1/2}(\lambda^{-1} + \lambda^{-4} + \lambda^{-6})) \\ &\qquad + \sigma^2 n^{-1/2} \lambda^{-3} \\ &\qquad + \sigma^4 n^{-1} \log^2n \cdot (\lambda^{-6} + \lambda^{-8}) \\ &\qquad + \sigma^4 n^{-1} \lambda^{-4} \end{align}\]

14.8.3 Universality for \(\mathbb{E}[\tilde{V}]\)↩︎

Because of the symmetry between in \(\mathbb{E}[\tilde{V}]\) between \(\boldsymbol{Z}^{(1)}\) and \(\boldsymbol{Z}^{(2)}\), the same bound holds for the difference \(|\mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \boldsymbol{Z}^{(2)})])] - \mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})])]|\). Therefore, \[\begin{align} |&\mathbb{E}[\varphi(V(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(V(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &=|\mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}({\boldsymbol{Z}}^{(1)}, {\boldsymbol{Z}}^{(2)})])] - \mathbb{E}[\varphi(\mathbb{E}_U[\tilde{V}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})])]| \\ &\lesssim\sigma^2 n^{-1/2} \log n \cdot \lambda^{-4} (1 + \lambda^{-2} + n^{-1/2}(\lambda^{-1} + \lambda^{-4} + \lambda^{-6})) + \sigma^2 n^{-1/2} \lambda^{-3} \\ &\qquad + \sigma^4 n^{-1} \log^2n \cdot (\lambda^{-6} + \lambda^{-8}) + \sigma^4 n^{-1} \lambda^{-4}. \end{align}\] which concludes the proof of Proposition 22.

14.9 Proof of Theorem 11↩︎

We now use Proposition 22 to finish our proof of Theorem 11. In particular, we use largely the same ideas as Appendix 12.2, where we control the difference between the corresponding ridge and ridgeless quantities of interest. Recall that, by Lemmas 19 and 20, for any small constant \(c > 0\), we have \[\begin{align} |B_2(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})| &= O(n^c \lambda \|\tilde{\boldsymbol{\beta}}\|_2^2) \\ |B_3(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - B_3(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})| &= O(n^c\lambda \|\tilde{\boldsymbol{\beta}}\|_2\|\boldsymbol{\beta}^{(2)}\|) \end{align}\] with high probability over \((\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\). Additionally, it follows from the second result in Lemma 38 that \[|V(\hat{\boldsymbol{\beta}}_\lambda; \boldsymbol{\beta}^{(2)}) - V(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)})| = O(n^c\lambda)\] with high probability. Finally, we prove the analogous bound for the first bias term.

That is, as \(\lambda \downarrow 0\) sufficiently slowly and \(n \to \infty\), the following four converge in probability: (1) \(B_{\lambda,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) converges in probability to \(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) for \(j = 1,2,3\), (2) \(B_{\lambda,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})\) converges in probability to \(B_{0,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})\) for \(j = 1,2,3\), (3) \(V_{\lambda}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\) converges in probability to \(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)})\), and (4) \(V_{\lambda}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})\) converges in probability to \(V_{0}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)})\). Recalling that \(\varphi\) is a continuous function, it follows that (1) \(\varphi(B_{\lambda,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))\) converges in probability to \(\varphi(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))\) for \(j = 1,2,3\), (2) \(\varphi(B_{\lambda,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))\) converges in probability to \(\varphi(B_{0,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))\) for \(j = 1,2,3\), (3) \(\varphi(V_{\lambda}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))\) converges in probability to \(\varphi(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))\), and (4) \(\varphi(V_{\lambda}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))\) converges in probability to \(\varphi(V_{0}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))\)

Now, recalling that \(\varphi\) is bounded, it follows from the dominated convergence theorem that, as \(\lambda\downarrow 0\) sufficiently slowly and \(n \to \infty\), we have \[\begin{align} |\mathbb{E}[\varphi(B_{\lambda,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))]| &\to 0,\quad j = 1,2,3 \\ |\mathbb{E}[\varphi(B_{\lambda,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))] - \mathbb{E}[\varphi(B_{0,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| &\to 0,\quad j = 1,2,3 \\ |\mathbb{E}[\varphi(V_{\lambda}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(V_{0}( \boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))]| &\to 0 \\ |\mathbb{E}[\varphi(V_{\lambda}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))] - \mathbb{E}[\varphi(V_{0}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| &\to 0. \end{align}\] Therefore, we can conclude that, taking \(\lambda\downarrow 0\) sufficiently slowly and taking \(n \to \infty\), we get \[\begin{align} &|\mathbb{E}[\varphi(B_{0,1}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_{0,1}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\le |\mathbb{E}[\varphi(B_{0,1}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_{\lambda,1}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))]| \\ &\qquad + |\mathbb{E}[\varphi(B_{\lambda,1}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_{\lambda,1}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\qquad + |\mathbb{E}[\varphi(B_{\lambda,1}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))] - \mathbb{E}[\varphi(B_{0,1}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| \\ &\to 0 \end{align}\] The same reasoning follows for \(j = 2,3\) and for the variance terms, from which we conclude \[\begin{align} |\mathbb{E}[\varphi(B_{0,j}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(B_{0,j}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| &\to 0,\quad j = 1,2,3 \\ |\mathbb{E}[\varphi(V_{0}(\boldsymbol{Z}^{(1)}, \boldsymbol{Z}^{(2)}))] - \mathbb{E}[\varphi(V_{0}(\tilde{\boldsymbol{Z}}^{(1)}, \tilde{\boldsymbol{Z}}^{(2)}))]| &\to 0. \end{align}\] Theorem 11 then follows immediately by the standard Lindeberg argument (cf.  [35], [43]).

15 Additional Proofs↩︎

In this section, we prove Theorems 12 and 13.

15.1 Proof of Theorem 12↩︎

In this section, we provide a proof of Theorem 12. Note that it suffices to characterize the behavior of \(R_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})\) and \(R_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})\) separately: we handle these two terms in Sections 15.1.1 and 15.1.2, respectively.

15.1.1 First term↩︎

Crucially, note that we can express \[\begin{align} \hat{\boldsymbol{\beta}}&= (\boldsymbol{X}^\top \boldsymbol{X})^\dagger (\boldsymbol{X}^{(1)\top} (\boldsymbol{X}^{(1)} \boldsymbol{\beta}^{(1)} + \boldsymbol{\epsilon}^{(1)}) + \boldsymbol{X}^{(2)\top} (\boldsymbol{X}^{(2)} \boldsymbol{\beta}^{(2)} + \boldsymbol{\epsilon}^{(2)})) \\ &\qquad + (\boldsymbol{X}^\top \boldsymbol{X})^\dagger (\boldsymbol{X}^{(1)\top}\boldsymbol{W}^{(1)} \boldsymbol{\theta}^{(1)} + \boldsymbol{X}^{(2)\top} \boldsymbol{W}^{(2)} \boldsymbol{\theta}^{(2)}) \end{align}\] where the first term \(\hat{\boldsymbol{\beta}}_{\boldsymbol{X}} := (\boldsymbol{X}^\top \boldsymbol{X})^\dagger (\boldsymbol{X}^{(1)\top} (\boldsymbol{X}^{(1)} \boldsymbol{\beta}^{(1)} + \boldsymbol{\epsilon}^{(1)}) + \boldsymbol{X}^{(2)\top} (\boldsymbol{X}^{(2)} \boldsymbol{\beta}^{(2)} + \boldsymbol{\epsilon}^{(2)}))\) matches the min-\(\ell_2\)-norm interpolator and setting studied in Theorem 1 and the second term \(\boldsymbol{\epsilon}_{\boldsymbol{W}} := (\boldsymbol{X}^\top \boldsymbol{X})^\dagger (\boldsymbol{X}^{(1)\top}\boldsymbol{W}^{(1)} \boldsymbol{\theta}^{(1)} + \boldsymbol{X}^{(2)\top} \boldsymbol{W}^{(2)} \boldsymbol{\theta}^{(2)})\) is mean zero, conditional on \(\boldsymbol{X}\). Additionally, by assumption, we have that \(\hat{\boldsymbol{\beta}}_{\boldsymbol{X}}\) and \(\boldsymbol{\epsilon}_{\boldsymbol{W}}\) are independent.

Now, using the fact that \(\boldsymbol{x}_0\) and \(\boldsymbol{w}_0\) are independent, we have \[\begin{align} R_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)}) &= \mathbb{E}[(\boldsymbol{x}_0^\top \hat{\boldsymbol{\beta}}- \boldsymbol{x}_0^\top \boldsymbol{\beta}^{(2)} - \mathbb{E}[\boldsymbol{w}_0 \mid \boldsymbol{x}_0]^\top \boldsymbol{\theta}^{(2)})^2 \mid \boldsymbol{X}] \\ &= \mathbb{E}[(\boldsymbol{x}_0^\top \hat{\boldsymbol{\beta}}- x_0^\top \boldsymbol{\beta}^{(2)})^2 \mid \boldsymbol{X}] \\ &= \mathbb{E}[\|\hat{\boldsymbol{\beta}}- \boldsymbol{\beta}^{(2)}\|_{\Sigma^{(2)}}^2 \mid \boldsymbol{X}] \\ &= \| \mathbb{E}[\hat{\boldsymbol{\beta}}\mid \boldsymbol{X}] - \boldsymbol{\beta}^{(2)}\|_{\boldsymbol{\Sigma}^{(2)}}^2 + \textrm{Tr}[\textrm{Cov}(\hat{\boldsymbol{\beta}}\mid \boldsymbol{X}) \boldsymbol{\Sigma}^{(2)}] \\ &= \|\mathbb{E}[\hat{\boldsymbol{\beta}}_{\boldsymbol{X}} \mid \boldsymbol{X}] - \boldsymbol{\beta}^{(2)}\|_{\boldsymbol{\Sigma}^{(2)}}^2 + \textrm{Tr}[\textrm{Cov}(\hat{\boldsymbol{\beta}}_{\boldsymbol{X}} \mid \boldsymbol{X}) \boldsymbol{\Sigma}^{(2)}] \\ &\qquad + \textrm{Tr}[\textrm{Cov}(\boldsymbol{\epsilon}_{\boldsymbol{W}} \mid \boldsymbol{X}) \boldsymbol{\Sigma}^{(2)}]. \end{align}\] The behavior of the first two summands follows directly from our model shift results.

The remaining term \(\textrm{Tr}[\textrm{Cov}(\boldsymbol{\epsilon}_{\boldsymbol{W}} \mid \boldsymbol{X}) \boldsymbol{\Sigma}^{(2)}]\) requires more care: we use the same symmetrization techniques as in Appendix 12. First, note that \[\begin{align} \textrm{Cov}(\boldsymbol{\epsilon}_{\boldsymbol{W}} \mid \boldsymbol{X}) &= \textrm{Cov}((\boldsymbol{X}^\top \boldsymbol{X})^\dagger \boldsymbol{X}^{(1)\top} \boldsymbol{W}^{(1)} \boldsymbol{\theta}^{(1)} \mid \boldsymbol{X}) \\ &\qquad + \textrm{Cov}((\boldsymbol{X}^\top \boldsymbol{X})^\dagger \boldsymbol{X}^{(2)\top} \boldsymbol{W}^{(2)} \boldsymbol{\theta}^{(2)} \mid \boldsymbol{X}) \\ &= (\boldsymbol{X}^\top \boldsymbol{X})^\dagger \boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)} (\boldsymbol{X}^\top \boldsymbol{X})^\dagger \cdot \|\boldsymbol{\theta}^{(1)}\|_{\boldsymbol{\Sigma}_m^{(1)}}^2 \\ &\qquad + (\boldsymbol{X}^\top \boldsymbol{X})^\dagger \boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)} (\boldsymbol{X}^\top \boldsymbol{X})^\dagger \cdot \|\boldsymbol{\theta}^{(2)}\|_{\boldsymbol{\Sigma}_m^{(2)}}^2. \end{align}\] and thus \[\begin{align} \textrm{Tr}[\textrm{Cov}(\boldsymbol{\epsilon}_{\boldsymbol{W}} \mid \boldsymbol{X})] &= \|\boldsymbol{\theta}^{(1)}\|_{\boldsymbol{\Sigma}_m^{(1)}}^2 \cdot \frac{1}{n} \textrm{Tr}\biggl[\hat{\boldsymbol{\Sigma}}^\dagger \biggl(\frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n}\biggr) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}^{(2)} \biggr] \\ &\qquad + \|\boldsymbol{\theta}^{(2)}\|_{\boldsymbol{\Sigma}_m^{(2)}}^2 \cdot \frac{1}{n} \textrm{Tr}\biggl[\hat{\boldsymbol{\Sigma}}^\dagger \biggl(\frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n}\biggr) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}^{(2)} \biggr] \end{align}\] It suffices to characterize \(\frac{1}{n} \textrm{Tr}[\hat{\boldsymbol{\Sigma}}^\dagger (\frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n}) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}^{(2)}]\), since the behavior of the latter term \(\frac{1}{n} \textrm{Tr}[\hat{\boldsymbol{\Sigma}}^\dagger (\frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n}) \hat{\boldsymbol{\Sigma}}^\dagger \boldsymbol{\Sigma}^{(2)}]\) follows by symmetry in this setting.

We begin by defining \[\begin{align} \boldsymbol{V}_M &:= \biggl(\frac{\boldsymbol{Z}^{(1)\top} \boldsymbol{Z}^{(1)}}{n}\biggr) (\hat{\boldsymbol{\Sigma}}_Z + \lambda \boldsymbol{\Sigma}^{-1})^{-2} \\ \boldsymbol{V}_M^F &:= \hat{\boldsymbol{\Sigma}}_Z (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{\Sigma}^{-1})^{-2} \end{align}\]

Thus, we yield the following characterization:

15.1.2 Second term↩︎

Now, consider the second term \(R_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})\). Note that \(\mathbb{E}[\boldsymbol{w}_0 \mid \boldsymbol{x}_0] = \mathbb{E}[\boldsymbol{w}_0] = 0\) by independence of \(\boldsymbol{x}_0\) and \(\boldsymbol{w}_0\), which implies that \[\begin{align} R_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)}) &= \mathbb{E}[(\mathbb{E}[\boldsymbol{w}_0 \mid \boldsymbol{x}_0]^\top \boldsymbol{\theta}^{(2)} - \boldsymbol{w}_0^\top \boldsymbol{\theta}^{(2)})^2] \\ &= \mathbb{E}[(\boldsymbol{w}_0^\top \boldsymbol{\theta}^{(2)})^2] = \|\boldsymbol{\theta}^{(2)}\|_{\boldsymbol{\Sigma}_m^{(2)}}^2 \end{align}\] as desired.

15.2 Proof of Theorem 13↩︎

In this section, we provide a proof of Theorem 13.

Denote \(n_\kappa = n_1 + \lfloor (1 - \kappa ) n_2 \rfloor\). For a given subset \(\mathcal{I} \subseteq [n_2]\), we denote \(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}} = \frac{\boldsymbol{X}^{(1)\top} \boldsymbol{X}^{(1)}}{n_\kappa} + \frac{\boldsymbol{X}^{(2)\top} \boldsymbol{X}^{(2)}}{n_\kappa}\) and \(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} = \frac{\boldsymbol{X}^{(2)\top } \boldsymbol{X}^{(2)}}{\kappa n_2}\), in which case \[\begin{align} \hat{\boldsymbol{\beta}}_{\text{BC}} &= \hat{\boldsymbol{\beta}}_{\mathcal{I}} + \hat{\boldsymbol{\delta}}_{\lambda_\delta} \\ &= \hat{\boldsymbol{\beta}}_{\mathcal{I}} + \frac{1}{\kappa n_2} (\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta \boldsymbol{I})^{-1} \boldsymbol{X}_{\mathcal{I}^c}^{(2)\top} (\boldsymbol{y}_{\mathcal{I}^c}^{(2)} - \boldsymbol{X}_{\mathcal{I}^c}^{(2)} \hat{\boldsymbol{\beta}}_{\mathcal{I}}) \\ &= (\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda _\delta \boldsymbol{I})^{-1} \frac{\boldsymbol{X}^{(2)\top}_{\mathcal{I}^c} \boldsymbol{y}_{\mathcal{I}^c}^{(2)}}{\kappa n_2} + \lambda_\delta(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta)^{-1} \hat{\boldsymbol{\beta}}_{\mathcal{I}}. \end{align}\] Therefore, we have \[\begin{align} \hat{\boldsymbol{\beta}}_{\text{BC}} - \boldsymbol{\beta}^{(2)} &= (\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda _\delta \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(2)\top}_{\mathcal{I}^c} \boldsymbol{y}_{\mathcal{I}^c}^{(2)}}{\kappa n_2} - \hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} \boldsymbol{\beta}^{(2)}\biggr) + \lambda_\delta(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta)^{-1} (\hat{\boldsymbol{\beta}}_{\mathcal{I}} - \boldsymbol{\beta}^{(2)}) \\ &= (\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda _\delta \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(2)\top}_{\mathcal{I}^c} \boldsymbol{\epsilon}_{\mathcal{I}^c}^{(2)}}{\kappa n_2} \biggr) + \lambda_\delta(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta)^{-1} (\hat{\boldsymbol{\beta}}_{\mathcal{I}} - \boldsymbol{\beta}^{(2)}) \end{align}\] In the above expression, note that the terms \((\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda _\delta \boldsymbol{I})^{-1} (\frac{\boldsymbol{X}^{(2)\top}_{\mathcal{I}^c} \boldsymbol{\epsilon}_{\mathcal{I}^c}^{(2)}}{\kappa n_2})\) and \(\lambda_\delta(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta)^{-1} (\hat{\boldsymbol{\beta}}_{\mathcal{I}} - \boldsymbol{\beta}^{(2)})\) are independent, given \(\boldsymbol{X}\), and that the first term has mean \(0\), given \(\boldsymbol{X}\). Therefore, we can decompose \[\begin{align} &R(\hat{\boldsymbol{\beta}}_{\text{BC}}; \boldsymbol{\beta}^{(2)}) \\&= \mathbb{E}[\|\hat{\boldsymbol{\beta}}_{\text{BC}} - \boldsymbol{\beta}^{(2)}\|_2^2 \mid \boldsymbol{X}] \\ &= \underbrace{\mathbb{E}\biggl[\biggl\|(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda _\delta \boldsymbol{I})^{-1} \biggl(\frac{\boldsymbol{X}^{(2)\top}_{\mathcal{I}^c} \boldsymbol{\epsilon}_{\mathcal{I}^c}^{(2)}}{\kappa n_2}\biggr)\biggr\|_2^2\biggm\vert \boldsymbol{X} \biggr]}_{R_1(\hat{\boldsymbol{\beta}}_{\text{BC}}; \boldsymbol{\beta}^{(2)})} + \underbrace{\mathbb{E}[\|\lambda_\delta(\hat{\boldsymbol{\Sigma}}_{\mathcal{I}^c} + \lambda_\delta)^{-1} (\hat{\boldsymbol{\beta}}_{\mathcal{I}} - \boldsymbol{\beta}^{(2)})\|_2^2 \mid \boldsymbol{X}]}_{R_2(\hat{\boldsymbol{\beta}}_{\text{BC}}; \boldsymbol{\beta}^{(2)})}. \end{align}\] The analysis of the first term is relatively straightforward.

Now, we can similarly use a local law to analyze the latter term.

16 Sketch of a Covariate Shift Result Without Simultaneous Diagonalizability↩︎

In Theorem 7, we imposed Assumption 6; that is, the covariance matrices \(\boldsymbol{\Sigma}^{(1)}\) and \(\boldsymbol{\Sigma}^{(2)}\) are diagonalizable in a shared eigenbasis \(\boldsymbol{V}\). In the underparametrized regime in [34], such an assumption can often be avoided by whitening with respect to one of the covariance matrices. However, in the overparametrized regime, this technique fails because whitening the design also changes the geometry of the resulting interpolator: in particular, the min-\(\ell_2\)-norm interpolator is transformed into a min-norm interpolator with respect to a different, covariance-dependent norm.

Fully overcoming Assumption 6 requires substantial technical work beyond the work in this manuscript. To this end, we sketch a promising pathway for generalizing our results beyond simultaneous diagonalizability while deferring a rigorous proof to future work.

Our sketch relies upon the observation that we may apply anisotropic local laws from [41] repeatedly, conditional on the source- or target-only datasets. In this appendix, we focus on the first bias term \[B_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}) = \lambda^2 \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}\] and note that our approach may be adapted to the other terms. We omit exact technical rigor (e.g., we assume that \(\lambda \in \mathbb{R}\)) and instead focus on the general ideas and challenges of our pathway.

Let \(\hat{\boldsymbol{\Sigma}}^{(i)} = \frac{\boldsymbol{X}^{(i)\top} \boldsymbol{X}^{(i)}}{n}\) for \(i = 1,2\). We begin by first expressing, as in the proof of Theorem 7, that \[\begin{align} B_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}) &= \lambda^2 \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)} \\ &= \lambda^2 \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}^{(1)} + \hat{\boldsymbol{\Sigma}}^{(2)} + \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}^{(1)} + \hat{\boldsymbol{\Sigma}}^{(2)} + \lambda\boldsymbol{I} )^{-1} \boldsymbol{\beta}^{(2)} \\ &=- \frac{\partial}{\partial t} \bigg\vert_{t=0} \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}^{(1)} + \hat{\boldsymbol{\Sigma}}^{(2)} + \lambda \boldsymbol{I} + \lambda t\boldsymbol{\Sigma}^{(2)})^{-1} \boldsymbol{\beta}^{(2)} \\ &= -\frac{\partial}{\partial t} \bigg\vert_{t=0} \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}^{(1)} + \lambda \boldsymbol{R}_1)^{-1} \boldsymbol{\beta}^{(2)} \\ &= -\frac{\partial}{\partial t} \bigg\vert_{t=0} \lambda \boldsymbol{\beta}^{(2)\top} \boldsymbol{R}_1^{-1/2} (\boldsymbol{R}_1^{-1/2} \hat{\boldsymbol{\Sigma}}^{(1)} \boldsymbol{R}_1^{-1/2} + \lambda \boldsymbol{I})^{-1} \boldsymbol{R}_1^{-1/2} \boldsymbol{\beta}^{(2)} \end{align}\] where we let \(\boldsymbol{R}_1 = \boldsymbol{I} + t \boldsymbol{\Sigma}^{(2)} + \frac{1}{\lambda} \hat{\boldsymbol{\Sigma}}^{(2)}\). Then, conditional on \(\hat{\boldsymbol{\Sigma}}^{(2)}\), the term \(\boldsymbol{R}_1^{-1/2} \hat{\boldsymbol{\Sigma}}^{(1)} \boldsymbol{R}_1^{-1/2}\) corresponds to a sample covariance matrix with covariance \(\boldsymbol{R}_1^{-1/2} \boldsymbol{\Sigma}^{(1)} \boldsymbol{R}_1^{-1/2}\). Therefore, by applying an anisotropic local law (e.g., [41]), we have \[\begin{align} &\lambda \boldsymbol{\beta}^{(2)\top} \boldsymbol{R}_1^{-1/2} (\boldsymbol{R}_1^{-1/2} \hat{\boldsymbol{\Sigma}}^{(1)} \boldsymbol{R}_1^{-1/2} + \lambda \boldsymbol{I})^{-1} \boldsymbol{R}_1^{-1/2} \boldsymbol{\beta}^{(2)} \\ &\approx \lambda \boldsymbol{\beta}^{(2)\top} (\lambda \boldsymbol{R}_1 + \tfrac{n_1}{n} \lambda r_n(-\lambda; \gamma_1, \tfrac{n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{R}_1^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}) \boldsymbol{\Sigma}^{(1)})^{-1} \boldsymbol{\beta}^{(2)} \end{align}\] where \(r_n(-\lambda; \gamma_1, \tfrac{n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{R}_1^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2})\) is the companion Stieltjes transform (cf. [41]) corresponding to the empirical spectral distribution of the matrix \[\begin{align} \frac{n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{R}_1^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2} &= \frac{\lambda n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}^{(2)} + \lambda t \boldsymbol{\Sigma}^{(2)} + \lambda \boldsymbol{I} )^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}. \end{align}\] If it can be shown that the empirical spectral distribution of \(\frac{\lambda n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}^{(2)} + \lambda t \boldsymbol{\Sigma}^{(2)} + \lambda \boldsymbol{I} )^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}\) converges in probability to some probability measure \(\mu_{1,t}\) (with respect to a suitable metric on probability measures), then we have \[\begin{align} &\lambda \boldsymbol{\beta}^{(2)\top} (\lambda \boldsymbol{R}_1 + \tfrac{n_1}{n} \lambda r_n(-\lambda; \gamma_1, \tfrac{n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} \boldsymbol{R}_1^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}) \boldsymbol{\Sigma}^{(1)})^{-1} \boldsymbol{\beta}^{(2)} \\ &\approx \lambda \boldsymbol{\beta}^{(2)\top} (\lambda \boldsymbol{R}_1 + \tfrac{n_1}{n} \lambda r(-\lambda; \gamma_1, \mu_{1,t}) \boldsymbol{\Sigma}^{(1)})^{-1} \boldsymbol{\beta}^{(2)} \\ &= \lambda \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}^{(2)} + t \lambda \boldsymbol{\Sigma}^{(2)} + \lambda \boldsymbol{I} + \tfrac{n_1}{n} \lambda r(-\lambda; \gamma_1, \mu_{1,t}) \boldsymbol{\Sigma}^{(1)})^{-1} \boldsymbol{\beta}^{(2)} \\ &= \lambda \boldsymbol{\beta}^{(2)} (\hat{\boldsymbol{\Sigma}}^{(2)} + \lambda \boldsymbol{R}_2)^{-1} \boldsymbol{\beta}^{(2)} \\ &= \lambda \boldsymbol{\beta}^{(2)} \boldsymbol{R}_2^{-1/2}(\boldsymbol{R}_2^{-1/2} \hat{\boldsymbol{\Sigma}}^{(2)} \boldsymbol{R}_2^{-1/2} + \lambda\boldsymbol{I})^{-1} \boldsymbol{R}_2^{-1/2} \boldsymbol{\beta}^{(2)} \end{align}\] where we let \(\boldsymbol{R}_2 = t \boldsymbol{\Sigma}^{(2)} + \boldsymbol{I} + \tfrac{n_1}{n} r(-\lambda; \gamma_1, \mu_{1,t}) \boldsymbol{\Sigma}^{(1)}\). Then, we can once again apply an anisotropic local law (e.g., [41]) to yield \[\begin{align} &\lambda \boldsymbol{\beta}^{(2)} \boldsymbol{R}_2^{-1/2}(\boldsymbol{R}_2^{-1/2} \hat{\boldsymbol{\Sigma}}^{(2)} \boldsymbol{R}_2^{-1/2} + \lambda\boldsymbol{I})^{-1} \boldsymbol{R}_2^{-1/2} \boldsymbol{\beta}^{(2)} \\ &\approx \lambda \boldsymbol{\beta}^{(2)\top} (\lambda \boldsymbol{R}_2 + \tfrac{n_2}{n} \lambda r_n(-\lambda; \gamma_1, \tfrac{n_2}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} \boldsymbol{R}_2^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2}) \boldsymbol{\Sigma}^{(2)})^{-1} \boldsymbol{\beta}^{(2)} \end{align}\] which is a deterministic quantity.

There remain two major technical challenges to completing such a proof: (1) demonstrating that the empirical spectral distribution of \(\frac{\lambda n_1}{n} (\boldsymbol{\Sigma}^{(1)})^{1/2} (\hat{\boldsymbol{\Sigma}}^{(2)} + \lambda t \boldsymbol{\Sigma}^{(2)} + \lambda \boldsymbol{I} )^{-1} (\boldsymbol{\Sigma}^{(1)})^{1/2}\) converges in probability to some deterministic probability measure \(\mu_{1,t}\) and (2) characterizing the measure \(\mu_{1,t}\) is nontrivial. With the diagonalizations \(\boldsymbol{\Sigma}^{(i)} = \boldsymbol{V}^{(i)} \boldsymbol{D}^{(i)} \boldsymbol{V}^{(i)\top}\) for \(i = 1,2\), the empirical spectral distribution is equal in distribution to that of \[\frac{\lambda n_1}{n} (\boldsymbol{D}^{(1)})^{1/2} \boldsymbol{V}^{(1)\top} \boldsymbol{V}^{(2)} (\hat{\boldsymbol{\Sigma}}^{(2)}_Z + \lambda t \boldsymbol{D}^{(2)} + \lambda \boldsymbol{I} )^{-1} \boldsymbol{V}^{(2)\top} \boldsymbol{V}^{(1)}(\boldsymbol{D}^{(1)})^{1/2}\] where \(\hat{\boldsymbol{\Sigma}}^{(2)}_Z = \frac{\boldsymbol{Z}^{(2)\top} \boldsymbol{Z}^{(2)}}{n}\). While analyzing the empirical spectral distribution of \((\hat{\boldsymbol{\Sigma}}^{(2)}_Z + \lambda t \boldsymbol{D}^{(2)} + \lambda \boldsymbol{I} )^{-1}\) is possible with existing tools, a major challenge is introduced through the conjugation by \(\boldsymbol{V}^{(2)\top} \boldsymbol{V}^{(1)}(\boldsymbol{D}^{(1)})^{1/2}\). We believe that linearization techniques (such as Definition 7) may be promising for tackling this challenge.

The second major technical challenge is related to analyzing the resulting object, the \(t\)-derivative of the object \[\lambda \boldsymbol{\beta}^{(2)\top} (\lambda \boldsymbol{R}_2 + \tfrac{n_2}{n} \lambda r_n(-\lambda; \gamma_1, \tfrac{n_2}{n} (\boldsymbol{\Sigma}^{(2)})^{1/2} \boldsymbol{R}_2^{-1} (\boldsymbol{\Sigma}^{(2)})^{1/2}) \boldsymbol{\Sigma}^{(2)})^{-1} \boldsymbol{\beta}^{(2)}.\] Even if one is able to determine that \(\lambda^2 \boldsymbol{\beta}^{(2)\top} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\Sigma}^{(2)} (\hat{\boldsymbol{\Sigma}}+ \lambda \boldsymbol{I})^{-1} \boldsymbol{\beta}^{(2)}\) converges to such an object, it is less clear how to make sense of it in a way that allows for insights similar to those of our main text. However, we believe that the resulting objects may reduce significantly for special cases of covariance matrices which are of interest (e.g., when \(\boldsymbol{V}^{(1)} = \boldsymbol{Q} \boldsymbol{V}^{(2)}\) for a rotation matrix \(\boldsymbol{Q}\) that only acts on a subspace of dimension \(O(1)\)) while still yielding insights into how to treat general covariance matrices.

17 Additional Simulations↩︎

17.1 Universality↩︎

In this section, we examine the robustness of our risk characterizations beyond the Gaussian setting. Theorem 11 reveals that, under Assumption 3, the asymptotic prediction risk of the pooled min-\(\ell_2\)-norm interpolator agrees with its Gaussian counterpart under model shift. Here, we test whether the risk characterization in Theorem 1 continues to provide accurate predictions for a range of non-Gaussian covariates, including some that fall outside the scope of Assumption 1 or 3.

17.1.1 Synthetic covariates↩︎

In Figure 5, we reproduce Figures 1 (a) and 1 (b) under three synthetic non-Gaussian covariate distributions. Figures 5 (a)-(d) fall directly within the scope of Theorem 11. In Figures 5 (a) and 5 (b), our covariates are drawn as i.i.d. Rademacher entries. In Figures 5 (c) and 5 (d), the entries are generated from standardized Hardy-Weinberg distributions: for each coordinate \(j = 1, \dots, p\), we first fix an allele frequency \(p_j\) drawn uniformly from \([0.25, 0.75]\). Then, for a given observation, the \(j\)th entry (i.e., the \(j\)th genotype count) is drawn from a rescaled and centered \(\text{Bin}(2, p_j)\) distribution. In all four of these figures, we see very close alignment between the empirical risk values and the Gaussian risk predictions, which provides strong evidence for the finite-sample applicability of our asymptotic universality result in Theorem 11.

In Figures 5 (e) and 5 (f), the entries are drawn from a standardized \(t_4\) distribution. Recalling that a \(t_4\) distribution lacks a fourth moment, it certainly follows that Assumption 3 fails to hold. However, we note that the empirical risk values and Gaussian risk predictions in Figures 5 (e) and 5 (f) still match tightly, which suggests that the universality phenomenon described in Figure 11 may extend to heavier-tailed covariates, though formally establishing such a result requires additional arguments.

17.1.2 Semi-synthetic covariates↩︎

In Figure 6, we evaluate our risk predictions using semi-synthetic covariates derived from real human genotype data. We select 600 SNPs from the 1000 Genomes Phase 3 panel [68] and fit the HMM implemented in the fastPHASE software [69]. With the fitted parameters, we then use the SNPknock package from [70] to sample new SNPs. When whitened, the sampled SNPs are not expected to be i.i.d., as Assumption 1 required. Therefore, like the \(t_4\) covariates from Figures 5 (e) and 5 (f), the covariates in Figure 6 once again fall outside the theoretical assumptions for our results. Nonetheless, in Figures 6 (a)-(c), we again see very close alignment between the empirical risk values and the Gaussian risk predictions, thus validating our risk predictions for a set of semi-synthetic covariates.

17.2 Misspecification↩︎

In Figure 7, we examine the effect of misspecification strength on the risk of the pooled min-\(\ell_2\)-norm interpolator under the misspecified model shift setting considered in Theorem 12. In Figure 7 (a), we vary the source misspecification strength \(\|(\boldsymbol{\Sigma}_m^{(1)})^{1/2} \boldsymbol{\theta}^{(1)}\|_2^2\) while fixing the target misspecification strength \(\|(\boldsymbol{\Sigma}_m^{(2)})^{1/2} \boldsymbol{\theta}^{(2)}\|_2^2\) at \(0\); conversely, in Figure 7 (b), we vary the target misspecification strength \(\|(\boldsymbol{\Sigma}_m^{(2)})^{1/2} \boldsymbol{\theta}^{(2)}\|_2^2\) while fixing the source misspecification strength \(\|(\boldsymbol{\Sigma}_m^{(1)})^{1/2} \boldsymbol{\theta}^{(1)}\|_2^2\) at \(0\). We note that, in both Figures 7 (a) and 7 (b), the empirical risk of the pooled min-\(\ell_2\)-norm interpolator closely matches the risk predicted by Theorem 12, thus again providing evidence for the strong finite-sample performance of our results.

In both Figures 7 (a) and 7 (b), we see that increasing the amount of misspecification increases the risk of the pooled min-\(\ell_2\)-norm interpolator, as expected. However, we note that different patterns emerge for the two sources of misspecification. As \(n_1\) increases, the effect of source misspecification increases, as larger fractions of the pooled data are affected by the source misspecification. On the other hand, target misspecification raises the risk of the pooled min-\(\ell_2\)-norm interpolator both through the training labels (as captured by an increased value of \(R_1(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})\) in Theorem 12) and through the irreducible target prediction error (as captured by an increased value of \(R_2(\hat{\boldsymbol{\beta}}; \boldsymbol{\beta}^{(2)}, \boldsymbol{\theta}^{(2)})\)). Therefore, for small and moderate \(n_1\), target misspecification is more harmful than analogous source misspecification. However, as \(n_1\) grows, source misspecification becomes more amplified and can have a greater impact than analogous target misspecification.

17.3 Bias-corrected estimator↩︎

We now provide empirical results related to the bias-corrected estimator in Algorithm 4.

First, in Figure 8, we verify the risk predictions of Theorem 13 for the bias-corrected estimator across a range of SNR, SSR, sample-split parameter, and second-stage regularization parameter values. Across these parameter settings and over a range of source sample sizes \(n_1\), the empirical risks closely match theoretical predictions from Theorem 13, validating the strong finite-sample performance of our results for the bias-corrected estimator.

Next, in Figure 9, we compare the performance of the bias-corrected estimator, the pooled min-\(\ell_2\)-norm interpolator, and the target-only min-\(\ell_2\)-norm interpolator across a range of source sample sizes \(n_1\). Figures 9 (a) and 9 (b) demonstrate qualitatively similar but substantively different patterns for different SNR values. In both cases, the pooled min-\(\ell_2\)-norm interpolator outperforms the bias-corrected estimator for sufficiently small \(n_1\), and this range increases as the SNR increases. Note also that the risk of the two transfer estimators peak at different values of \(n_1\): relative to the bias-corrected estimator, the pooled estimator becomes unstable earlier.

17.4 Comparison to null risk↩︎

We briefly note that, for sufficiently small SNR, the pooled min-\(\ell_2\)-norm interpolator is not guaranteed to outperform even the null estimator \(\hat{\boldsymbol{\beta}}_{\text{null}} = 0\). In Figure 10, we replicate Figures 1 (a) and 3 (c), respectively, but include horizontal dotted lines to denote the risk of the null estimator, which always has risk \(\mathbb{E}[\|\hat{\boldsymbol{\beta}}_{\text{null}} - \boldsymbol{\beta}^{(2)}\|_{\boldsymbol{\Sigma}^{(2)}}^2] = \|\boldsymbol{\beta}^{(2)}\|_2^2 = \text{SNR} \cdot \sigma^2\) in these two settings. Indeed, in both Figures 10 (a) and 10 (b), we see that there are typically at least some source sample sizes \(n_1\) for which the pooled min-\(\ell_2\)-norm interpolator outperforms the null estimator. However, for \(\text{SNR} = 1\) in the both figures, we see that the risk of the pooled min-\(\ell_2\)-norm interpolator is greater than that of the null estimator for the entire range of the overparametrized setting.

References↩︎