Related Work

Our proposed model of sampling bias, conditional \(\Gamma\)-biased sampling, builds on previous models for sampling bias [@aronow2013interval; @miratrix2018shape], where samples \(Y_{i}\) are drawn i.i.d. from the target distribution \(Q\) but only included in the training dataset with a latent probability \(\pi_{i} \in [\alpha, \beta]\), for \(\alpha, \beta \in (0, 1]\). Under this model, previous works focus on partial identification of the population mean outcome \(\EE[Q]{Y}\). If we interpret \(\pi_i := \PP[\tilde{Q}]{S_i \cond X_i, \, Y_i}\), then our \(\Gamma\)-biased sampling model as specified in Definition 1 is statistically equivalent to an extension of the model used in [@aronow2013interval] and [@miratrix2018shape] that includes covariates in such a way that we allow the unobserved probability of sample selection \(\pi\) to be arbitrarily affected by the covariates \(X\) but bounds on the amount of unexplained variation in \(\pi_i\). Also, unlike [@aronow2013interval] and [@miratrix2018shape], we focus on learning a robust decision rules rather than on partial identification of moments of \(Q\).

Our model is also connected to the broader literature on sensitivity analysis in causal inference [@andrews2019simple; @dorn2021doubly; @freidling2022optimization; @jin2022sensitivity; @nie2021covariate; @yadlowsky2018bounds], the goal of which is to understand how causal analyses justified by assuming randomized or unconfounded treatment assignment could be affected by a failure of these assumptions. In particular, our \(\Gamma\)-biased sampling model has a similar statistical structure as the \(\Gamma\)-marginal sensitivity model used by [@tan2006distributional] to quantify failures of unconfoundedness. However, in these sensitivity analyses, the concern is typically regarding threats to internal validity (i.e., failures of unconfoundedness), whereas here we model sampling bias as a threat to external validity.

As discussed above, our work fits within the broader DRO literature [@ben2013robust; @shapiro2017distributionally], but because the vast majority of that literature focuses on robustness to global or unconditional shifts the resulting methods and analytic techniques are not directly applicable to our setting. We do note, however, that there are a handful of recent works that also consider robustness sets that place restrictions on conditional shifts [@esteban2021distributionally; @oberst2021regularizing; @thams2022evaluating]. [@esteban2021distributionally] take statistical uncertainty to be the source of the distribution shift and considers shifts in the empirical conditional distribution for subsets of \(\mathcal{X}\) with sufficiently large measure. In contrast, we consider sampling bias, which is present even in the population case with infinite samples, as the source of the distribution shift we seek to be robust against. Furthermore, our problem also requires placing constraints on the conditional shift for every \(x\), not just subsets of \(\mathcal{X}\). [@oberst2021regularizing] leverage access to noisy proxies of unobserved variables for learning models that are robust to shifts in the distribution of unobservables. [@thams2022evaluating] study how to evaluate the worst-case loss under a parametric robustness set, which consists of interpretable, conditional shifts. Our work differs from [@oberst2021regularizing; @thams2022evaluating] in that we do not make any fine-grained assumptions on the nature of the shift, such as access to proxy variables or a parametric form.

We note that the challenge of considering robustness sets that enforce conditional restrictions has also recently been considered in the literature on sensitivity analysis in causal inference [@dorn2021doubly; @jin2022sensitivity; @nie2021covariate; @yadlowsky2018bounds]. Most related to our work, [@dorn2021doubly] consider a robustness set based on the marginal sensitivity model, where pointwise bounds are placed on the ratio between the odds of treatment assignment conditional on observables and unobservables and the odds of treatment assignment based on observables alone, and uses results of [@rockafellar2000optimization] to obtain partial identification bounds on conditional treatment effects. In similar veins, [@yadlowsky2018bounds] consider a robustness set based on the Rosenbaum selection model, where pointwise bounds are placed on the odds ratio of treatment between two units with the same covariates but different unobservables, and [@nie2021covariate] consider a robustness set based on a transport model, where pointwise bounds are placed on the ratio between the conditional distribution of unobservables given observables in two locations. In a departure from conditional robustness sets that enforce pointwise bounds, [@jin2022sensitivity] propose \(f\)-sensitivity model to build robustness sets that consist of distributions under which the expectation of a convex function \(f\) of the odds ratio is bounded; such robustness sets bound the amount of unobserved confounding “on average." These works use these robustness sets to obtain partial identification bounds on treatment effects from observational data that suffers from unmeasured confounding, whereas our goal is to learn a robust decision rule from data that suffers from sampling bias.

Finally, our contribution is related to the broader literature on data-driven decision making. This literature has been active in recent years, including contributions from [@athey2021policy], [@bertsimas2020predictive], [@elmachtoub2022smart], [@foster2019orthogonal], [@kallus2021minimax], [@kitagawa2018should], [@manski2004statistical], [@nie2021quasi], [@stoye2009minimax], [@swaminathan2015batch], [@zhao2012estimating] and [@zhou2022offline]. A recurring theme of this line of work is in choosing loss functions \(L(\cdot)\) that capture relevant aspects of various decision tasks [@bertsimas2020predictive]. Our results pair naturally with this line of work, in that our approach can be applied with generic loss functions to learn decision rules that are robust to potential sampling bias. We also draw attention to [@kallus2021minimax], who consider learning optimal treatment rules from confounded data, i.e., where the “treated” and “control” samples available for training may be biased according to unobservable attributes. Like [@kallus2021minimax], we use robust optimization techniques to learn from data potentially corrupted via biased sampling; however, the type of bias we consider (test/train vs. treatment/control), and resulting algorithmic and conceptual remedies, are different.

Extension to Constrained Function Classes

Thus far, we have focused on learning robust decision rules without any constraints on the functional form of \(h\). But sometimes we may want to impose functional form constraints on \(h\), e.g., we may want to find the best robust linear or tree-shaped predictor under our \(\Gamma\)-biased sampling model. The derivation of RU Regression above relied on the fact that, in the unconstrained case, the optimal robust decision rule is agnostic to the target covariate distribution \(Q_{X}\) as long as it is absolutely continuous with respect to the training covariate distribution \(P_{X}\), and has \(\sup_{x \in \mathcal{X}} {dP_{X}(x)}\,/\,{dQ_{X}(x)} < \infty\). This is because the optimal rule in fact minimizes the worst-case conditional loss for \(x \in \text{supp}(P_{X})\) almost surely. This reduction, however, is only applicable when \(h\) and \(\alpha\) can represent the conditionally-optimal decision rule; and so in particular may not apply if we want to impose functional form constraints on \(h\).

In order to learn robust constrained decision rules under \(\Gamma\)-biased sampling, we can still use a variant of RU-regression; however, explicit weighting to account for any shift in the distribution of observed features \(X\) is now required. Specifically, we show below that a weighted minimizer of the RU loss, where the weights are given by the density ratio between the target and train covariate distribution, still identifies the optimal robust decision rules within a constrained function class.

The above result suggests that if the covariate density ratio \(r\) is known and we aim to learn robust decision rules from a function class \(\mathcal{H}\), which may not necessarily be \(L^{2}(P_{X}, \mathcal{X}),\) then we can consider the following weighted risk minimization problem \[\label{eq:risk95restricted} \inf_{h \in \mathcal{H}} \sup_{Q \in \mathcal{S}_{\Gamma}(P, Q_{X})} \EE[P]{r(X) \cdot L(h(X), Y)} = \inf_{(h, \alpha) \in \mathcal{H} \times L^{2}(P_{X}, \mathcal{X})} \EE[P]{r(X) \cdot L_{\mathrm{RU}}^{\Gamma}(h(X), \alpha(X), Y)}.\tag{20}\] Thus, Weighted RU Regression can be applied to learning a robust decision rule from a constrained function class \(\mathcal{H}.\) Unlike the setting where the decision rule can take value in \(L^{2}(P_{X}, \mathcal{X})\), when the decision rule is restricted to \(\mathcal{H}\), the optimal robust decision rule is not agnostic to the target covariate distribution \(Q_{X}\), and so Weighted RU Regression can only be applied if the target covariate distribution is identifiable. This limitation is inherent to any distributionally robust optimization approach that places restrictions on the conditional distribution \(Y|X\) instead of the joint distribution \((X, Y)\).

Properties of Population RU Risk First, we consider the problem of minimizing the population RU risk with respect to \((h, \alpha)\) over \(L^{2}(P_{X}, \mathcal{X}) \times L^{2}(P_{X}, \mathcal{X})\). We use the following norm on this product space \[||(h, \alpha)||_{L^{2}(P_{X}, \mathcal{X})} = \sqrt{||h||_{L^{2}(P_{X}, \mathcal{X})}^{2} + ||\alpha||_{L^{2}(P_{X}, \mathcal{X})}^{2}}.\] Under the following assumptions, we can show that the population RU risk has a unique minimizer over \(L^{2}(P_{X}, \mathcal{X}) \times L^{2}(P_{X}, \mathcal{X})\).

Since the RU loss is not strongly convex, we require the following additional condition on the data distribution to establish the existence of a unique minimizer.

Building on this characterization of the minimizer, we can show in a \(||\cdot||_{\infty}\)-ball about the minimizer, the population RU loss is strongly convex and smooth. To show smoothness, we require the loss function \(L\) to be \(D\)-smooth (have second derivative upper bounded by \(D\)) for some constant \(0< D < \infty.\) Theorem 7 below implies that in a neighborhood about the minimizer, \[\left|\EE[P]{L_{\text{RU}}^{\Gamma}(h_{\Gamma}^{*}(X), \alpha_{\Gamma}^{*}(X), Y) } - \EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y) }\right| \asymp ||(h_{\Gamma}^{*}, \alpha_{\Gamma}^{*}) - (h, \alpha)||^{2}_{L^{2}(P_{X}, \mathcal{X})},\] and this property will be useful for establishing nonparametric estimation guarantees.

Estimation Guarantees under Hölder-Smoothness

We next demonstrate how the general regularity properties established above translate into convergence guarantees for RU regression in the familiar setting where \(h_{\Gamma}^{*}\) is known to belong to a Hölder class. Optimal estimation in Hölder classes is a widely studied problem [@chen2007large]; and, in particular the minimax-optimal rate of convergence for nonparametric regression over the Hölder class of \(p\)-smooth functions in \(d\) dimensions is known to be \(O_{P}(n^{-\frac{p}{2p+d}})\) [@stone1982optimal].

Studying the behavior of RU regression in this setting provides a transparent benchmark for statistical properties of empirical minimization with the RU loss. When the minimizer of the RU loss belongs to a Hölder class, we demonstrate that the sample-complexity of RU regression is comparable to the minimax sample-complexity of standard nonparametric regression.⁴ In other words, we find that minimax learning under \(\Gamma\)-biased sampling doesn’t meaningfully change the rate of convergence at which we can achieve good performance via empirical minimization.

A first question we need to address is: If we assume that our estimation target \(h_{\Gamma}^{*}\) is \(p\)-smooth in the Hölder sense, what does this imply about the auxiliary parameter \(\alpha_{\Gamma}^{*}\) that emerged from our RU regression construction? Theorem 6 showed that \(\alpha_{\Gamma}^{*}(x)\) is a conditional quantile of the losses, and so any smoothness of \(\alpha_{\Gamma}^{*}\) will depend on the smoothness of the conditional quantile function of the losses—which in turn depends on the smoothness of the conditional distribution of \(Y|X\), the smoothness of the loss function \(\ell\), and the smoothness of \(h_{\Gamma}^{*}\). It is thus not a-priori obvious that \(\alpha_{\Gamma}^{*}\) should generally inherit regularity properties from \(h_{\Gamma}^{*}\); however, as shown below, it does hold that if \(h_{\Gamma}^{*}\) is \(p\)-smooth then \(\alpha_{\Gamma}^{*}\) will also be \(p\)-smooth under mild additional assumptions.

This result motivates learning \(h(\cdot)\) and \(\alpha(\cdot)\) by running RU regression over a function class that can effectively represent \(p\)-smooth functions. The full class of \(p\)-smooth functions is an infinite dimensional space that is challenging to optimize over directly. For this reason, we instead consider the method of sieves [@geman1982nonparametric], where we optimize the empirical risk over a sequence of finite-dimensional sieve spaces \(\mathcal{H}_{1} \times \mathcal{A}_{1} \subseteq \dots \subseteq \mathcal{H}_{J} \times \mathcal{A}_{J} \subseteq \dots\), whose span provides increasingly sharp approximation to all \(p\)-smooth functions as the sieve index \(J\) increases. Empirical risk minimization over the sieve space can then be written as \[\label{eq:erm952} (\hat{h}_{n}, \hat{\alpha}_{n}) \in \argmin_{(h, \alpha) \in \mathcal{H}_{J_{n}} \times \mathcal{A}_{J_{n}}} \hEE[P]{L_{\text{RU}}(h(X), \alpha(X), Y)},\tag{22}\] where \(J_{n}\) corresponds to the size of the sieve basis for a given sample size.

Standard choices of sieves for approximating smooth functions include polynomials and univariate splines [@chen2007large]. For technical reasons, it is helpful to constrain our sieve functions to take values only within a bounded interval; and to accomplish this we follow the truncation strategy of [@jin2022sensitivity]. We refer to Appendix [subsec:truncated95sieve] for formal definitions of truncated polynomial and/or univariate spline sieves used in our analysis.

Obtaining the optimal rate of convergence for sieve estimation requires balancing the estimation error and sieve approximation error [@chen2007large]. Estimation error is given by the error between the empirical RU risk minimizer in the sieve space and the population RU risk minimizer in the sieve space, and can be bounded using the metric entropy of the sieve space. Sieve approximation error is the error that arises from projecting the minimizer over the infinite-dimensional model space \((h_{\Gamma}^{*}, \alpha_{\Gamma}^{*}) \in \Lambda^{p}(\mathcal{X}) \times \Lambda^{p}(\mathcal{X})\) onto a finite-dimensional sieve. To get a handle on the sieve approximation error, our proofs adapt the result from [@timan1963theory] that \[\inf_{(h, \alpha) \in \tilde{\mathcal{H}}_{J_{n}} \times \tilde{\mathcal{A}}_{J_{n}}} || (h, \alpha) - (h_{\Gamma}^{*}, \alpha_{\Gamma}^{*})||_{\infty} = O(J_{n}^{-p}),\] where \(\tilde{\mathcal{H}}_{J_{n}} \times \tilde{\mathcal{A}}_{J_{n}}\) denotes a (non-truncated) polynomial or univariate spline sieve.

We note that a key step in establishing Theorem 9 is demonstrating that \[\left|\EE[P]{L_{\text{RU}}(\hat{h}_{n}(X), \hat{\alpha}_{n}(X), Y)} - \EE[P]{L_{\text{RU}}(h_{\Gamma}^{*}(X), \alpha_{\Gamma}^{*}(X), Y)} \right| \asymp ||(\hat{h}_{n}, \hat{\alpha}_{n}) - (h^{*}_{\Gamma}, \alpha^{*}_{\Gamma}) ||^{2}_{L^{2}(P_{X}, \mathcal{X})}.\] This step combined with the estimation result in Theorem 9 yields the following generalization guarantee \[\label{eq:generalization95guarantee} \left|\EE[P]{L_{\text{RU}}(\hat{h}_{n}(X), \hat{\alpha}_{n}(X), Y)} - \EE[P]{L_{\text{RU}}(h_{\Gamma}^{*}(X), \alpha_{\Gamma}^{*}(X), Y)} \right| \asymp O\Big(\Big(\frac{\log n}{n}\Big)^{\frac{2p}{2p + d}}\Big).\tag{23}\] In particular, in the regime where \(p >\frac{d}{2}\), the guarantee in 23 allows us to obtain “fast” (i.e., faster than \(1/\sqrt{n}\) rates) for generalization.

Notation

Technical Lemmas Our main results rely on the following technical lemmas.

Proof of Lemma 1 First, suppose that \(Q\) generates \(P\) via conditional \(\Gamma\)-biased sampling. We show that ?? holds and that \(\sup_{x \in \mathcal{X}} \frac{dP_{X}(x)}{dQ_{X}(x)} < C\) for some \(C < \infty.\) By definition, we have that \[\begin{align} \frac{dQ_{Y\mid X=x}(y)}{dP_{Y \mid X=x}(y)} &= \frac{d\tilde{Q}_{Y \mid X}(y)}{d\tilde{Q}_{Y \mid X, S=1}(y)} \\ &= \frac{d\tilde{Q}_{X, Y}(x, y)}{d\tilde{Q}_{X}(x)} \cdot \frac{d\tilde{Q}_{X\mid S=1}(x)}{d\tilde{Q}_{X, Y \mid S=1}(x, y)}\\ &= \frac{d\tilde{Q}_{X, Y}(x, y)}{d\tilde{Q}_{X, Y\mid S=1}(x, y)} \cdot \frac{d\tilde{Q}_{X \mid S=1}(x)}{d\tilde{Q}_{X}(x)}. \end{align}\]

Multiplying numerator and denominator by \(\PP[\tilde{Q}]{S=1}\) yields \[\begin{align} \frac{dQ_{Y\mid X=x}(y)}{dP_{Y \mid X=x}(y)} &= \Big( \frac{d\tilde{Q}_{X \mid S=1}(x)}{d\tilde{Q}_{X}(x)} \cdot \PP[\tilde{Q}]{S=1} \Big) \cdot \Big(\frac{d\tilde{Q}_{X, Y}(x, y)}{d\tilde{Q}_{X, Y\mid S=1}(x, y)} \cdot \frac{1}{\PP[\tilde{Q}]{S=1}} \Big) \\ &= \frac{\PP[\tilde{Q}]{S=1 \mid X=x}}{\PP[\tilde{Q}]{S=1 \mid X=x, Y=y}} \tag{43} \\ &\in [\Gamma^{-1}, \Gamma]. \tag{44} \end{align}\] 43 follows from Bayes’ Rule. 44 follows from 4 . So, ?? holds. We also show that the covariate density ratio between \(P\) and \(Q\) is bounded. By Bayes’ rules \[\begin{align} \PP[\tilde{Q}]{S=1, \, X=x} = \PP[\tilde{Q}]{S=1}dP_{X}(x) = dQ_{X}(x) \PP[\tilde{Q}]{S=1 \mid X=x}, \end{align}\] and so \[\frac{dP_{X}(x)}{dQ_{X}(x)} = \frac{\PP[\tilde{Q}]{S=1 \mid X=x}}{\PP[\tilde{Q}]{S=1}} \leq \frac{1}{\PP[\tilde{Q}]{S=1}}\] is in fact uniformly bounded above for all \(x \in \xx\).

Second, we show the converse. Let \(Q\) be a distribution over \((X, Y)\) that satisfies ?? . We define \(\tilde{Q}\) to be a distribution over \((X, Y, S)\), where \(X \in \mathcal{X}, Y \in \mathcal{Y}, S \in \{0, 1\}.\) We set \(\tilde{Q}_{X, Y} = Q\) and define that \[\label{eq:pi} \PP[\tilde{Q}]{S=1 \mid X=x, Y=y} = \frac{1}{N}\cdot \frac{dP(x, y)}{dQ(x, y)},\tag{45}\] where \(N\le \frac{1}{C\Gamma}\). Note that \(\PP[\tilde{Q}]{S =1 \mid X =x , Y=y} \in [0, 1]\) because ?? holds and \(\sup_{x \in \mathcal{X}} \frac{dP_{X}(x)}{dQ_{X}(x)} < C\).

To show that converse holds, we must verify 4 holds for \(\tilde{Q}\) and that \(\tilde{Q}_{X, Y \mid S=1}=P.\) First, we verify 4 . We compute \(\PP[\tilde{Q}]{S=1 \mid X=x}.\) \[\label{eq:original95exp95pi95cond95x} \PP[\tilde{Q}]{S=1 \mid X=x} = \EE[\tilde{Q}]{\frac{1}{N} \cdot \frac{dP(X, Y)}{dQ(X, Y)} \mid X=x} = \frac{1}{N} \cdot \frac{dP_{X}(x)}{dQ_{X}(x)}.\tag{46}\]

Note that \[\PP[\tilde{Q}]{S=1 \mid X=x, Y=y} = \frac{1}{N} \cdot \frac{dP(x, y)}{dQ(x, y)} = \PP[\tilde{Q}]{S=1 \mid X=x} \cdot \frac{dP_{Y|X=x}(y)}{dQ_{Y|X=x}(y)}.\] From ?? , we have that \[\frac{dP_{Y|X=x}(y)}{dQ_{Y|X=x}(y)} \in [\Gamma^{-1}, \Gamma].\] So, we have that 4 holds for \(\tilde{Q}\).

Now, we can verify that \(\tilde{Q}_{X, Y \mid S=1}=P\). We aim to verify that \[\label{eq:accept95reject95result95covariates} d\tilde{Q}_{X, Y \mid S=1}(x, y) = dP(x, y).\tag{47}\]

We have that \[\begin{align} d\tilde{Q}_{X, Y, S=1}(x, y)&= \PP[\tilde{Q}]{S=1 \mid X=x, Y=y} \cdot \PP[\tilde{Q}]{X=x, Y=y} \\ &= \frac{1}{N} \cdot \frac{dP_{X, Y}(x, y)}{dQ_{X, Y}(x, y)} \cdot dQ_{X, Y}(x, y)\\ &= \frac{1}{N} \cdot dP(x, y). \end{align}\] In addition, from 46 , we have that \[\PP[\tilde{Q}]{S=1} = \EE[\tilde{Q}_{X}]{\PP[\tilde{Q}]{S=1 \mid X=x}}= \EE[\tilde{Q}_{X}]{ \frac{1}{N} \cdot \frac{dP_{X}(X)}{dQ_{X}(X)}}= \frac{1}{N}.\] Thus, we have that \[d\tilde{Q}_{X, Y \mid S=1}(x, y) = \frac{d\tilde{Q}_{X, Y, S=1}(x, y)}{\PP[\tilde{Q}]{S=1}} = dP_{X, Y}(x, y).\]Therefore, we have that \(Q\) can generate \(P\) under conditional \(\Gamma\)-biased sampling.

Proof of Lemma 4 Let \(h^{*} \in L^{2}(Q_{X}, \mathcal{X})\) be the solution to 5 . Let the function \(\tilde{h}\) almost surely be minimizer of ?? at every \(x\). Since \(\tilde{h}\) almost surely solves ?? for every \(x \in \text{supp}(Q_{X})\), \[\sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q_{Y|X}]{L(\tilde{h}(X), Y) \mid X=x} \leq \sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q_{Y|X}]{L(h^{*}(X), Y) \mid X=x}.\] Given any marginal distribution \(Q_{X}\), we can marginalize over \(X\) to see that \[\EE[Q_{X}]{\sup_{Q_{Y|X}: Q \in S_{\Gamma}(P)} \EE[Q_{Y|X}]{L(\tilde{h}(X), Y) \mid X} } \leq \EE[Q_{X}]{ \sup_{Q_{Y|X}:Q \in S_{\Gamma}(P)} \EE[Q_{Y|X}]{L(h^{*}(X), Y) \mid X=x}}.\]

Based on our definition of \(S_{\Gamma}(P, Q_{X}),\) we note that for any \(h \in L^{2}(Q_{X}, \mathcal{X})\) \[\sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q]{L(h(X), Y)} = \EE[Q_{X}]{ \sup_{Q_{Y|X}: Q \in S_{\Gamma}(P, Q_{X})} \EE[Q_{Y|X}]{L(h(X), Y)\mid X}}.\] Thus, we have that \[\sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q]{L(\tilde{h}(X), Y)} \leq \sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q]{L(h^{*}(X), Y)}.\] Finally, by definition of \(h^{*}\) we must also have that \[\sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q]{L(h^{*}(X), Y)} \leq \sup_{Q \in S_{\Gamma}(P, Q_{X})} \EE[Q]{L(\tilde{h}(X), Y)}.\] These last two inequalities yield the desired equivalence.

Proof of Theorem 6 First, we establish the existence and uniqueness of the minimizer of the population RU risk. After that, we give a characterization of the optimal \(\alpha_{\Gamma}^{*}\). Finally, we show that there are upper and lower bounds on the value of \(\alpha_{\Gamma}^{*}(x)\) for all \(x\).

Proof of Theorem 7 We define some new notation. Recall that under Assumption 2, we can rewrite \(L(z, y)= \ell(y - z)\). Let \(\ell_{1}^{-1}\) be the inverse of \(\ell(z)\) where \(z > 0.\) Let \(\ell_{2}^{-1}\) be the inverse of \(\ell(z)\) where \(z \leq 0\). Further, let \(0 < \epsilon < \frac{1- \eta(\Gamma)}{2 P_{\max}}.\)

 
Proof of Strong Convexity. We demonstrate that the population RU risk is strongly convex with constant \(\kappa_{1}(\epsilon)\) in an \(||\cdot||_{\infty}\) ball about minimizer with radius \(\delta(\epsilon)\). We show that we can pick the radius \(\delta(\epsilon)\) so that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\), we have that \(0 < M^{-} - \delta(\epsilon) < \alpha < M^{+}_{\Gamma},\) where \(M^{+}_{\Gamma}\) is as defined in 37 .

First, we find a radius \(\delta_{1}(\epsilon)\) where the lower bound will be satisfied. Define \[\label{eq:g} g_{i}(x; h, \alpha) = h(x) + \ell_{i}^{-1}(\alpha(x)).\tag{56}\] By continuity, for every \(\epsilon > 0\), there exists \(0 < \delta_{1}(\epsilon) < M^{-}\) such that if \((h, \alpha) \in \mathcal{C}_{\delta_{1}(\epsilon)}\), then \[\sup_{x \in \mathcal{X}, i\in \{1, 2\}} |g_{i}(x; h, \alpha) - g_{i}(x; h^{*}_{\Gamma}, \alpha^{*}_{\Gamma})| < \epsilon.\] Note that for \((h, \alpha) \in C_{\delta_{1}(\epsilon)}\), we have that \(\alpha > 0\) because \(||\alpha - \alpha_{\Gamma}^{*}||_{\infty} \leq \delta_{1}(\epsilon)\), \(\alpha^{*}_{\Gamma}(x) > M^{-}\) for all \(x \in \mathcal{X}\), and finally, \(\delta_{1}(\epsilon) < M^{-}.\)

Second, we find a radius \(\delta_{2}\) where the upper bound will be satisfied. By Theorem 6 and the definition of \(M^{+}_{\Gamma}\) in 37 , \[\begin{align} \alpha^{*}_{\Gamma}(x) &= q_{\eta(\Gamma)}^{L}(x; h^{*}_{\Gamma}(x)) \leq \sup_{x \in \mathcal{X}, h: ||h||_{\infty} \leq 2B } q_{\eta(\Gamma)}^{L}(x; h^{*}_{\Gamma}(x)) \\ M^{+}_{\Gamma} &= \sup_{x \in \mathcal{X}, h: ||h||_{\infty} \leq 2B } q_{\eta(\Gamma)}^{L}(x; h^{*}_{\Gamma}(x)) + 1. \end{align}\] Let \(\delta_{2} = \frac{1}{2}.\) We note that for \((h, \alpha) < C_{\delta_{2}}\), we have that \(\alpha(x) < \alpha_{\Gamma}^{*}(x) + \frac{1}{2} < M^{+}_{\Gamma}\) for all \(x \in \mathcal{X}.\)

Thus, if we pick the radius of the \(||\cdot||_{\infty}\) to be \(\delta(\epsilon) :=\min(\delta_{1}(\epsilon), \delta_{2})\), then both of these properties will be satisfied. As a result, \(\delta(\epsilon)\) is constant that depends on \(M^{+}_{\Gamma}, M^{-}, \epsilon.\)

Let \(L(h, \alpha), L_{1}(h, \alpha), L_{3}(h, \alpha)\) be shorthand for the population RU risk, the first term of the population RU risk, and the third term of the population RU risk, respectively. \[\begin{align} L(h, \alpha) &= \EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y)},\\ L_{1}(h, \alpha) &= \EE[P]{L_{\text{RU}, 1}^{\Gamma}(h(X), Y)},\\ L_{3}(h, \alpha) &= \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}. \end{align}\] We compute the second Gâteaux derivative of the population RU risk. \[\begin{align} &\langle L''(h, \alpha; \psi, \phi), (\psi, \phi) \rangle \\ &=\langle L_{1}''(h, \alpha; \psi, \phi) + L_{3}''(h, \alpha; \psi, \phi) , (\psi, \phi) \rangle \tag{57}\\ &\geq \langle L_{3}''(h, \alpha; \psi, \phi), (\psi, \phi) \rangle \tag{58} \\ &= (\Gamma - \Gamma^{-1}) \cdot \EE[P_{X}]{\begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} \nabla^{2} T_{3, X}(h(X), \alpha(X)) \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix} } \tag{59}\\ &\geq (\Gamma - \Gamma^{-1}) \EE[P_{X}]{\begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} A_{X}(h(X), \alpha(X)) \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix} } \tag{60} \end{align}\] 57 follows from Lemma 14. 58 holds because \(\langle L''_{1}(h, \alpha; (\psi, \phi), (\psi, \phi)) \rangle \geq 0\) because \(L_{1}(h, \alpha)\) is strongly convex in \(h\) (see Section [subsec:lru95195strongly95convex95h] for a proof) and does not depend on \(\alpha\). Next, 59 follows from Lemma 11. Finally, \(A_{x}(c, d)\) is the lower bound on the Hessisan matrix of \(T_{3, x}(c, d)\) defined in Lemma 13. To develop a lower bound for \(\langle L''(h, \alpha; \psi, \phi), (\psi, \phi) \rangle\), we aim to apply Lemma 16 to \(A_{x}(h(x), \alpha(x)).\)

Before we verify the conditions and apply Lemma 16, we bound quantities that appear in the trace and determinant of \(A_{x}(h(x), \alpha(x))\) when \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}.\) These quantities include \(a_{i, x}\) for \(i \in \{1, 2\}, \sum_{i \in \{1, 2\}} f_{i, x},\) and \(1 - F_{x; h(x)}(\alpha(x))\), where \[\begin{align} a_{i, x} &:= |\ell'(\ell_{i}^{-1}(\alpha(x)))| \quad i=1, 2, \tag{61}\\ f_{i, x} &:= p_{Y|X=x}(h(x) + \ell_{i}^{-1}(\alpha(x))) \quad i=1,2. \tag{62} \end{align}\]

First, we focus on \(a_{i, x}.\) We note that for \((h, \alpha) \in C_{\delta(\epsilon)}\), we have \(\alpha < M^{+}_{\Gamma}\). Since \(|\ell'(\ell_{i}^{-1}(y))|\) is strictly increasing in \(y\) and on \(C_{\delta(\epsilon)}\), \(\alpha(x) \geq M^{-} - \delta(\epsilon)\) for all \(x \in \mathcal{X}.\) So, we have \[0 < C_{M^{-}, \delta(\epsilon)} \leq a_{i, x} \leq C_{M^{+}_{\Gamma}} < \infty \quad i=1, 2, x \in \mathcal{X},\] where \(C_{M^{-}, \delta}\) is defined in 39 and \(C_{M^{+}_{\Gamma}}\) is defined in 38 . We note that \(C_{M^{-}, \delta(\epsilon)}\) is a constant that depends on \(M^{-}\) and \(\delta(\epsilon)\) and \(C_{M^{+}_{\Gamma}}\) is a constant that depends on \(M^{+}_{\Gamma}.\)

Second, we aim to show that \(\sum_{i= \{1, 2\}} f_{i, x}\) is similarly upper and lower bounded. The upper bound is straightforward from Assumption 3. To obtain the lower bound, we first analyze \(\sum_{i \in \{1, 2\}} p_{Y|X=x}(h^{*}_{\Gamma}(x) + \ell_{i}^{-1}(\alpha^{*}_{\Gamma}(x))\), which can be written as \(\sum_{i \in \{1, 2\}} p_{Y|X=x}(g_{i}(x; h^{*}_{\Gamma}, \alpha^{*}_{\Gamma}))\) using the definition of \(g\) in 56 .

For some quantile \(c_{i, x}\), we can write that \[\ell^{-1}_{i}(q_{\eta(\Gamma)}^{L}(x;h^{*}_{\Gamma}(x))) = q^{Y}_{c_{i, x}}(x) - h^{*}_{\Gamma}(x),\] where the first term corresponds to the \(c_{i, x}\)-th quantile of \(Y\), where \(Y\) is distributed following \(P_{Y|X=x}\). We realize that \[\begin{align} \sum_{i \in \{1, 2\}} p_{Y|X=x}(g_{i}(x; h^{*}_{\Gamma}, \alpha^{*}_{\Gamma})) &= \sum_{i \in \{1, 2\}} p_{Y|X=x}(h^{*}_{\Gamma}(x) + \ell_{i}^{-1}(\alpha^{*}_{\Gamma}(x)) \\ &= \sum_{i \in \{1, 2 \}} p_{Y|X=x}(h^{*}_{\Gamma}(x) + \ell_{i}^{-1}(q_{\eta(\Gamma)}^{L}(x; h_{\Gamma}^{*}(x)))) \\ &= \sum_{i \in \{1, 2 \}} p_{Y|X=x}(h^{*}_{\Gamma}(x) + q_{c_{i, x}}^{Y}(x) - h^{*}_{\Gamma}(x)) \\ &= \sum_{i \in \{1, 2 \}} p_{Y|X=x}(q_{c_{i, x}}^{Y}(x)). \end{align}\]

Furthermore, we realize that either \(c_{1, x}\) or \(c_{2, x}\) lies in \([1-\frac{\eta(\Gamma)}{2}, 1 + \frac{\eta(\Gamma)}{2}].\) Because \(q_{\eta(\Gamma)}^{L}(x;h^{*}_{\Gamma}(x))\) corresponds to the \(\eta(\Gamma)\)-th quantile of the conditional losses, we must have that \[\label{eq:c95diff} c_{1, x} - c_{2, x} = \eta(\Gamma).\tag{63}\] In addition, \(c_{1, x} \leq 1\), so \(c_{2, x} \leq 1 - \eta(\Gamma).\) So, \(c_{2, x} \in [0, 1 - \eta(\Gamma)].\) Suppose that \(c_{2, x} \in [1 - \frac{\eta(\Gamma)}{2}, 1 - \eta(\Gamma)],\) then clearly the desired claim holds. If \(c_{2, x} \notin [1 - \frac{\eta(\Gamma)}{2}, 1 - \eta(\Gamma)],\) this means that \(c_{2, x} \in [0, 1 - \frac{\eta(\Gamma)}{2})\). So, we must have that \(c_{1, x} \in [\eta(\Gamma), 1+ \frac{\eta(\Gamma)}{2}).\) Thus, we have that at least one of \(c_{1, x}, c_{2, x}\) lies in the interval \([1-\frac{\eta(\Gamma)}{2}, 1 + \frac{\eta(\Gamma)}{2}].\)

Now, we have that \[\sum_{i \in \{1, 2\}} f_{i, x} = \sum_{i \in \{1, 2\}} p_{Y|X=x}(g_{i}(x; h, \alpha)),\] and \(\delta(\epsilon)\) was chosen so that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\) \[\sup_{x \in \mathcal{X}, i \in \{1, 2\}} |g_{i}(x; h, \alpha) - g_{i}(x; h^{*}_{\Gamma}, \alpha^{*}_{\Gamma})| = \sup_{x \in \mathcal{X}, i \in \{1, 2\}} |g_{i}(x; h, \alpha) - q_{c_{i, x}}^{Y}(x)| < \epsilon.\] Thus, we realize that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\), \[\label{eq:g95h95alp} g_{i}(x; h, \alpha) = q_{c_{i, x}}^{Y}(x) + b_{i}(x), \quad b_{i}(x) \in (-\epsilon, \epsilon), i \in \{1, 2\}, x\in\mathcal{X}.\tag{64}\] So, for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)},\) we realize that a lower bound on \(\sum_{i \in \{1, 2\}} f_{i, x} = \sum_{i \in \{1, 2\}} p_{Y|X}(g_{i}(x; h, \alpha))\) is given by \[P_{\min, \Gamma, \epsilon} := \inf_{c \in [1 - \frac{\eta(\Gamma)}{2}, 1+ \frac{\eta(\Gamma)}{2}], b \in [-\epsilon, \epsilon],x \in \mathcal{X}} p_{Y|X=x}(q_{c}^{Y}(x) + b) \label{eq:C95pl95epsilon}.\tag{65}\] Thus, we have that \[0< P_{\min, \Gamma, \epsilon} \leq \sum_{i \in \{1, 2\}} f_{i, x} \leq 2P_{\max} < \infty \quad i=1,2, x \in \mathcal{X},\] and clearly each \(f_{i, x}\) must be nonnegative.

Third, we aim to show that \(1 - F_{x;h(x)}(\alpha(x))\) is similarly upper and lower bounded on \(\mathcal{C}_{\delta(\epsilon)}.\) Clearly, an upper bound on this quantity is \(1\). To compute the lower bound, we see that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\), \[\begin{align} 1 - F_{x; h(x)}(\alpha(x)) &= 1 - P_{Y|X=x}(g_{1}(x;h, \alpha)) + P_{Y|X=x}(g_{2}(x; h, \alpha)) \\ &= 1 - P_{Y|X=x}(q_{c_{1, x}}^{Y}(x) + b_{1}(x)) + P_{Y|X=x}(q_{c_{2, x}}^{Y}(x) + b_{2}(x)) \quad b_{1}(x), b_{2}(x) \in (-\epsilon, \epsilon) \\ &\geq 1 - c_{1, x} - P_{\max} \cdot \epsilon + c_{2, x} - P_{\max} \cdot \epsilon \\ &= 1 - \eta(\Gamma) -2 P_{\max} \epsilon\\ &> 0. \end{align}\] The first line follows from the definition of \(F\) and \(g_{i}\) from 56 . In the second line, we apply 64 . In the third line, we note that the c.d.f. of \(P_{Y|X=x}\) at \(q_{c_{i, x}}^{Y}(x) + b_{i}(x)\) can be closely approximated by the value of the c.d.f. at \(q_{c_{i, x}}^{Y}(x).\) Next, we apply 63 . The last line follows because \(\epsilon < \frac{1 - \eta(\Gamma)}{2P_{\max}}\). Thus, we have that \[\label{eq:cdf95bound} 1- F_{x;h(x)}(\alpha(x)) \geq 1 - \eta(\Gamma) -2 P_{\max} \epsilon > 0.\tag{66}\]

Now, we finally verify the conditions of Lemma 16. We note that \(A_{x}(h(x), \alpha(x))\) is a symmetric matrix by definition. We realize that \(\tr A_{x}(h(x), \alpha(x)) \geq 0\) because \[\begin{align} \tr A_{x}(h(x), \alpha(x)) &=A_{x, 11}(h(x), \alpha(x)) + A_{x, 22}(h(x), \alpha(x)) \\ &= \sum_{i \in \{1, 2\} } a_{i, x} \cdot f_{i, x} + \sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}} + C (1-F_{x;h(x)}(\alpha(x))) \\ &\geq C (1-F_{x;h(x)}(\alpha(x))) \tag{67} \\ &> 0. \tag{68} \end{align}\] 67 follows from the observation that \(f_{i, x}, a_{i, x} \geq 0\). 68 follows from 66 . In addition, we see that \(\det A_{x}(h(x), \alpha(x)) \geq 0\) because \[\begin{align} &\det A_{x}(h(x), \alpha(x)) \\ &= A_{x, 11}(h(x), \alpha(x)) \cdot A_{x, 22}(h(x), \alpha(x)) - (A_{x, 12}(h(x), \alpha(x)))^{2} \\ &= \Big(\sum_{i \in \{1, 2\} } a_{i, x} \cdot f_{i, x} + C (1 - F_{x; h(x)}(\alpha(x)) )\Big) \cdot (\sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}}) - (f_{1, x} - f_{2, x})^{2} \\ &= \Big(\frac{a_{1, x}}{a_{2, x}} + \frac{a_{2, x}}{a_{1, x}} + 2\Big) \cdot f_{1, x} \cdot f_{2, x} + C (1 - F_{x;h(x)}(\alpha(x)) ) \cdot \Big(\sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}}\Big) \\ &\geq C \cdot (\sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}}) \cdot (1 - F_{x;h(x)}(\alpha(x)) ) \\ &\geq C \cdot \frac{1}{C_{M^{+}_{\Gamma}}} \cdot (\sum_{i \in \{1, 2\}} f_{i, x}) \cdot (1 - \eta(\Gamma) -2 P_{\max} \epsilon) \\ &\geq C \cdot \frac{1}{C_{M^{+}_{\Gamma}}} \cdot P_{\min, \Gamma, \epsilon} \cdot (1 - \eta(\Gamma) -2 P_{\max} \epsilon) \label{eq:detA95lower} \\ &> 0. \end{align}\tag{69}\]

Thus, we can apply Lemma 16 to \(A_{x}(h(x), \alpha(x))\) to see that \[\lambda_{\min}(A_{x}(h(x), \alpha(x)) \geq \frac{\det A_{x}(h(x), \alpha(x))}{\tr A_{x}(h(x), \alpha(x))}.\] We can combine the lower bound on \(\det A\) from 69 with the following upper bound on \(\tr A\) to find a lower bound on \(\lambda_{\min}(A_{x}(h(x), \alpha(x))\) that does not depend on the choice of \(x \in \mathcal{X}\) and \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}.\) \[\begin{align} \tr A_{x}(h(x), \alpha(x))&= \sum_{i \in \{1, 2\} } a_{i, x} \cdot f_{i, x} + \sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}} + C (1- F_{x; h(x)}(\alpha(x))) \\ &\leq 2 P_{\max}(C_{M^{+}_{\Gamma}} + \frac{1}{C_{M^{-}, \delta(\epsilon)}}) + C \\ &= \frac{2 P_{\max}(C_{M^{+}_{\Gamma}} \cdot C_{M^{-}, \delta(\epsilon)} + 1) + C \cdot C_{M^{-}, \delta(\epsilon)}}{C_{M^{-}, \delta(\epsilon)}} \label{eq:trA95upper} \end{align}\tag{70}\] Therefore, applying@eq:eq:trA95upper and 69 , we find that \[\begin{align} &\lambda_{\min}(A_{x}(h(x), \alpha(x)) \\ &\geq \frac{\det A_{x}(h(x), \alpha(x))}{\tr A_{x}(h(x), \alpha(x))} \\ &\geq \Big( C \cdot (1 - \eta(\Gamma) -2 P_{\max} \epsilon) \cdot \frac{1}{C_{M^{+}_{\Gamma}}} \cdot P_{\min, \Gamma, \epsilon} \Big) \cdot \Big( \frac{C_{M^{-}, \delta(\epsilon)}}{2 P_{\max}(C_{M^{+}_{\Gamma}} \cdot C_{M^{-}, \delta(\epsilon)} + 1) + C \cdot C_{M^{-}, \delta(\epsilon)}} \Big) \\ &\geq \frac{C \cdot (1 - \eta(\Gamma) - 2P_{\max} \cdot \epsilon) \cdot P_{\min, \Gamma, \epsilon}}{2 P_{\max} \cdot (C_{M^{+}_{\Gamma}} \cdot C_{M^{-}, \delta(\epsilon)} + 1) + C \cdot C_{M^{-}, \delta(\epsilon)}} \cdot \frac{C_{M^{-}, \delta(\epsilon)}}{C_{M^{+}_{\Gamma}}}. \end{align}\]

We define \(\kappa_{1}(\epsilon)\) as follows \[\begin{align} \kappa_{1}(\epsilon) := (\Gamma - \Gamma^{-1}) \cdot \frac{C \cdot (1 - \eta(\Gamma) - 2 P_{\max} \cdot \epsilon) \cdot P_{\min, \Gamma, \epsilon}}{2 P_{\max} \cdot (C_{M_{\Gamma}^{+}} \cdot C_{M^{-}, \delta(\epsilon)} + 1) + C \cdot C_{M^{-}}} \cdot \frac{C_{M^{-}, \delta(\epsilon)}}{C_{M^{+}}}. \end{align}\]

We realize that for all \(x \in \mathcal{X}, (h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\), we have that \[(\Gamma - \Gamma^{-1}) \cdot A_{x}(h(x), \alpha(x)) \succeq \kappa_{1}(\epsilon) \cdot I_{2}.\] Revisiting 60 , we have that \[\begin{align} \langle L''(h, \alpha; (\psi, \phi)), (\psi, \phi) \rangle &\geq (\Gamma - \Gamma^{-1}) \cdot \EE[P_{X}]{ \begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} A_{X}(h(X), \alpha(X)) \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix} } \\ &\geq \EE[P_{X}]{ \begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} \kappa_{1}(\epsilon) I_{2} \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix} } \\ &= \kappa_{1}(\epsilon) \EE[P_{X}]{ \psi(X)^{2} + \phi(X)^{2}} \\ &= \kappa_{1}(\epsilon) ||(\psi, \phi)||^{2}. \end{align}\]

Thus, \(\EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y)}\) is \(\kappa_{1}(\epsilon)\) strongly convex in \((h, \alpha)\) on \(\mathcal{C}_{\delta(\epsilon)}.\)

We investigate the behavior of \(\kappa_{1}(\epsilon)\) as \(\epsilon \rightarrow 0\). We can show that as \(\epsilon \rightarrow 0\), \(\kappa_{1}(\epsilon) \rightarrow \kappa\), where \[\begin{align} \kappa_{1} &:= (1 - \Gamma^{-1}) \cdot \frac{C \cdot P_{\min, \Gamma}}{2 P_{\max} \cdot (C_{M^{+}_{\Gamma}} \cdot C_{M^{-}} + 1) + C \cdot C_{M^{-}}} \cdot \frac{C_{M^{-}}}{C_{M^{+}_{\Gamma}}} \label{eq:kappa951}, \end{align}\tag{71}\] where \(C_{M^{-}}\) is defined in 40 .

To see this, note that if \(\epsilon \rightarrow 0\), then \(\delta(\epsilon) \rightarrow 0\). So, we have that as \(\epsilon \rightarrow 0\), \(C_{M^{-}, \delta(\epsilon)} \rightarrow C_{M^{-}}\), \(P_{\min, \Gamma, \epsilon} \rightarrow P_{\min, \Gamma}\), and \(\epsilon \cdot P_{\max} \rightarrow 0\). So, we have that \[\begin{align} \lim_{\epsilon \rightarrow 0} \kappa_{1}(\epsilon) &=\lim_{\epsilon \rightarrow 0} (\Gamma - \Gamma^{-1}) \frac{C \cdot (1 - \eta(\Gamma) - 2P_{\max} \cdot \epsilon) \cdot P_{\min, \Gamma, \epsilon}}{2 P_{\max} \cdot (C_{M^{+}_{\Gamma}} \cdot C_{M^{-}, \delta(\epsilon)} + 1) + C \cdot C_{M^{-}, \delta(\epsilon)}} \cdot \frac{C_{M^{-}, \delta(\epsilon)}}{C_{M^{+}_{\Gamma}}} \\ &= (\Gamma - \Gamma^{-1}) \cdot \frac{C \cdot (1 - \eta(\Gamma)) \cdot P_{\min, \Gamma}}{2 P_{\max} \cdot (C_{M^{+}_{\Gamma}} \cdot C_{M^{-}} + 1) + C \cdot C_{M^{-}}} \cdot \frac{C_{M^{-}}}{C_{M^{+}_{\Gamma}}} \\ &= (1 - \Gamma^{-1}) \cdot \frac{C \cdot P_{\min, \Gamma}}{2 P_{\max} \cdot (C_{M^{+}_{\Gamma}} \cdot C_{M^{-}} + 1) + C \cdot C_{M^{-}}} \cdot \frac{C_{M^{-}}}{C_{M^{+}_{\Gamma}}} \\ &= \kappa_{1}. \end{align}\] Thus, as \(\epsilon \rightarrow 0,\) then \(\kappa_{1}(\epsilon) \rightarrow \kappa_{1},\) where \(\kappa_{1}\) is defined in 71 . Finally, we note that \(M_{\Gamma}^{+}\) is a constant that depends on \(B, \Gamma\) (as defined in 37 ) and loss function \(L\), and \(M^{-}\) is a constant that depends on \(P_{\max}\) and loss function \(L\), \(C_{M^{-}}\) depends on \(M^{-}\), and \(C_{M^{+}_{\Gamma}}\) depends on \(M^{+}_{\Gamma}\), so \(\kappa_{1}\) is a constant that depends on \(B, C, P_{\max}, P_{\min, \Gamma}, \Gamma\), and the loss function \(L\).

 
Proof of Smoothness. To show that the population RU risk is \(\kappa_2\)-smooth on \(\mathcal{C}_{\delta(\epsilon)}\), we show that \[\langle L''_{h\alpha}(h, \alpha; \psi, \phi), (\psi, \phi) \rangle \leq \kappa_{2} ||(\psi, \phi)||^{2}_{L^{2}(P_{X}, \mathcal{X})}.\] We have that \[\begin{align} &\langle L''(h, \alpha; \psi, \phi), (\psi, \phi) \rangle \\ &= \langle L''_{1}(h, \alpha; \psi, \phi) + L''_{3}(h, \alpha; \psi, \phi), (\psi, \phi) \rangle \\ &\leq \EE[P_{X}]{\begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} \cdot \Big( \Gamma^{-1} \nabla^{2}T_{1, X}(h(X), \alpha(X)) + (\Gamma - \Gamma^{-1}) \nabla^{2}T_{3, X}(h(X), \alpha(X)) \Big) \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix}}\\ &\leq \EE[P_{X}]{\begin{bmatrix} \psi(X) & \phi(X) \end{bmatrix} \cdot \Big( \Gamma^{-1} \nabla^{2}T_{1, X}(h(X), \alpha(X)) + (\Gamma - \Gamma^{-1}) \nabla^{2}B_{X}(h(X), \alpha(X)) \Big) \begin{bmatrix} \psi(X) \\ \phi(X) \end{bmatrix}}. \end{align}\] The second line follows from Lemma 14. The matrix \(B_{x}(h(x), \alpha(x))\) is as defined in Lemma 13. It suffices to show that there is \(\kappa_{2}(\epsilon)\) such that \[\Gamma^{-1} \nabla^{2}T_{1, x}(h(x), \alpha(x)) + (\Gamma - \Gamma^{-1}) B_{x}(h(x), \alpha(x)) \preceq \kappa_{2}(\epsilon) I_{2} \quad \forall x \in \mathcal{X}.\] Applying Lemma 10 and Assumption 4, we have that \[\label{eq:t195hessian95bound} \nabla^{2}T_{1, x}(h(x), \alpha(x)) = \begin{bmatrix} \EE[P_{Y|X=x}]{\ell''(Y-h(x))} & 0 \\ 0 & 0 \\ \end{bmatrix} \preceq D I_{2}.\\\tag{72}\]

From the proof of strong convexity, for \(0 <\epsilon < \frac{1 - \eta(\Gamma)}{2P_{\max}}\), there exists \(0 < \delta(\epsilon) < M^{-}\) so that for \((h, \alpha) \in C_{\delta(\epsilon)}\), \((\Gamma - \Gamma^{-1}) \nabla^{2}T_{3, x}(h(x), \alpha(x))\) is positive definite. So, on this set \(\mathcal{C}_{\delta(\epsilon)}, B_{x}(h(x), \alpha(x))\) is also certainly positive definite. So, \(B_{x}(h(x), \alpha(x))\) satisfies the conditions of Lemma 16, so we can conclude that \(\lambda_{\max}(B_{x}(h(x), \alpha(x))) \leq \tr B_{x}(h(x), \alpha(x)).\) We can write that \[\tr B_{x}(h(x), \alpha(x)) = \sum_{i \in \{1, 2\} } a_{i, x} \cdot f_{i, x} + \sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}} + \EE[P_{Y|X=x}]{\ell''(Y-h(x))},\] where \(a_{i, x}\) and \(f_{i, x}\) are defined in 61 and 62 , respectively.

We note that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)},\) we have that \(0 < M^{-} - \delta(\epsilon) < \alpha(x) < M_{\Gamma}^{+}\) for all \(x \in \mathcal{X}\). We can use this property to upper bound \(\tr B_{x}(h(x), \alpha(x))\). Recall \(C_{M^{-}, \delta}\) from 39 and \(C_{M^{+}_{\Gamma}}\) from 38 .

\[\begin{align} \tr B_{x}(h(x), \alpha(x)) &= \sum_{i \in \{1, 2\} } a_{i, x} \cdot f_{i, x} + \sum_{i \in \{1, 2\}} \frac{f_{i, x}}{a_{i, x}} + \EE[P_{Y|X=x}]{\ell''(Y-h(x))} \\ &\leq 2 P_{\max}(C_{M^{+}_{\Gamma}} + \frac{1}{C_{M^{-}, \delta(\epsilon)}}) + D. \end{align}\] We arrive at the second inequality by recalling the definition of \(P_{\max}\) from Assumption 3, \(C_{M^{+}_{\Gamma}}\) from 38 , \(C_{M^{-}, \delta}\) from 39 , and \(D\) from Assumption 4. So, we have that \[\label{eq:b95bound} B_{x}(h(x), \alpha(x)) \preceq \Big(2 P_{\max}(C_{M^{+}_{\Gamma}} + \frac{1}{C_{M^{-}}}) + D\Big)I_{2}.\tag{73}\] Let \[\label{eq:kappa95295epsilon} \kappa_{2}(\epsilon) := (\Gamma - \Gamma^{-1}) \cdot \Big(2 P_{\max}\Big(C_{M^{+}_{\Gamma}} + \frac{1}{C_{M_{-}, \delta(\epsilon)}}\Big) \Big) + \Gamma \cdot D.\tag{74}\] Combining the constants from 72 and 73 , we have that for \((h, \alpha) \in \mathcal{C}_{\delta(\epsilon)}\) \[\Gamma^{-1} \nabla^{2}T_{1, x}(h(x), \alpha(x)) + (\Gamma - \Gamma^{-1}) B_{x}(h(x), \alpha(x)) \preceq \kappa_{2}(\epsilon)I_{2} \quad \forall x \in \mathcal{X}.\]Thus, we conclude that \(\EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y)}\) is \(\kappa_{2}(\epsilon)\)-smooth in \((h, \alpha)\) on \(\mathcal{C}_{\delta(\epsilon)}.\) Let \[\kappa_{2} := (\Gamma - \Gamma^{-1}) \cdot \Big(2 P_{\max}\Big(C_{M^{+}_{\Gamma}} + \frac{1}{C_{M^{-}}}\Big) \Big) + \Gamma \cdot D.\] As \(\epsilon \rightarrow 0\), \(\delta(\epsilon) \rightarrow 0\). So, \(C_{M_{-}, \delta(\epsilon)} \rightarrow C_{M^{-}}.\) This implies that \(\kappa_{2}(\epsilon) \rightarrow \kappa_{2}\) as the radius of the \(||\cdot||_{\infty}\)-ball shrinks. Finally, we note that \(M_{\Gamma}^{+}\) is a constant that depends on \(B\) and \(\Gamma\) (as defined in 37 ), \(M^{-}\) is a constant that depends on \(P_{\max}\), and \(C_{M^{-}}\) depends on \(M^{-}\), and \(C_{M_{\Gamma}^{+}}\) depends on \(M_{\Gamma}^{+}\), so \(\kappa_{2}\) is a constant that depends on \(B, D, P_{\max}, \Gamma\) and the loss function \(L\).

Proof of Lemma 8 We demonstrate the smoothness of \(\alpha_{\Gamma}^{*}\) through an application of Theorem 28, an implicit function theorem that we derive for Hölder-smooth functions. First, we specify a function \(v\) that yields an implicit representation of the optimal \(\alpha_{\Gamma}^{*}.\) Second, we verify that \(v\) satisfies the conditions of Theorem 28, which requires (1) defining a neighborhood where the derivative of \(v\) with respect to one of its arguments is a bounded away from zero and (2) guaranteeing that \(v\) lies in a \(p\)-Hölder ball with radius \(b\). Finally, we apply Theorem 28 to guarantee that \(\alpha_{\Gamma}^{*}\) has the desired smoothness.

Details of Sieve Estimators and Proof of Theorem 9

Proof of Corollary 5 We find that

\[\begin{align} \sup_{Q \in \mathcal{S}_{\Gamma}(P, Q_{X})} \EE[Q]{L(h(X), Y)} &= \EE[Q_{X}]{ \inf_{\alpha(x) \in \mathbb{R}} \EE[P_{Y \mid X}]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(x), Y) \mid X=x}} \\ &= \inf_{\alpha \in L^{2}(P_{X}, \mathcal{X})} \EE[Q_{X}]{\EE[P_{Y \mid X}]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y) \mid X=x}} \\ &= \inf_{\alpha \in L^{2}(P_{X}, \mathcal{X})} \EE[P]{r(X) \cdot L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y) }. \end{align}\]

The first line follows from the proof of Theorem 2. The second line follows from the fact that a function \(\alpha \in L^{2}(P_{X}, \mathcal{X})\) minimizes \(\EE[Q_{X}]{\EE[P_{Y \mid X}]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y) \mid X}}\) if and only if it minimizes \(\EE[P_{Y \mid X}]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(x), Y) \mid X=x}\) for every \(x\), which can be seen through an identical argument as the proof of Lemma 4. The last line follows from the definition of \(r\).

Proof of Lemma 10 We have that \(T_{1, x}^{c}(c) = -\EE[P_{Y|X=x}]{\ell'(Y-c)}.\) In addition, \(T_{1, x}^{cc}(c) = \EE[P_{Y|X=x}]{\ell''(Y-c)}.\) So, \(T_{1, x}\) is twice differentiable in \(c\). In addition, we realize that \[\begin{align} \EE[P]{L_{\text{RU}, 1}^{\Gamma}(h(X), Y)} &= \EE[P_{X}]{\EE[P_{Y|X=x}]{L_{\text{RU}, 1}^{\Gamma}(h(X), Y) }} \\ &= \Gamma^{-1}\EE[P_{X}]{T_{X, 1}(h(X))}. \end{align}\]

Proof of Lemma 11

Proof of Lemma 12 We start with two lemmas with proofs presented at the end of this subsection.

Let \[\begin{align} F(h, \alpha) &= \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)} \\ G(h) &= \EE[P]{L_{\text{RU}, 1}^{\Gamma}(h(X), Y)} \\ H(h, \alpha) &= \EE[P]{L_{\text{RU}, 1}^{\Gamma}(h(X), Y)} + \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}. \end{align}\] Note that \[\label{eq:loss95decomp} \EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y)} = H(h, \alpha) + \EE[P]{L_{\text{RU}, 2}^{\Gamma}(\alpha(X))}.\tag{81}\] Since the population RU risk is the sum of \(H\) and a function that is convex in \((h, \alpha)\), then it suffices to show that \(H\) is strictly convex. It remains to show that the conditions of Lemma 22 hold so that we can conclude that \(H\), as defined above, is strictly convex in \((h, \alpha).\) Strong convexity of \(G\) follows from Lemma 23.

 
Verifying Gâteaux differentiability, strong convexity of \(G\), and convexity of \(F\). First, we note that \(F, G\) are both Gâteaux differentiable by Lemma 14.

By the first part of Theorem 2, \(F\) is jointly convex in \((h, \alpha)\).

 
Verifying the strict convexity of \(F\) in \(\alpha\) on \(\mathcal{A}\). Let \(L_{3}(h, \alpha) = \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}.\) By Lemma 14, we have that \(\EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}\) is Gâteaux differentiable in \(\alpha\) with \[L'_{3, \alpha}(h, \alpha; \phi) = (\Gamma - \Gamma^{-1}) \cdot \EE[X]{T^{d}_{3, X}(h(X), \alpha(X)) \cdot \phi(X)}.\] We need to show that for \(\alpha_{1}, \alpha_{2} \in \mathcal{A}\) that differ on a set of positive measure, we have that \[\EE[P_{X}]{(T^{d}_{3, X}(h(X), \alpha_{1}(X)) - T^{d}_{3, X}(h(X), \alpha_{2}(X)) \cdot (\alpha_{1}(X) - \alpha_{2}(X)) } > 0.\]

From Lemma 11, we have that \[\begin{align} T^{d}_{3, x}(h(x), \alpha(x)) &= \begin{cases} t^{d}_{3, x}(h(x), \alpha(x)) & \alpha(x) > 0 \\ -1 & \alpha(x) \leq 0\\ \end{cases}, \end{align}\] where \[t^{d}_{3, x}(h(x), \alpha(x)) = -1 + P_{Y|X=x}(h(x) + \ell_{1}^{-1}(\alpha(x))) - P_{Y|X=x}(h(x) + \ell_{2}^{-1}(\alpha(x))).\] By the definition of \(\ell_{1}^{-1}, \ell_{2}^{-1}\) from Lemma 11, we have that \[\ell_{1}^{-1}(\alpha(x)) > \ell_{2}^{-1}(\alpha(x)).\] Under Assumption 3, we have that \(P_{Y|X=x}\) is strictly increasing, so \[P_{Y|X=x}(h(x) + \ell_{1}^{-1}(\alpha(x))) - P_{Y|X=x}(h(x) + \ell_{2}^{-1}(\alpha(x))) > 0,\] which implies that \[\label{eq:t95d95ineq} t_{3, x}^{d}(h(x), \alpha(x)) > -1.\tag{82}\] Under Assumption 2, \(\ell_{1}^{-1}\) is strictly increasing and \(\ell_{2}^{-1}\) is strictly decreasing. We realize that if \(\alpha_{1}(x) > \alpha_{2}(x),\) then \[\begin{align} P_{Y|X=x}(h(x) + \ell_{1}^{-1}(\alpha_{1}(x))) &> P_{Y|X=x}(h(x) + \ell_{1}^{-1}(\alpha_{2}(x))) \\ P_{Y|X=x}(h(x) + \ell_{2}^{-1}(\alpha_{1}(x))) &< P_{Y|X=x}(h(x) + \ell_{2}^{-1}(\alpha_{2}(x))), \end{align}\] so \[\label{eq:t95d95ineq952} t_{3, x}^{d}(h(x), \alpha_{1}(x)) > t_{3, x}^{d}(h(x), \alpha_{2}(x)).\tag{83}\]

Let \(D = \{ x \in \mathcal{X} \mid \alpha_{1}(x) \neq \alpha_{2}(x) \}.\) Now, we compute \[\begin{align} &\EE[P_{X}]{(T^{d}_{3, X}(h(X), \alpha_{1}(X)) - T^{d}_{3, X}(h(X), \alpha_{2}(X)) \cdot (\alpha_{1}(X) - \alpha_{2}(X)) } \\ &= \EE[P_{X}]{(T^{d}_{3, X}(h(X), \alpha_{1}(X)) - T^{d}_{3, X}(h(X), \alpha_{2}(X)) \cdot (\alpha_{1}(X) - \alpha_{2}(X)) \mathbb{I}(D)} \\ &= \EE[P_{X}]{((t^{d}_{3, X}(h(X), \alpha_{1}(X)) - t^{d}_{3, X}(h(X), \alpha_{2}(X))) \cdot (\alpha_{1}(X) - \alpha_{2}(X)) \cdot \mathbb{I}(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0} \cap D)} \tag{84} \\ &\indent+ \EE[P_{X}]{(t^{d}_{3, X}(h(X), \alpha_{1}(X)) + 1) (\alpha_{1}(X) - \alpha_{2}(X)) \cdot \mathbb{I}(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0}^{c} \cap D) } \tag{85} \\ &\indent+ \EE[P_{X}]{(-1 - t^{d}_{3, X}(h(X), \alpha_{2}(X)) ) (\alpha_{1}(X) - \alpha_{2}(X)) \cdot \mathbb{I}(S_{\alpha_{1}, 0}^{c} \cap S_{\alpha_{2}, 0} \cap D) }. \tag{86} \end{align}\] The first line holds because \((T^{d}_{3, x}(h(x), \alpha_{1}(x)) - T^{d}_{3, x}(h(x), \alpha_{2}(x)) \cdot (\alpha_{1}(x) - \alpha_{2}(x)) = 0\) on \(D^{c}\). The decomposition into 84 , 85 , 86 holds because \(T^{d}_{3, x}(h(x), \alpha_{1}(x)) - T^{d}_{3, x}(h(x), \alpha_{2}(x) = 0\) when \(\alpha_{1}(x) \leq 0\) and \(\alpha_{2}(x) \leq 0.\)

Since we have that \(\alpha_{1}, \alpha_{2} \in \mathcal{A}\) and \(D\) has positive measure, we can show that \(P(S_{\alpha_{1}, 0}^{c} \cap S_{\alpha_{2}, 0}^{c} \cap D) < P(D)\). We consider two cases 1) \(S_{\alpha_{1}, 0} \cap D\) has positive measure and 2) \(S_{\alpha_{1},0} \cap D = \emptyset\). Suppose \(S_{\alpha_{1}, 0} \cap D\) has positive measure, then clearly \[P(S_{\alpha_{1},0}^{c} \cap S_{\alpha_{2},0}^{c} \cap D) \leq P(S_{\alpha_{1}}^{c} \cap D) < P(D).\] If \(S_{\alpha_{1},0} \cap D\) empty, this means that \(\alpha_{1}(x) \leq 0\) for all \(x \in D.\) At the same time, we have that for all \(\alpha \in \mathcal{A},\) \(\alpha(x) \geq 0\) for every \(x \in \mathcal{X}\). So, we must have that \(\alpha_{1} = 0\) on \(D\). We must have that \(\alpha_{2}(x) > 0\) on \(D\), because \(\alpha_{1}, \alpha_{2}\) must differ on \(D\) and \(\alpha_{2}(x) \geq 0\) for all \(x \in \mathcal{X}\). So, this means that \(S_{\alpha_{2}, 0} \cap D\) has positive measure, so \[P(S_{\alpha_{1}, 0}^{c} \cap S_{\alpha_{2}, 0}^{c} \cap D) \leq P(S_{\alpha_{2},0}^{c} \cap D) < P(D).\] So, at least at least one of the sets \(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0} \cap D\), \(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0}^{c} \cap D\), \(S_{\alpha_{1}, 0}^{c} \cap S_{\alpha_{2}, 0} \cap D\) has positive measure.

Suppose \(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0} \cap D\) has positive measure. WLOG, if \(\alpha_{1}(x) > \alpha_{2}(x)\), then \(T^{d}_{3, x}(h(x), \alpha_{1}(x) - T^{d}_{3, x}(h(x), \alpha_{2}(x)) > 0\). In addition, if \(\alpha_{1}(x) < \alpha_{2}(x),\) then \(T^{d}_{3, x}(h(x), \alpha_{1}(x) - T^{d}_{3, x}(h(x), \alpha_{2}(x)) < 0\). Then 84 must be positive. We can use a similar argument to verify that 85 will be positive if \(S_{\alpha_{1}, 0} \cap S_{\alpha_{2}, 0}^{c} \cap D\) has positive measure and 86 will be positive if \(S_{\alpha_{1}, 0}^{c} \cap S_{\alpha_{2}, 0}\) has positive measure. Thus, we conclude that \[\EE[P_{X}]{(T^{d}_{3, X}(h(X), \alpha_{1}(X)) - T^{d}_{3, X}(h(X), \alpha_{2}(X)) \cdot (\alpha_{1}(X) - \alpha_{2}(X)) } > 0\] so \(\alpha \mapsto \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}\) is strictly convex on \(\mathcal{A}\).

Proof of Lemma 13 Now, define a symmetric \(2\times 2\) matrix \(A_{x}(c, d)\) where \[\begin{align} A_{x, 11}(c, d) &= T^{cc}_{3, x}(c, d) - \EE[Y|X=x]{\ell''(Y-c) \mathbb{I}(\ell(Y-c) > d)} + C_{L, l} \cdot \text{Pr}(\ell(Y-c) > d \mid X=x) \\ A_{x, 22}(c, d) &= T^{dd}_{3, x}(c, d) \\ A_{x, 12}(c, d) &= T^{dc}_{3, x}(c, d), \end{align}\] where \(F\) is the distribution over \(\ell(Y-c)\) where \(Y\) follows \(P_{Y|X=x}.\) Under Assumption 2, we have that \(\ell\) is \(C_{\ell, l}\)-strongly convex, so \[\EE[Y|X=x]{\ell''(Y-c) \mathbb{I}(\ell(Y-c) > d)} - C_{L, l} \cdot \text{Pr}(\ell(Y-c) > d | X=x) \geq 0.\] Thus, we have that \[\begin{align} \nabla^{2} T_{3, x}(c, d) - A_{x}(c, d) &= \begin{bmatrix} \EE[Y|X=x]{(\ell''(Y-c) - C_{L, l}) \mathbb{I}(\ell(Y-c) > d)} & 0 \\ 0 & 0 \\ \end{bmatrix} \\ &= \begin{bmatrix} \EE[Y|X=x]{(\ell''(Y-c) \mathbb{I}(\ell(Y-c) > d)} - C_{L, l}\text{Pr}(\ell(Y-c) > d \mid X=x) & 0 \\ 0 & 0 \\ \end{bmatrix}\\ &\succeq 0. \end{align}\]So, \[\label{eq:a} \nabla^{2} T_{3, x}(c, d) \succeq A_{x}(c, d).\tag{87}\]

We can also define a symmetric \(2 \times 2\) matrix \(B_{x}(c, d)\) where \[\begin{align} B_{x, 11}(c, d) &= T^{cc}_{3, x}(c, d) + \EE[Y|X=x]{\ell''(Y-c) \mathbb{I}(\ell(Y-c) \leq d)} \\ B_{x, 22}(c, d) &= T^{dd}_{3, x}(c, d) \\ B_{x, 12}(c, d) &= T^{dc}_{3, x}(c, d). \end{align}\] Under Assumption 2, we have that \(\ell\) is strongly convex, so \[\EE[Y|X=x]{\ell''(Y-c) \mathbb{I}(\ell(Y-c) \leq d)} \geq C_{\ell, l} \cdot \text{Pr}(\ell(Y-c) \leq d | X=x) \geq 0.\] Thus, we have that \[B_{x}(c, d) - \nabla^{2} T_{3, x}(c, d) = \begin{bmatrix} \EE[Y|X=x]{\ell''(Y-c) \mathbb{I}(\ell(Y-c) \leq d)} & 0 \\ 0 & 0 \\ \end{bmatrix}\succeq 0.\]So, \[\label{eq:b} \nabla^{2} T_{3, x}(c, d) \preceq B_{x}(c, d).\tag{88}\] Combining 87 and 88 yields the desired result.

Proof of Lemma 14 Let \(L(h, \alpha) = \EE[P]{L_{\text{RU}}^{\Gamma}(h(X), \alpha(X), Y)}\). First, we verify Gâteaux differentiability with respect to \(\alpha\). We show that the directional derivative of \(L(h, \alpha)\) with respect to \(\alpha\) in the direction \(\phi\) exists for all \(\phi \in L^{2}(P_{X}, \mathcal{X})\). We note that the directional derivative with respect to \(\alpha\) in the direction \(\phi\) is given by \[\begin{align} L'_{\alpha}(h, \alpha; \phi) &= \lim_{\theta \rightarrow 0^{+}} \frac{L(h, \alpha + \theta \phi) - L(h, \alpha)}{\theta}. \end{align}\] We simplify the numerator as follows \[\begin{align} &L(h, \alpha + \theta\phi) - L(h, \alpha) \\ &= \EE[P]{L_{\text{RU}, 2}^{\Gamma}((\alpha+\theta\phi)(X))} + \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), (\alpha + \theta\phi)(X), Y)}\\ &\indent- \EE[P]{L_{\text{RU}, 2}^{\Gamma}(\alpha(X))} - \EE[P]{L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)} \\ &= \theta (1- \Gamma^{-1}) \cdot \EE[P_{X}]{\phi(X)} + (\Gamma - \Gamma^{-1}) \cdot \EE[P_{X}]{T_{3, X}(h(X),(\alpha + \theta \phi)(X)) - T_{3, X}(h(X), \alpha(X))}. \end{align}\] The first line follows because only the second and third term of the RU loss depend on \(\alpha\). The second line follows by Lemma 11. We analyze the second term on the right side of the above equation. We note that by Lemma 11, the map \(T_{3, x}(c, d)\) is differentiable with respect to \(c, d.\) So, \[\begin{align} \lim_{\theta \rightarrow 0^{+}} \frac{T_{3, x}(h(x), \alpha(x) + \theta\phi(x)) - T_{3, x}(h(x), \alpha(x))}{\theta} = T^{d}_{3, x}(h(x), \alpha(x))\phi(x). \\ \end{align}\] Therefore, we have that \[\begin{align} L'_{\alpha}(h, \alpha; \phi) &= \lim_{\theta \rightarrow 0^{+}} (1 - \Gamma^{-1}) \cdot \frac{\theta \EE[P_{X}]{\phi(X)}}{\theta} \\ &\indent+ \lim_{\theta \rightarrow 0^{+}} (\Gamma - \Gamma^{-1}) \cdot \frac{\EE[P_{X}]{T_{3, X}(h(X), \alpha(x) + \theta\phi(X)) - T_{3, X}(h(X), \alpha(X))}}{\theta} \\ &= (1 - \Gamma^{-1}) \cdot \EE[P_{X}]{\phi(X)} + (\Gamma - \Gamma^{-1}) \cdot \EE[P_{X}]{T_{3, X}^{d}(h(X), \alpha(X))\phi(X)} \\ &= \EE[P_{X}]{( (1 - \Gamma^{-1}) + (\Gamma - \Gamma^{-1}) \cdot T^{d}_{3, X}(h(X), \alpha(X)) \cdot \phi(X)}. \end{align}\] Since the directional derivative of \(L(h, \alpha)\) with respect to \(\alpha\) and in the direction \(\phi\) exists for all \(\phi \in \mathcal{A},\) then \(L(h, \alpha)\) is Gâteaux differentiable in \(\alpha\).

We use a similar technique to verify Gâteaux differentiability with respect to \(h\). We show that the directional derivative of \(L(h, \alpha)\) with respect to \(h\) in the direction \(\psi\) exists for \(\psi \in \mathcal{H}.\) We recall that the directional derivative of \(L(h, \alpha)\) with respect to \(h\) in the direction \(\psi\) is given by \[\label{eq:gateaux95h} L'_{h}(h, \alpha; \psi) = \lim_{\theta \rightarrow 0^{+}} \frac{L(h + \theta \psi, \alpha) - L(h, \alpha) }{\theta}.\tag{89}\]

We simplify the directional derivative in 89 as follows. \[\begin{align} L_{h}'(h, \alpha; \psi) &= \lim_{\theta \rightarrow 0^{+}} \frac{L(h + \theta \psi, \alpha) - L(h, \alpha)}{\theta} \\ &= \frac{\EE[P]{L_{\text{RU}, 1}^{\Gamma}((h + \theta \psi)(X), Y) - L_{\text{RU}, 1}^{\Gamma}(h(X), Y)}}{\theta} \\ &\indent + \lim_{\theta \rightarrow 0^{+}} \frac{\EE[P]{L_{\text{RU}, 3}^{\Gamma}((h + \theta \psi)(X), \alpha(X), Y) - L_{\text{RU}, 3}^{\Gamma}(h(X), \alpha(X), Y)}}{\theta} \\ &= \lim_{\theta \rightarrow 0^{+}} \Gamma^{-1} \cdot \frac{\EE[P_{X}]{T_{1, X}(h(X) + \theta \psi(X)) - T_{1, X}(h(X))}}{\theta} \\ &\indent + \lim_{\theta \rightarrow 0^{+}} (\Gamma - \Gamma^{-1}) \cdot \frac{\EE[P_{X}]{T_{3, X}(h(X) + \theta \psi(X),\alpha(X)) - T_{3, X}(h(X), \alpha(X))}}{\theta} \\ &= \EE[P_{X}]{(\Gamma^{-1} \cdot T_{1, X}^{c}(h(X)) + (\Gamma - \Gamma^{-1}) \cdot T_{3, X}^{c}(h(X), \alpha(X)) \cdot \psi(X)}. \end{align}\] The first line follows because only the first and third terms of the RU loss depend on \(h\). The second line follows because of Lemma 10 and Lemma 11. The third line follows from the differentiability of \(T_{1, x}, T_{3, x}\), which is given by Lemmas 10 and 11. Since the directional derivative of \(L(h, \alpha)\) with respect to \(h\) and in the direction \(\psi\) exists for all \(\psi \in L^{2}(P_{X}, \mathcal{X}),\) and the directional derivative can be expressed as a continuous linear function (given the inner product on \(L^{2}(P_{X}, \mathcal{X})\)), then \(L(h, \alpha)\) is Gâteaux differentiable in \(h\).

We can compute second derivatives of \(L(h, \alpha)\) on \(\mathcal{C}\) by applying Lemma 10 and Lemma 11. Note that \(T_{3, x}\) is twice-differentiable when \(d > 0\). For \((h, \alpha) \in \mathcal{C}\), we have that \(\alpha(x) \geq 0.\) We note that the restriction of \(\mathcal{C}\) to the coordinate that corresponds to \(h\) is \(L^{2}(P_{X}, \mathcal{X})\). Let \(\mathcal{A}'\) be the resitrction of \(\mathcal{C}\) to the coordinate that corresponds to \(\alpha\). In the following result, we consider \(\psi_{1}, \psi_{2} \in L^{2}(P_{X}, \mathcal{X})\) and \(\phi_{1}, \phi_{2} \in \mathcal{A}'\). We find that \[\begin{align} L_{hh}''(h, \alpha; \psi_{1}, \psi_{2}) &= L_{1, hh}''(h, \alpha; \psi_{1}, \psi_{2}) + L_{3, hh}''(h, \alpha; \psi_{1}, \psi_{2}) \\ &= \Gamma^{-1} \EE[P_{X}]{T^{cc}_{1, X}(h(X))\psi_{1}(X)\psi_{2}(X)} + (\Gamma - \Gamma^{-1}) \EE[P_{X}]{T_{3, X}^{cc}(h(X), \alpha(X))\psi_{1}(X) \psi_{2}(X)}.\\ L_{h\alpha}''(h, \alpha; \psi_{1}, \phi_{1}) &= L_{3, h\alpha}''(h, \alpha; \psi_{1}, \phi_{1}) \\ &= (\Gamma - \Gamma^{-1}) \EE[P_{X}]{T_{3, X}^{cd}(h(X), \alpha(X))\psi_{1}(X)\phi_{1}(X)}. \\ L_{\alpha\alpha}''(h, \alpha; \phi_{1}, \phi_{2}) &= L_{3, \alpha\alpha}''(h, \alpha; \phi_{1}, \phi_{2}) \\ &= (\Gamma - \Gamma^{-1}) \EE[P_{X}]{T_{3, X}^{dd}(h(X), \alpha(X))\phi_{1}(X)\phi_{2}(X)}. \end{align}\]

Proof of Lemma 15 First, by the Mean Value Theorem, we have that for any \(z \in \mathbb{R},\) \[|\ell'(z)| = |\ell'(z) - \ell'(0)| \leq |\ell''(\tilde{z})| \cdot |z|,\] where \(\tilde{z}\) is between \(z\) and \(0\). By Assumption 4, \(|\ell''(\tilde{z})| \leq C_{L, u},\) so \[\label{eq:l95prime} |\ell'(z)| \leq C_{L, u} \cdot |z|.\tag{90}\] Again, by the Mean Value Theorem, we have that for any \(h \in \Lambda_{c}^{p}(\mathcal{X})\) and \(x \in \mathcal{X}\), \[\begin{align} |L(h(x), y) - L(h^{*}(x), y)| &= |\ell(y - h(x)) - \ell(y - h^{*}(x))| \\ &= |\ell'(y - (\lambda(x) \cdot h(x) + (1- \lambda(x)) \cdot h^{*}(x)) )| \cdot |h(x) - h^{*}(x)| & \lambda(x) \in [0, 1]. \end{align}\] We can define \(\bar{L}(x, y) = |\ell'(y - (\lambda(x) \cdot h(x) + (1- \lambda(x)) \cdot h^{*}(x)) )|.\) Now, we aim to verify that there exists some \(0< M< \infty\) such that \[\sup_{x \in \mathcal{X}} \EE[P_{Y|X}]{\bar{L}(X, Y)^{2} \mid X=x} < M.\] We apply 90 . \[\begin{align} \EE[P_{Y|X}]{\bar{L}(x, Y)^{2} \mid X=x} &= \EE[P_{Y|X}]{(\ell'(Y - (\lambda(x) \cdot h(x) + (1- \lambda(x)\cdot h^{*}(x)) \cdot h^{*}(x)) ))^{2} \mid X=x} \\ &= \EE[P_{Y|X}]{(\ell'(Y - (\lambda(x) \cdot h(x) + (1- \lambda(x))\cdot h^{*}(x))) \cdot h^{*}(x))^{2} \mid X=x} \\ &= \EE[P_{Y|X}]{C_{L, u}^{2} \cdot ((Y - (\lambda(x) \cdot h(x) + (1- \lambda(x))\cdot h^{*}(x))) \cdot h^{*}(x))^{2} \mid X=x } \\ &\lesssim \EE[P_{Y|X}]{Y^{2} \mid X=x} + h(x)^{2} + h^{*}(x)^{2} \\ &\lesssim \sup_{x \in \mathcal{X}} \EE[P_{Y|X}]{Y^{2} \mid X=x} + c^{2}\\ &< \infty. \end{align}\] The last two lines follow from Assumption 5 and 7. Assumption 5 gives that \(h, h^{*} \in \Lambda^{p}_{c}(x),\) so \(|h(x)| \leq c\) and \(|h^{*}(x)|\leq c.\) Assumption 7 gives that \(\sup_{x \in \mathcal{X}} \EE[P_{Y|X}]{Y^{2} \mid X=x}\) is finite. Thus, \(\sup_{x \in \mathcal{X}} \EE[P_{Y|X}]{\bar{L}(X, Y)^{2} \mid X=x} < \infty.\)

Proof of Lemma 16 For any \(2\times 2\) matrix \(A\), \[\lambda_{\max}(A) + \lambda_{\min}(A) = \tr A, \quad \lambda_{\max}(A) \cdot \lambda_{\min}(A) = \det A.\] Since \(\tr A \geq 0\) and \(\det A \geq 0\), it must be that \(\lambda_{\max}(A)\ge \lambda_{\min}(A)\ge 0\). Thus, \[\lambda_{\max}(A)\le \tr A,\] and \[\lambda_{\min}(A) = \frac{\det A}{\lambda_{\max}(A)}\ge \frac{\det A}{\tr A}.\]

Proof of Lemma 24 We note that \[||fg||_{\Lambda^{p}(\mathcal{X})} = \sum_{|\beta|_{1} \leq m} ||D^{\beta}fg||_{\infty} + \sum_{|\beta|_{1} = m} ||D^{\beta} fg||_{\Lambda^{\gamma}(\mathcal{X})}.\] By the product rule, we have that \[D^{\beta} fg = \sum_{\lambda \leq \beta} C(\beta, \lambda) \cdot \partial^{\beta - \lambda} f \cdot \partial^{\lambda} g\] for absolute constants \(C(\beta, \lambda) \geq 0\).

We note that \[\begin{align} ||\partial^{\beta - \lambda} f \cdot \partial^{\lambda} g ||_{\infty} \leq ||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda} g||_{\infty}. \end{align}\] In addition, we have that \[\begin{align} ||\partial^{\beta - \lambda} f \cdot \partial^{\lambda} g||_{\Lambda^{\gamma}(\mathcal{X})} &= \sup_{x, x' \in \mathcal{X}, x \neq x' }\frac{|\partial^{\beta - \lambda}f(x) \cdot \partial^{\lambda}g(x) - \partial^{\beta - \lambda}f(x') \cdot \partial^{\lambda}g(x')|}{|x - x'|^{\gamma}} \\ &= \sup_{x, x' \in \mathcal{X}, x \neq x' } \frac{|\partial^{\beta - \lambda}f(x) \cdot ( \partial^{\lambda}g(x) - \partial^{\lambda}g(x')) + (\partial^{\beta - \lambda}f(x) - \partial^{\beta - \lambda}f(x')) \cdot \partial^{\lambda}g(x')|}{|x - x'|^{\gamma}} \\ &\leq ||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda}g||_{\Lambda^{\gamma}(\mathcal{X})} + ||\partial^{\lambda} g||_{\infty} \cdot ||\partial^{\beta - \lambda} f||_{\Lambda^{\gamma}(\mathcal{X})}. \end{align}\] So, we have that

\[\begin{align} ||fg||_{\Lambda^{p}(\mathcal{X})} &= \sum_{|\beta|_{1} \leq m} ||D^{\beta}fg||_{\infty} + \sum_{|\beta|_{1} = m} ||D^{\beta} fg||_{\Lambda^{\gamma}(\mathcal{X})} \\ &= \sum_{|\beta|_{1} \leq m} || \sum_{\lambda \leq \beta} C(\beta, \lambda) \cdot \partial^{\beta - \lambda} f \cdot \partial^{\lambda} g ||_{\infty} + \sum_{|\beta|_{1} = m} ||\sum_{\lambda \leq \beta} C(\beta, \lambda) \cdot \partial^{\beta - \lambda} f \cdot \partial^{\lambda} g||_{\Lambda^{\gamma}(\mathcal{X})} \\ &\leq \sum_{|\beta|_{1} \leq m} \sum_{\lambda \leq \beta} C(\beta, \lambda) ||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda} g||_{\infty} \\ &\quad+ \sum_{|\beta|_{1} = m} \sum_{\lambda \leq \beta} C(\beta, \lambda) \cdot (||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda}g||_{\Lambda^{\gamma}(\mathcal{X})} + ||\partial^{\lambda} g||_{\infty} \cdot ||\partial^{\beta - \lambda} f||_{\Lambda^{\gamma}(\mathcal{X})}) \\ &\leq C_{m} \cdot \Big( \sum_{|\beta|_{1} \leq m} \sum_{\lambda \leq \beta} ||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda} g||_{\infty} \\ &\quad+ \sum_{|\beta|_{1} = m} \sum_{\lambda \leq \beta} ||\partial^{\beta - \lambda} f||_{\infty} \cdot ||\partial^{\lambda}g||_{\Lambda^{\gamma}(\mathcal{X})} + ||\partial^{\lambda} g||_{\infty} \cdot ||\partial^{\beta - \lambda} f||_{\Lambda^{\gamma}(\mathcal{X})} \Big) \\ &\leq C_{m} \cdot \Big( \sum_{|\beta|_{1} \leq m} \sum_{|\lambda|_{1} \leq m} ||\partial^{\beta} f||_{\infty} \cdot ||\partial^{\lambda} g||_{\infty} + \sum_{|\beta|_{1} \leq m} \sum_{|\lambda|_{1} \leq m} ||\partial^{\beta} f||_{\infty} \cdot ||\partial^{\lambda}g||_{\Lambda^{\gamma}(\mathcal{X})} + ||\partial^{\lambda} g||_{\infty} \cdot ||\partial^{\beta} f||_{\Lambda^{\gamma}(\mathcal{X})} \Big) \\ &\leq C'_{m} \cdot ||f||_{\Lambda^{p}(\mathcal{X})} \cdot ||g||_{\Lambda^{p}(\mathcal{X})}. \end{align}\]

Proof of Lemma 25 We use the Faa di Bruno formula for the chain rule.

Let \(x_{0} \in \mathcal{X}\). Let \(y_{0} = t(x_{0}).\) Let \(u(x) := v(x, t(x)).\) Let \(v^{(\beta, r)}(x_{0}) = \frac{\partial^{|\beta|_{1} + r}}{\partial x_{1}^{\beta_{1}} \dots \partial x_{d}^{\beta_{d}} \partial t^{r}} v(x, y_{0})\) for any \(\beta, r\) with \(|\beta|_{1} + r \leq m.\) Let \(g(x) := (g_{1}(x), g_{2}(x), \dots g_{d}(x), t(x)),\) where \(g_{i}(x) = x_{i}.\) Define \(\{\ell_{i}\}_{i=1}^{q}\) as the complete listing of vectors \(\ell \in \mathbb{Z}_{+}^{d}\) with \(\ell \leq \beta\) where \([\ell] > 0.\)

We can express \(\lambda\) from multivariate Faa di Bruno formula as \(\lambda := (\beta', r),\) where \(\beta' \in \mathbb{Z}_{d}^{+}\) and \(r \in \mathbb{Z}_{+}.\) Applying multivariate Faa di Bruno gives that \[\begin{align} D^{\beta} u(x_{0}) &= \sum_{\substack{(\beta', r) \in \mathbb{Z}_{+}^{d+1}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot v^{(\beta', r)}(x_{0}) \prod_{j=1}^{q} [D^{\ell_{j}} g(x_{0})]^{k_{j}}, \end{align}\] where \[P(\beta, \beta', r) = \left\{ (k_{1}, k_{2}, \dots k_{q}) \big\vert k_{i} \in \mathbb{Z}_{+}^{d+1},\, |k_{i}|_{1} \geq 0,\, \sum_{i=1}^{q} k_{i} = (\beta', r), \, \sum_{i=1}^{q} |k_{i}|_{1} \cdot \ell_{i} = \beta\right\}.\]

Proof of Lemma 26 When \(p \leq 1\), we show that \(u\in \Lambda^{p^{2}}(\mathcal{X}).\) Let \(x, x' \in \mathcal{X}.\) \[\begin{align} |u(x') - u(x)| &= |v(x', t(x')) - v(x, t(x))| \\ &\leq ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot |(x, t(x)) - (x', t(x'))|_{2}^{p} \\ &\leq C ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot (|x - x'|_{2}^{p} + |t(x) - t(x')|_{2}^{p}) \\ &\leq C ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot (|x - x'|_{2}^{p} + ||t||_{\Lambda^{p}(\mathcal{X})}^{p} \cdot |x - x'|_{2}^{p^{2}}) \\ &\leq C ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot |x - x'|_{2}^{p - p^{2}} \cdot |x - x'|^{p^{2}}_{2} + C||v||_{\Lambda^{p}(\mathcal{X})} \cdot ||t||_{\Lambda^{p}(\mathcal{X})}^{p} |x - x'|_{2}^{p^{2}} \\ &\leq C ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot \Big(\sup_{x, x' \in \mathcal{X}} |x - x'|_{2}^{p - p^{2}}\Big) \cdot |x - x'|^{p^{2}}_{2} + C||v||_{\Lambda^{p}(\mathcal{X})} \cdot ||t||_{\Lambda^{p}(\mathcal{X})}^{p} |x - x'|_{2}^{p^{2}} \\ &\leq K |x - x'|_{2}^{p^{2}}. \end{align}\] The first line holds because \(v \in \Lambda^{p}(\mathcal{X} \times \mathcal{T})\). The second line follows because \(||z_{1} + z_{2}||^{p} \leq 2^{p}(||z_{1}||^{p} + ||z_{2}||^{p})\) for any \(p \geq 0.\) The third line holds because \(t \in \Lambda^{p}(\mathcal{X}).\) Finally, we use the compactness of \(\mathcal{X}\) to demonstrate that there exists some constant \(K \in \mathbb{R}_{+}\) such that \(|u(x') - u(x)| \leq K ||x - x'||_{2}^{p^{2}}.\) Thus, \(u \in \Lambda^{p^{2}}(\mathcal{X}).\)

With no additional assumptions, we can conclude that \(u \in \Lambda^{\min\{ p, p^{2}\} }(\mathcal{X}).\)

If we additionally have that \(p \leq 1\) and \(|\frac{\partial }{\partial t} v(x, t)| \leq L\), we have that \[\begin{align} |u(x') - u(x)| &= |v(x', t(x')) - v(x, t(x))| \\ &= |v(x', t(x')) - v(x', t(x)) + v(x', t(x)) - v(x, t(x))| \\ &\leq |v(x', t(x')) - v(x', t(x))| + |v(x', t(x)) - v(x, t(x))| \\ &\leq L \cdot |t(x') - t(x)| + ||v||_{\Lambda^{p}(\mathcal{X})} |x' - x|^{\gamma}_{2} \\ &= L \cdot ||t||_{\Lambda^{p}(\mathcal{X})} \cdot |x' - x|^{\gamma}_{2} + ||v||_{\Lambda^{p}(\mathcal{X})} |x' - x|^{\gamma}_{2} \\ &= (L \cdot ||t||_{\Lambda^{p}(\mathcal{X})} + ||v||_{\Lambda^{p}(\mathcal{X})}) \cdot |x - x'|^{\gamma}_{2}. \end{align}\]

Thus, \(u \in \Lambda^{p}(\mathcal{X})\) and \(||u||_{\Lambda^{p}(\mathcal{X})} \leq ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} + L \cdot ||t||_{\Lambda^{p}(\mathcal{X})}.\)

When \(p > 1,\) we show that \(u \in \Lambda^{p}(\mathcal{X})\). Clearly, \(u \in C^{m}(\mathcal{X})\) because \(v \in C^{m}(\mathcal{X} \times \mathcal{T})\) and \(t \in C^{m}(\mathcal{X}).\) By Lemma 24, it remains to show that for any \(\beta \in \mathbb{Z}_{+}^{d}\) where \(|\beta|_{1} = m\), \(D^{\beta} u\) is a sum of product of functions that lie in \(\Lambda^{\gamma}(\mathcal{X})\), so this implies that \(D^{\beta} u \in \Lambda^{\gamma}(\mathcal{X}).\) From Lemma 25, \(D^{\beta} u(x)\) is the sum of products of \(v^{(\beta', r)}\), \(D^{\ell_{j}}t(x)\), and \(D^{\ell_{j}} g_{i}(x)\), where these terms are defined in Lemma 25. The values of \((\beta', r)\) involved in the sum satisfy \(|\beta'| + r \leq |\beta|_{1} = m\), so we have that \(v^{(\beta', r)} \in \Lambda^{p - |\beta'| - r} \subset \Lambda^{\gamma}(\mathcal{X}).\) We note that by definition \(\ell_{j} \leq \beta\), so \([\ell_{j}] \leq m\). Since \(t(x) \in \Lambda^{p}(\mathcal{X})\), we have that \(D^{\ell_{j}}t \in \Lambda^{p- [\ell_{j}]}(\mathcal{X}) \subset \Lambda^{\gamma}(\mathcal{X}).\) In addition \(g_{i}(x) \in \Lambda^{q}(\mathcal{X})\) for any \(q < \infty\) for all \(i=1, 2, \dots d\), so we also have that \(D^{\ell_{j}} g_{i}(x) \in \Lambda^{\gamma}(\mathcal{X}).\) Thus, we have that \(D^{\beta} u \in \Lambda^{\gamma}(\mathcal{X}),\) so \(u \in \Lambda^{p}(\mathcal{X}).\)

We use the property 91 ; we bound on \(||u||_{\Lambda^{p}(\mathcal{X})}\) by obtaining a bound on \(||D^{\beta}u||_{\Lambda^{p-1}(\mathcal{X})}\), where \(\beta \in \mathbb{Z}_{+}^{d}\). We recall from Lemma 25 that \[\begin{align} D^{\beta} u = \sum_{\substack{\beta' \in \mathbb{Z}_{+}^{d},\, r \in \mathbb{Z}_{+}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot v^{(\beta', r)}(x_{0}) \prod_{j=1}^{q} [D^{\ell_{j}} g(x_{0})]^{k_{j}}, \end{align}\] where \(g(x) = (x, t(x))\), \(\ell_{j} \in \mathbb{Z}_{+}^{d}\), \(\ell_{j} \leq \beta\), and \(\sum_{j=1}^{q} [k_{j}] \cdot \ell_{j} = \beta.\)

As an intermediate step to bounding the Hölder norm of \(||D^{\beta}u||_{\Lambda^{p-1}(\mathcal{X})}\), we bound \(||D^{\ell_{j}} g_{i}||_{\Lambda^{p}(\mathcal{X})}\) and \(||\prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}}||_{\Lambda^{p-1}(\mathcal{X})}\). for some choice of \(\beta', r\) and \(k_{1}, k_{2}, \dots k_{q} \in P(\beta, \beta', r).\) Since we have that \(g_{i}(x) = x_{i}\), we have that \(||D^{\ell_{j}} g_{i}||_{\Lambda^{p-1}(\mathcal{X})}\) is a constant that depends only on \(||x_{i}||_{\infty}\). So, we can say that there exists \(C > 0\) such that \(||D^{\ell_{j}} g_{i}||_{\Lambda^{p-1}(\mathcal{X})} \leq C\). We have that \[\begin{align} \Big|\Big|\prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}}\Big|\Big|_{\Lambda^{p-1}(\mathcal{X})} &= \Big|\Big| \prod_{j=1}^{q} \Big( (D^{\ell_{j}} t)^{k_{j}, d+1} \cdot \prod_{i=1}^{d} (D^{\ell_{j}} g_{i})^{k_{j, i}} \Big) \Big|\Big|_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C_{p-1} \cdot \prod_{j=1}^{q} ||D^{\ell_{j}} t||^{k_{j}, d+1}_{\Lambda^{p-1}(\mathcal{X})} \cdot \prod_{i=1}^{d} ||(D^{\ell_{j}} g_{i})||^{k_{j, i}}_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C'_{p-1} \cdot \prod_{j=1}^{q} ||D^{\ell_{j}} t||^{k_{j}, d+1}_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C'_{p-1} \cdot ||D^{\ell_{j}} t||^{m}_{\Lambda^{p-1}(\mathcal{X})}. \end{align}\] The second line follows from Lemma 24. The third line follows from the observation that \(D^{\ell_{j}} g_{i} = 1\) if \(\ell_{j}= e_{i}\), where \(e_{i}\) is the \(i\)-th unit vector in \(\mathbb{Z}^{d}_{+}\), otherwise \(D^{\ell_{j}} g_{i}= 0\), so \(||D^{\ell_{j}} g_{i}||_{\Lambda^{p-1}(\mathcal{X})} \leq 1\), so we can view \(\prod_{i=1}^{d} ||(D^{\ell_{j}} g_{i})||^{k_{j, i}}_{\Lambda^{p-1}(\mathcal{X})}\) as a constant with no dependence on \(d\).

We bound the Hölder norm of the RHS. \[\begin{align} || D^{\beta} u ||_{\Lambda^{p-1}(\mathcal{X})} &\leq \Big|\Big| \sum_{\substack{\beta' \in \mathbb{Z}_{+}^{d},\, r \in \mathbb{Z}_{+}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot v^{(\beta', r)} \prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}} \Big|\Big|_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq \sum_{\substack{\beta' \in \mathbb{Z}_{+}^{d},\, r \in \mathbb{Z}_{+}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot || v^{(\beta', r)} \prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}} ||_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq \sum_{\substack{\beta' \in \mathbb{Z}_{+}^{d},\, r \in \mathbb{Z}_{+}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot || v^{(\beta', r)} \prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}} ||_{\Lambda^{p-1}(\mathcal{X})} \\ &= \sum_{\substack{\beta' \in \mathbb{Z}_{+}^{d},\, r \in \mathbb{Z}_{+}: \\ 1 \leq |\beta'| + r \leq |\beta|_{1}}} \sum_{P(\beta, \beta', r)} C(\beta, \beta', r, k_{1}, k_{2}, \dots k_{q}) \cdot C_{p-1}|| v^{(\beta', r)}||_{\Lambda^{p-1}(\mathcal{X})} \cdot || \prod_{j=1}^{q} [D^{\ell_{j}} g]^{k_{j}} ||_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C_{\beta} \cdot ||v^{(\beta', r)}||_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \cdot C'_{p-1} \cdot ||D^{\ell_{j}} t||_{\Lambda^{p-1}(\mathcal{X})}^{m} \\ &= C'_{\beta, p-1} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot ||t||_{\Lambda^{m}(\mathcal{X})}^{p}. \end{align}\]

Thus, there exists \(C_{p} > 0\) such that \[||u||_{\Lambda^{p}(\mathcal{X})} \leq C_{p} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot||t||_{\Lambda^{p}(\mathcal{X})}^{m}.\]

Proof of Lemma 27 We consider \(p \leq 1\), so \(m=0\) and \(\gamma \in (0, 1]\). For any \(x, x' \in \mathcal{X}\), we have that \[\begin{align} \Big| \frac{1}{t(x)} - \frac{1}{t(x')} \Big| &= \Big| \frac{t(x') - t(x)}{t(x) t(x')} \Big| \\ &= \frac{|t(x') - t(x)| }{|t(x)| \cdot |t(x')|} \\ &\leq \frac{1}{(K_{1})^{2}} \cdot ||t||_{\Lambda^{p}(\mathcal{X})} \cdot |x' - x|^{p}_{2}. \end{align}\]

Thus, we also have that \(1/t \in \Lambda^{p}(\mathcal{X}).\) In addition, there exists a constant \(C_{K_{1}}\) such that \(||\frac{1}{t}||_{\Lambda^{p}(\mathcal{X})} \leq C_{K_{1}} \cdot ||t||_{\Lambda^{p}(\mathcal{X})}.\)

Now we consider the case where \(p > 1\). Let \(v(x, t) = 1/t\), where \(K = ||t(x)||_{\infty}\) which is finite because \(\mathcal{X}\) is compact and \(t\) is continuous and define \(\mathcal{T} = [K_{1}, K_{2}]\). Then we have that \(v(x, t) \in \Lambda^{p}(\mathcal{X} \times \mathcal{T}).\) Thus, we can consider the function \(u(x) := v(x, t(x)) = \frac{1}{t(x)}\) and apply Lemma 26 to see that \(u = 1/t \in \Lambda^{p}(\mathcal{X})\) and \(||u||_{\Lambda^{p}(\mathcal{X})} \leq C_{p} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot ||t||_{\Lambda^{p}(\mathcal{X})}^{m}.\) Since the function \(v\) only depends on its argument \(t\), we realize that \(||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} = ||v||_{\Lambda^{p}(\mathcal{T})}\) and consequently, there exists \(C_{K_{1}, K_{2}} > 0\) such that \(||v||_{\Lambda^{p}(\mathcal{T})} < C_{K_{1}, K_{2}}.\) Thus, we have that for \(p >1,\) \(||\frac{1}{t}||_{\Lambda^{p}(\mathcal{X})} \leq C_{p} \cdot C_{K_{1}, K_{2}} \cdot ||t||^{m}_{\Lambda^{p}(\mathcal{X})}.\)

Proof of Theorem 28 We prove the result by induction. We start with the case that \(p \leq 1\), so \(m=0\) and \(\gamma \in (0, 1].\) Let \(x, x' \in \mathcal{X}\). By the Mean Value Theorem and the fact that \(v\) is strictly increasing in \(t\), we have that \[\begin{align} |v(x, t(x)) - v(x, t(x'))| \geq K_{1} \cdot |t(x') - t(x)|. \end{align}\] Also, since \(v \in \Lambda^{p}(\mathcal{X} \times \mathcal{T})\) \[|v(x', t(x')) - v(x, t(x'))| \leq ||v||_{\Lambda^{\gamma}(\mathcal{X})} \cdot |x - x'|_{2}^{\gamma}.\] Combining these results, we have that \[\begin{align} c \cdot |t(x') - t(x)| &\leq |v(x, t(x)) - v(x, t(x'))|. \end{align}\] We can use the fact that \(v(x, t(x)) = v(x', t(x')) = 0\) to rewrite the RHS of the above inequality. \[\begin{align} |v(x, t(x)) - v(x, t(x'))| &= |v(x', t(x')) - v(x, t(x'))| \\ &= |v(x', t(x')) - v(x, t(x'))| \\ &\leq ||v(\cdot, t(x'))||_{\Lambda^{\gamma}(\mathcal{X})} \cdot |x - x'|_{2}^{\gamma}. \end{align}\] Thus, \[|t(x') - t(x)| \leq \frac{||v(\cdot, t(x'))||_{\Lambda^{\gamma}(\mathcal{X})}}{K_{1}} \cdot |x - x'|_{2}^{\gamma},\] so \(t \in \Lambda^{\gamma}(\mathcal{X})\) and there is a constant \(C_{K_{1}, K_{2}, \gamma}\) such that

\[\label{eq:p95less95one} ||t||_{\Lambda^{\gamma}(\mathcal{X})} \leq C_{K_{1}, K_{2}, \gamma} \cdot ||v||_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})}.\tag{93}\]

Next we show that when \(m = 1\), \(D^{\beta}t\) exists for any \(\beta\) with \(|\beta|_{1} = 1\) and \(j= 1, 2, \dots, d\), \[\label{eq:deriv} \frac{\partial}{\partial x_{j}} t(x) = -\frac{ \frac{\partial}{\partial x_{j}} v(x, t) \big\vert_{t=t(x)} }{ \frac{\partial}{\partial t} v(x, t) \big\vert_{t =t(x)}}.\tag{94}\]

We note that the previous case proves that \(t\) is Lipschitz because \(t \in \Lambda^{\gamma}(\mathcal{X})\) for \(\gamma= 1\). Again, let \(x, x' \in \mathcal{X}\). We have that \(|t(x') - t(x)| = O(|x' - x|_{2}).\) We note that \[v(x', t(x')) - v(x', t(x)) + v(x', t(x)) - v(x, t(x)) = 0.\] Since \(m = 1\), we can apply the Mean Value Theorem twice to the above equation to see that \[\label{eq:mvt} \frac{\partial}{\partial t} v(x', t) \big\vert_{t=t(x)} \cdot (t(x') - t(x)) + \sum_{j=1}^{d} \frac{\partial}{\partial x_{j}} v(x, t) \big\vert_{t = t(x)} \cdot (x_{j} - x'_{j}) = o(|x - x'|_{2}).\tag{95}\] Since \(v \in \Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T}),\) then \(\frac{\partial}{\partial t} v(x, t) \in \Lambda^{\gamma}(\mathcal{X} \times \mathcal{T}),\) so \[\frac{\partial}{\partial t} v(x', t) \big\vert_{t = t(x)} = \frac{\partial}{\partial t} v(x, t) \big\vert_{t=t(x)} + O(|x - x'|^{\gamma}_{2}).\] Plugging this into 95 , we have that \[\label{eq:mvt952} \Big(\frac{\partial}{\partial t} v(x, t) \big\vert_{t=t(x)} + O(|x - x'|^{\gamma}_{2}) \Big) \cdot (t(x') - t(x)) + \sum_{j=1}^{d} \frac{\partial}{\partial x_{j}} v(x, t) \big\vert_{t = t(x)} \cdot (x_{j} - x'_{j}) = o(|x - x'|_{2}).\tag{96}\] Since \(t(x) - t(x') = O(|x - x'|_{2}),\) 96 simplifies to \[\begin{align} \frac{\partial}{\partial t} v(x, t) \big\vert_{t=t(x)} \cdot (t(x') - t(x)) + \sum_{j=1}^{d} \frac{\partial}{\partial x_{j}} v(x, t) \big\vert_{t = t(x)} \cdot (x_{j} - x'_{j}) = o(|x - x'|_{2}). \end{align}\] The above equation implies 94 . We this form 94 and the property 91 to obtain a bound on \(||t||_{\Lambda^{1 + \gamma}(\mathcal{X})}\) by obtaining an upper bound on \(||\frac{\partial }{\partial x_{j}} t ||_{\Lambda^{\gamma}(\mathcal{X})}.\)

As a preliminary, we upper bound \[\Bigg|\Bigg| \frac{\frac{\partial}{\partial x_{j}} v(x, t) }{ \frac{\partial}{\partial t} v(x, t) } \Bigg| \Bigg|_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})}.\] Denote \(f(x,t) = \frac{\partial}{\partial x_{j}} v(x, t)\) and \(g(x, t) = \frac{\partial}{\partial t} v(x, t).\) Recall that \(f, g \in \Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})\), \(|f| \leq L\) and \(g(x, t) \in [K_{1}, K_{2}]\). By 91 , we must have that \(||f||_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})}, ||g||_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} \leq ||v||_{\Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T})}.\) \[\begin{align} \Big| \frac{f(x, t)}{g(x,t)} - \frac{f(x', t')}{g(x',t')} \Big| &=\Big| \frac{f(x, t)}{g(x,t)} - \frac{f(x', t')}{g(x, t)} + \frac{f(x', t')}{g(x, t)} - \frac{f(x', t')}{g(x', t')} \Big| \\ &\leq \frac{1}{g(x, t)} \cdot |f(x,t) - f(x', t')| + |f(x', t')| \cdot \frac{|g(x, t) - g(x', t')|}{|g(x, t)| \cdot |g(x', t')|} \\ &\leq \frac{1}{K_{1}} \cdot |f(x,t) - f(x', t')| + L \cdot \frac{|g(x, t) - g(x', t')|}{(K_{1})^{2}} \\ &= C_{K_{1}, K_{2}, L} \cdot (||f||_{\Lambda^{\gamma}(\mathcal{X})} + ||g||_{\Lambda^{\gamma}(\mathcal{X})}) \cdot |(x, t) - (x', t')|^{\gamma}_{2} \\ &=C_{K_{1}, K_{2}, L} \cdot ||v||_{\Lambda^{1 + \gamma}(\mathcal{X})} \cdot |(x, t) - (x', t')|^{\gamma}_{2} \end{align}\] This implies that \[\label{eq:first95part95comp} \Big| \Big| \frac{f}{g} \Big|\Big|_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} = \Big| \Big| \frac{\frac{\partial}{\partial x_{j}} v(x, t) }{ \frac{\partial}{\partial t} v(x, t) } \Big| \Big|_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} \leq C_{K_{1}, K_{2}, L} ||v||_{\Lambda^{1 + \gamma}(\mathcal{X})}.\tag{97}\] We can use this result to show that \(||\frac{\partial }{\partial x_{j}} t ||_{\Lambda^{\gamma}(\mathcal{X})} \leq ||v||_{\Lambda^{1 + \gamma}(\mathcal{X})}.\)

We have that \[\begin{align} \Big|\Big|\frac{\partial }{\partial x_{j}} t \Big|\Big|_{\Lambda^{\gamma}(\mathcal{X})} &= \Bigg| \Bigg| \frac{\frac{\partial}{\partial x_{j}} v(x, t) }{ \frac{\partial}{\partial t} v(x, t) } \Big|_{t=t(x)} \Bigg| \Bigg|_{\Lambda^{\gamma}(\mathcal{X})}\\ &\leq \Bigg| \Bigg| \frac{\frac{\partial}{\partial x_{j}} v(x, t) }{ \frac{\partial}{\partial t} v(x, t) } \Bigg| \Bigg|_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} + K_{2} \cdot ||t||_{\Lambda^{\gamma}(\mathcal{X})} \\ &\leq C_{K_{1}, K_{2}, L} ||v||_{\Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T})} + K_{2} ||t||_{\Lambda^{\gamma}(\mathcal{X})} \\ &\leq C_{K_{1}, K_{2}, L} ||v||_{\Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T})} + C_{K_{1}, K_{2}, p-1} \cdot ||v||_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} \\ &= C'_{K_{1}, K_{2}, L, p} ||v||_{\Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T})}. \end{align}\] The first line follows from 94 . The second line is due to the norm of the composition of Hölder functions in Lemma 26. The third line is due to 97 . The fourth line is due to 93 . The fifth line holds because \(||v||_{\Lambda^{\gamma}(\mathcal{X} \times \mathcal{T})} \leq ||v||_{\Lambda^{1 + \gamma}(\mathcal{X} \times \mathcal{T})}.\) Thus, there exists \(C_{K_{1}, K_{2}, L, p, d} > 0\) such that \[||t||_{\Lambda^{1 + \gamma}(\mathcal{X})} \leq C_{K_{1}, K_{2}, L, p, =d} \cdot ||v||_{\Lambda^{1 + \gamma}(\mathcal{X})}.\]

We are left to prove the result for \(p > 1.\) We prove a stronger claim that \[\label{eq:stronger95claim} ||t||_{\Lambda^{p}(\mathcal{X})} \leq ||v||^{k_{p}}_{\Lambda^{p}(\mathcal{X})},\tag{98}\] where \(k_{p} = 1\) for any \(p<2\) and \(k_{p}=m+(m-1) \cdot k_{p-1}.\) We note that showing that this claim holds yields our desired result in ?? because \(k_{p}\geq 1,\) so \[k_{p}= m(k_{p-1}+1)-1\leq m(k_{p-1}+1)\leq m! (k_{p-m} + 1)\leq 2m!.\]

We prove by induction that under the conditions of Theorem 28, 98 holds. Suppose that this result holds for \(p-1\). Since \(v(x, t) \in \Lambda^{p}(\mathcal{X} \times \mathcal{T}) \subset \Lambda^{p-1}(\mathcal{X} \times \mathcal{T}),\) the inductive hypothesis implies that \(t \in \Lambda^{p-1}(\mathcal{X})\) and \[\label{eq:inductive95hypothesis} ||t||_{\Lambda^{p-1}(\mathcal{X})} \leq C_{K_{1}, K_{2}, L, p-1, d} \cdot ||v||^{k_{p-1}}_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})}.\tag{99}\]

We aim to show that \(t \in \Lambda^{p}(\mathcal{X})\) and the desired bound on the norm ?? holds. Note that \[||t||_{\Lambda^{p}(\mathcal{X})} \leq \sum_{j=1}^{d} \Big|\Big| \frac{\partial}{\partial x_{j}} t(x)\Big|\Big|_{\Lambda^{p-1}(\mathcal{X})}\] by 91 . So, we bound the norm by bounding \(|| \frac{\partial}{\partial x_{j}} t(x)||_{\Lambda^{p-1}(\mathcal{X})}\) and we recall that \(\frac{\partial}{\partial x_{j}} t(x)\) is given by 94 . As a preliminary, we note that since \(v \in \Lambda^{p}(\mathcal{X} \times \mathcal{T})\), we have that \(\frac{\partial}{\partial x_{j}} v(x, t), \frac{\partial}{\partial t} v(x, t) \in \Lambda^{p-1}(\mathcal{X} \times \mathcal{T}).\) We analyze \[\begin{align} \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \cdot \frac{1}{\frac{\partial}{\partial t} v(x, t) } \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} &\leq C_{p-1} \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \cdot \Big| \Big| \frac{1}{\frac{\partial}{\partial t} v(x, t) } \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \\ &\leq C_{p-1} \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \cdot C_{K_{1}} \cdot \Big|\Big| \frac{\partial}{\partial t} v(x, t) \Big|\Big|^{m-1}_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \\ & \leq C_{K_{1}, p-1} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}^{m-1} \\ & \leq C_{K_{1}, p-1} ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}^{m}. \end{align}\] The first line follows from the product rule for Hölder functions in Lemma 24. The second line follows from the rule for reciprocals of Hölder functions in Lemma 27. The third line holds by definition of the Hölder norm. Thus, we have that \[\label{eq:first95part95comp95high95order} \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \cdot \frac{1}{\frac{\partial}{\partial t} v(x, t) } \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \leq C_{K_{1}, p-1} ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}^{m}.\tag{100}\]

Now, we can obtain a bound on \(|| \frac{\partial}{\partial x_{j}} t||_{\Lambda^{p-1}(\mathcal{X})}.\) \[\begin{align} \Big| \Big| \frac{\partial}{\partial x_{j}} t \Big| \Big|_{\Lambda^{p-1}(\mathcal{X})} &= \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \cdot \frac{1}{\frac{\partial}{\partial t} v(x, t) } \Big\vert_{t=t(x)} \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \\ &\leq C_{p-1} \cdot \Big| \Big| \frac{\partial}{\partial x_{j}} v(x, t) \cdot \frac{1}{\frac{\partial}{\partial t} v(x, t) } \Big| \Big|_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})} \cdot ||t||^{m-1}_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C'_{p-1} ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}^{m} \cdot ||t||^{m-1}_{\Lambda^{p-1}(\mathcal{X})} \\ &\leq C''_{p-1} ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}^{m} \cdot \Big(C_{K_{1}, K_{2}, L, p-1, d} \cdot ||v||^{k_{p-1}}_{\Lambda^{p-1}}\Big)^{m-1} \\ &\leq C''_{K_{1}, K_{2}, L, p-1, d} \cdot ||v||_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})}^{m + k_{p-1} \cdot (m-1)}\\ &= C''_{K_{1}, K_{2}, L, p-1, d} \cdot ||v||_{\Lambda^{p-1}(\mathcal{X} \times \mathcal{T})}^{k_{p}}. \end{align}\] The first line follows by definition. The second line follows from the composition of Hölder functions in Lemma 26. The third line is due to 100 . The fourth line is due to the inductive hypothesis 99 . Thus, we show by induction that 98 holds and as a consequence, ?? holds and \(t \in \Lambda^{p}(\mathcal{X}).\)

Proof of Corollary 29 Note that \(\mathcal{X}\) is compact because \(f\) is continuous and \(\mathcal{T}\) is compact. Since \(f\) is strictly increasing, it is invertible, so \(f(f^{-1}(x)) - x = 0.\) Let \(v: \mathcal{X} \times \mathcal{T} \rightarrow \mathbb{R}\), where \(v(x, t) := f(t) - x\) Clearly, \(v(x, t) \in \Lambda^{p}(\mathcal{X} \times \mathcal{T}).\) In addition, \(\frac{\partial}{\partial t} v(x, t) = f'(t) > 0.\) Since \(\mathcal{T}\) is compact, we have that \(\inf_{t \in \mathcal{T}} f'(t) > 0.\) In addition, \(v\) has bounded derivative with respect to \(x\). We note that \(||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}\) can be upper bounded by a constant that depends on \(c\). By Theorem 28, \(f^{-1} \in \Lambda^{p}(\mathcal{X})\). If \(p \leq 1\), then \(||f^{-1}||_{\Lambda^{p}} \leq C_{K_{1}, K_{2}, p} \cdot ||v||_{\Lambda^{p}(\mathcal{X} \times \mathcal{T})}.\) If \(p > 1\), then a bound on \(||f^{-1}||_{\Lambda^{p}(\mathcal{X})}\) is given by a constant that depends on \(c, K_{1}, K_{2}, p, d\) by applying ?? .

Proof of Lemma 30 This result holds through an application of Theorem 1 of [@brezis2019sobolev]. For completeness, we state the full theorem below.

Let \(W^{r, q}\) denote the fractional-order Sobolev space [@brezis2018gagliardo] where \(0 < r < 1\) and \(1 \leq q \leq \infty\). Let \(\mathcal{X}\) be a compact set \(\mathbb{R}^{d}.\) Let \(s_{1}, s_{2}, r, p_{1}, p_{2}, q, \theta, d\) satisfy the following conditions \[\begin{align} &0 \leq s_{1} \leq s_{2},\,\quad r \geq 0,\,\quad 1 \leq p_{1}, p_{2}, q \leq \infty,\,\quad (s_{1}, p_{1}) \neq (s_{2}, p_{2}),\,\quad \theta \in (0, 1), \\ &r < s := \theta s_{1} + (1 - \theta)s_{2}, \\ &\frac{1}{q} = \Big( \frac{\theta}{p_{1}} + \frac{1 - \theta}{p_{2}} \Big) - \frac{s - r}{d}. \end{align}\] Then the Gagliardo-Nirenberg-Sobolev (GNS) inequality holds (with a few exceptions): \[||f||_{W^{r, q}(\mathcal{X})} \leq C \cdot ||f||^{\theta}_{W^{s_{1}, p_{1}}(\mathcal{X})} \cdot ||f||^{1 - \theta}_{W^{s_{2}, p_{2}}(\mathcal{X})} \quad \forall f \in W^{s_{1}, p_{1}}(\mathcal{X}) \cap W^{s_{2}, p_{2}}(\mathcal{X}),\] where \(C\) is a constant that depends on \(s_{1}, s_{2}, r, p_{1}, p_{2}, q, d\) but not \(\theta.\) The exceptions include

1. \(d=1\), \(s_{2}\) is an integer \(\geq 1\), \(1 < p_{1} < \infty,\) \(p_{2}=1\), \(s_{1} = s_{2} -1 + \frac{1}{p_{1}},\) \(s_{2} + \frac{\theta}{p_{1}} - 1 < r< s_{2} + \frac{\theta}{p_{1}} - \theta,\) and \(r \geq s_{1}.\)

2. \(d \geq 1,\) \(p_{1} = \infty,\) \(1 < p_{2} < \infty\), \(q = \infty\), \(s_{1} = r \geq 0\) is an integer, \(s_{2} = r + \frac{d}{p_{2}}\) (for every \(\theta \in (0, 1)\)).

First, we note that we can express the spaces \(L^{\infty}(\mathcal{X}), L^{2}(\mathcal{X}),\) and \(\Lambda^{p}(\mathcal{X})\) as (fractional order) Sobolev spaces [@brezis2018gagliardo]. In particular, \(L^{q}(\mathcal{X}) = W^{0, q}(\mathcal{X})\) and \(\Lambda^{p}(\mathcal{X}) = W^{p, \infty}(\mathcal{X}).\) The definition of the norm on general \(W^{r,q}(\mathcal{X})\) spaces can be found in [@brezis2018gagliardo], though we will not use it.

We apply Theorem 1 of [@brezis2019sobolev] with the following choice of constants: \[\begin{align} r = s_{1} = 0,\, \quad q =p_{2}= \infty,\, \quad p_{1} = 2, \quad s_{2} = p,\, \quad \theta = \frac{2p}{2p + d}, \quad s = \frac{pd}{2p + d}. \end{align}\] It is straightforward to verify that these constants satisfy the conditions of the theorem (and do not fall under the exceptions): \(s_{1}, s_{2}, r \in \mathbb{R}_{+}\), \(1 \leq p_{1}, p_{2}, q \leq \infty,\) \(\theta \in (0, 1)\), \(r < s\), \[\theta s_{1} + (1 - \theta) s_{2} = (1 - \theta) p = \Big(1 - \frac{2p}{2p + d}\Big) \cdot p = \frac{pd}{2p+ d} = s,\] and \[\Big(\frac{\theta}{p_{1}} + \frac{1 - \theta}{p_{2}} \Big) - \frac{s - r}{d} = \frac{p}{2p + d} - \frac{1}{d} \cdot \frac{pd}{2p + d} = 0 = \frac{1}{q}.\] Thus, we have that for \(f \in W^{0, 2}(\mathcal{X}) \cap W^{p, \infty}(\mathcal{X})\) \[|| f||_{W^{0, \infty}(\mathcal{X})} \leq C \cdot || f||_{W^{0, 2}(\mathcal{X})}^{\frac{2p}{2p+d}} \cdot || f||^{1 -\frac{2p}{2p+d}}_{W^{p, \infty}},\] where \(C\) is a constant that does not depend on \(f\). The above inequality is equivalent to \[|| f||_{L^{\infty}(\mathcal{X})} \leq C \cdot || f||_{L^{2}(\mathcal{X})}^{\frac{2p}{2p+d}} \cdot || f||^{1 - \frac{2p}{2p+d}}_{\Lambda^{p}(\mathcal{X})}.\] Since \(f \in \Lambda^{p}_{c}(\mathcal{X})\), we have that \(||f||_{\Lambda^{p}(\mathcal{X})} \leq 2c.\) Thus, we have that the above inequality is equivalent to \[|| f||_{L^{\infty}(\mathcal{X})} \leq C \cdot || f||_{L^{2}(\mathcal{X})}^{\frac{2p}{2p+d}} \cdot(2c)^{1 - \frac{2p}{2p+d}},\] which yields the desired result.