1 Introduction↩︎

Mean estimation is an ever-present problem: Medical researchers wish to understand the prevalence of disease, machine learning engineers want to understand the average performance of their models, and so on. Often, practitioners have access to a large set of unlabeled examples, e.g., clinical notes that may or may not record the presence of a disease, images with unknown labels, etc. A typical approach is to gather (expensive) gold-standard labels for a small number of randomly selected examples, and take a simple average of the relevant metric across them.

In many scenarios, it is possible to construct cheap pseudo-labels using machine learning models or other heuristics. For instance, large language models (LLMs) and vision-language models (VLMs) can achieve reasonable “zero shot” performance on a variety of tasks [1]. These pseudo-labels have the potential to improve estimation: For instance, if they were perfect substitutes for gold-standard labels, they would dramatically expand our sample size without introducing bias. In fact, it is already becoming a routine practice to use pseudo-labels from LLMs in “LLM-as-Judge” evaluations [2].

However, the quality of these pseudo-labels is often unknown a priori, and the naive approach of treating them as true standard labels can lead to erroneous conclusions. Prediction-Powered Inference (PPI) [3] proposes an estimator that is unbiased for any set of pseudo-labels, by using the small labeled dataset (on which pseudo-labels can also be obtained) to estimate and correct for any pseudo-label bias. While PPI is unbiased, it is not guaranteed to improve variance / statistical efficiency, particularly when the pseudo-labels have weak correlation with the true labels. To this end, [4] proposed Power-Tuned PPI (PPI++), which uses the labeled dataset to infer the pseudo/true label correlation and discount the pseudo-labels if this correlation is small. Informed by analysis showing that the (asymptotic) variance of PPI++ is never greater than the variance of the classical estimator that only uses labeled examples, they note (emphasis added)

However, this guarantee is asymptotic in nature, applying in the regime where the number of labeled examples becomes large, while the highest-value use of PPI++ comes precisely when labels are scarce. This leaves an open question: If the labeled sample is already too small to give a reliable estimate of the mean, when can we trust its assessment of the pseudo-labels?

We resolve this question by providing an exact non-asymptotic expression for the finite-sample variance of PPI++ under a sample-splitting implementation, and in the regime where the unlabeled dataset is taken to be arbitrarily large. Our expression holds for any joint distribution of pseudo- and true labels that has bounded fourth moments.

This analysis yields practical insights. First, our analysis produces a “no free lunch” result: PPI++ pays a price attributable to errors in estimating the correlation between pseudo- and true labels. If this correlation is small, then the costs of PPI++ outweigh the benefits. From our general results, we derive expressions for the minimum number of labeled samples that are required for PPI++ to improve upon the classical estimator. While our results apply to any distribution, they simplify considerably for Gaussian and binary labels. For Gaussian labels, the correlation must be larger than \((n-2)^{-1/2}\) for PPI++ to improve over the classical estimator, where \(n\) is the labeled sample size.¹

Second, our results allow for direct calculations of the benefit / downside of using PPI vs PPI++ vs.classical estimators, depending on the assumed quality of pseudo labels and available size of labeled samples. Thus, practitioners can operationalize our framework into a recommended course of action by applying broad assumptions of the region in which the quality of their pseudo-labels lie. This process is described in 1. However, absent any assumption of the quality of the pseudo-labels, our method does not purport to identify the best performing method: that would fall into the same free-lunch trap that we caution against. Our conclusion is that PPI++ is a useful tool, but requires practitioners to implicitly assume that their pseudo-labels are “good enough”, a notion we define precisely in this work. Our overall contributions can be summarized as follows:

• In 4.1, we analyze PPI++ in the Gaussian setting, and provide interpretable bounds on the correlation/sample size required for PPI++ to outperform the classical estimator that only uses the labeled sample.

• In 4.2, we analyze PPI++ with sample splitting and cross-fitting, for general joint pseudo/true label distributions. We provide an exact expression for the variance of this unbiased estimator with finite labeled samples, and showing that similar intuitions hold as in the Gaussian case.

• In 4.3, we return to PPI++ without sample splitting, showing that it has two undesirable properties: It is biased, and standard variance-estimation approaches can produce overly optimistic estimates of its variance.

• In 5, we illustrate our findings on real data, highlighting settings where the use of PPI++ hurts variance, and show that PPI++ without sample splitting tends to draw overly narrow confidence intervals.

2 Related Work↩︎

The term “Prediction-Powered Inference” was coined by [3], who introduced the method in the context of both mean estimation and more general inference tasks. Recent work has proposed various extensions to PPI and PPI++, including their use with multiple estimands using additional shrinkage [5], their use with cross-fitting of predictors [6], and their use in the context of active statistical inference [7]. Others have proposed the use of PPI for application in clinical trials [8], causal inference [9], social science research [10], and the efficient evaluation of AI systems [11], [12].

PPI shares similarities with doubly-robust estimators in causal inference, as noted by [7], and can be seen as a special case of the Augmented Inverse Propensity Weighted (AIPW) estimator [13]. Through this lens, the unbiased nature of PPI (even when pseudo-labels are uninformative) is mirrored by the consistency of AIPW estimators under incorrect outcome-regression models. The core idea of PPI++ (taking linear combinations of unbiased estimators to minimize variance) dates back at least as far as [14]. However, the theoretical analysis of these estimators is often limited to potentially loose non-asymptotic error bounds [15] or study of their asymptotic efficiency. A distinct but related problem arises in the study of combining biased and unbiased estimators, where the bias of the former can be estimated from data. In that setting, a similar “no free lunch” phenomenon has been observed, where these methods improve asymptotic variance for any fixed bias, but perform poorly when the bias is comparable in magnitude to \(1/\sqrt{n}\) [16]–[18]. While our analysis strategy and problem setting differs substantially, our core result shows a conceptually similar point, that PPI++ tends to under-perform the classical estimator when the correlation between pseudo-labels and true labels is smaller than \(\approx 1 / \sqrt{n}\).

PPI++ can also be seen as a control variate method, where an unbiased estimator using \(n\) samples is additively combined with a mean-zero, correlated control variable to reduce variance. In the control variate literature, it is well known that estimating the optimal coefficient for the control variate from the same \(n\) samples can lead to variance inflation. For instance, [19] give a variance-inflation factor for Gaussian data which corresponds precisely to the threshold for improvement that we recover in our “warm up” on the Gaussian setting in 4.1 for the single-sample PPI++ estimator with infinite \(N\). We go further, however, in providing thresholds in the Gaussian setting for both a cross-fit variant of PPI++ and the finite \(N\) case. Sample-splitting is also proposed as a “remedy” in that literature for bias and over-optimistic variance estimates that can arise from re-using the same sample [20], analogous to the cross-fitting variant of PPI++ we analyze. However, to our knowledge, prior work characterizes variance only under the simplified setting of Gaussian data or asymptotically, while our main contribution (4.2) is an exact, distribution-free finite-sample expression.

Close connections exist between PPI, PPI++, and estimators from the survey sampling literature. In particular, as noted by [21], the PPI estimator for the mean is equivalent to the difference estimator, and the PPI++ estimator is equivalent to the generalized regression estimator (GREG) [22], [23]. Both estimators arise from a similar motivation (in survey sampling) to the motivation of PPI: The availability of auxiliary information (i.e., covariates) on many unsurveyed (i.e., unlabeled) units, and the use of predictive models to incorporate that information has a long history [24].

However, despite the depth of literature on the topic of similar estimators across causal inference, control variate methods, and survey sampling, analysis of performance is often asymptotic in nature, with the characterization of the exact finite-sample variance (or mean squared estimation error) perceived to be intractable. For instance, [25], a textbook on survey sampling estimators, states (see Chapter 9.4., page 203) that “It is not possible to accurately calculate the expectation and variance of the regression estimator” before introducing standard approximations. We perform that seemingly intractable exact calculation in our work, using the insight that this analysis becomes tractable precisely in the setting that motivates PPI++, where the model generating pseudo-labels is fixed, and an extremely large unlabeled dataset is available.

The theoretical characterization of PPI++ in other related work is largely limited to characterizing upper bounds on the benefits. [26] studies the limits of the benefits of PPI++, but focus on establishing upper bounds on the theoretical benefit, in the (optimistic) scenario where the optimal weight is chosen exactly. Likewise, [27], whose proposed method is largely identical to PPI++ in the mean estimation setting, do not consider finite-sample behavior of their estimator beyond establishing that the bias is \(O(1/n)\).

Also of interest is Cross-PPI [6] (distinct from the cross-fitting variant of PPI++ discussed in this work) which uses a portion of the data to train a predictor. We find there is no need to “demystify” the performance claim of Cross-PPI, since [6] already gives some sense of the trade-offs that come with poor pseudo-labels. In contrast, PPI++ stands out for its claim to “always” do no worse than classical, even if the predictions are poor. That said, if there existed a version of Cross-PPI with a similar “power-tuning” component, we believe our core insight (no free lunch) would carry over.

3 Setup and Preliminaries↩︎

Notation: We use upper-case letters to denote a random variable (e.g., \(Y\)) and lower-case letters to denote a particular value (e.g., \(y\)). We use \(P_Y\) to denote the distribution of a variable \(Y\). We denote the mean of a random variable \(Y\) by \(\mu_y\), the variance by \(\sigma_{y}^2\), and the covariance between two random variables \(Y, X\) by \(\sigma_{yx}\) or \(\sigma_{xy}\). Given \(n\) observations \(\{x_i,y_i\}_{i=1}^n\) of two random variables \(X, Y\), the empirical covariance is given by \(\hat{\sigma}_{xy} = \ch(X,Y) = \frac{1}{n-1}\sum_{i=1}^n(x_i - \bar{x}_n)(y_i - \bar{y}_n)\) where \(\bar{x}_n\) and \(\bar{y}_n\) are the sample means of \(x\) and \(y\) respectively over \(n\) samples.

Problem Setting: We focus on the mean estimation problem: Our goal is to estimate the expectation of a variable (or “label”) \(Y\), denoted \(\theta \mathrel{\vcenter{:}}= \mathbb{E}[Y]\).² We have access to a dataset of size \(n\) (typically small), containing both covariates \(X\) and true labels \(Y\). Meanwhile, we have access to a dataset of size \(N\) (typically much larger), which only contains unlabeled examples \(X\). In this setting, we make the standard assumption that both samples are drawn from a common distribution. This assumption holds in the common scenario where a random subset of the entire dataset is chosen for labeling. We assume the ability to obtain cheap pseudo-labels \(F\) on both the labeled and unlabeled datasets. One way to obtain such labels is via a function \(f(X)\) (e.g., querying an LLM) that can be queried to get pseudo-labels, which motivates our use of upper-case \(F\) to denote these pseudo-labels. The classical estimator simply takes the average of the labeled examples to estimate the mean.

Prediction Powered Inference: Prediction Powered Inference is a family of estimators that combine the labeled dataset of \(n\) samples \(\cD_n = \{y_i, f_i\}_{i=1}^n\) with the larger set of \(N\) pseudo-label samples \(\{f_j\}_{j=1}^N\). For mean estimation, the original  estimator takes the following form.

Notice that  is unbiased for any \(f\). That is, \(\mathbb{E}[\thetahp] = \mathbb{E}[Y] + \mathbb{E}[F] - \mathbb{E}[F] = \mathbb{E}[Y]\). However, the variance of  depends on the variance of \(Y - F\), which can be worse than the variance of \(Y\) alone if \(F\) is not sufficiently correlated with \(Y\). Motivated by this concern, the  estimator [4] uses a data-driven parameter \(\lambda\) to weight the contribution of the pseudo-labels. For mean estimation, the  estimator takes the following form.

Note that if \(\lambda = 0\), then  reduces to , and if \(\lambda = 1\), then  reduces to . [4] note that the variance-optimal choice of \(\lambda^*\) is given by \(\lambda^* = (\sigma_{fy} / ((1 + \frac{n}{N})\sigma_f^2)\), which motivates the “plug-in” estimator of \(\hat{\lambda}\) above. We make two observations: First, the optimal value \(\lambda^*\) increases with higher covariance between the pseudo-labels and the true labels (\(\sigma_{fy}\)), and with lower variance in the pseudo-labels (\(\sigma_{f}^2)\). Both quantities need to be estimated from data, but the covariance in particular needs to be estimated using labeled data: Notably, the true labels \(Y\) are used for both (a) estimating the mean, and (b) estimating the quality of the pseudo-labels.

In many practical applications, the unlabeled sample size \(N\) is extremely large, and in our work, we consider it to be effectively infinite, such that \(\sigma_{f}^2\) and \(\mu_f\) can be estimated with arbitrary precision, and the optimal tuning parameter \(\lambda^*\) is given by \(\sigma_{yf} / \sigma_{f}^2\). Notably, even when \(N\) is infinite,  still relies heavily on the small labeled sample of size \(n\), which in practice may be very small. We now describe two variants of , which vary in how they use the labeled data to obtain their estimate of the optimal \(\lambda\). The first approach is to use the entire labeled sample for both (a) estimating \(\hat{\lambda}\), and (b) estimating \(\hat{\theta}\), which we refer to as .

This estimator is simple, but the lack of sample splitting can complicate variance estimation and the construction of confidence intervals. For example, as we show in [prop:optimistic95variance95single95sample95ppi4343], when the same data are used to both (a) construct \(\hat{\lambda}\), which minimizes the empirical variance, and (b) estimate the empirical variance while treating \(\hat{\lambda}\) as fixed, the resulting variance estimates can be overly optimistic. Cross-fitting is a standard solution to this type of problem: splitting the labeled data, using one half to estimate \(\lambda\), and the other to construct \(\hat{\theta}\) using the chosen \(\hat{\lambda}\). This procedure is repeated by swapping the roles of each half and the two estimates are averaged to produce a final estimate.

We note that the cross-fit estimator uses only \(n/2\) samples per-fold, and thus the estimate of the covariance \(\ch(f, Y; \cD^{(k)}_{n/2})\) is computed using \(n/2\) labels.

4 Theoretical Analysis↩︎

To understand the relative performance of the classical and PPI++ estimators, we focus on an exact characterization of \(\mathbb{E}[(\hat{\theta} - \theta)^2]\), the finite-\(n\) estimation error. Since both the  and  estimators are unbiased, analyzing the finite-\(n\) estimation error is equivalent to the finite-\(n\) variance \(\mathrm{Var}(\thetah) \mathrel{\vcenter{:}}= \mathbb{E}[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2]\).

Our main result ([theorem:cross-ppi-general-mse]) makes no substantive assumptions regarding the distribution of \(Y\) and \(F\). That said, we warm-up in 4.1 by analyzing the special case where labels and pseudo-labels are Gaussian, and the analysis / results are far simpler. In this setting, both the  and  estimators are both unbiased, and their variances are comparable. The behavior in this setting is qualitatively similar to the more general results we derive later on.

In 4.2, we move on to analyze the variance of the  estimator for general distributions, where it remains unbiased. We decompose the variance into four parts: The variance of the classical estimator, a term that represents the gain in efficiency (i.e., a decrease in variance) attributable to pseudo-label quality, a term that represents the loss in efficiency due to errors in estimating the quality of the pseudo-labels and an additional term that captures the dependence between the two folds of the cross-fit estimator. The gain term depends directly on the covariance between the labels and the pseudo-labels while the loss term depends on the error in estimating this covariance from the labeled samples. The core of our no-free lunch result is that if this efficiency loss in estimating the covariance from labeled data exceeds the gain offered by the covariance itself, then will perform worse than .

A complete specification of the finite-sample variance of the   estimator requires an analysis of the efficiency loss term. We motivate this analysis with the illustrative case when the pseudo-labels are completely random. Here, the variance of   is always greater than the variance of the classical estimator, by a multiplicative factor of \(O(1/n)\). Then, in 4.2.2, we characterize the variance of the  estimator for any joint distribution of gold-standard and pseudo-labels. Finally, in 4.3, we return to the discussion of the  estimator, which is more difficult to analyze theoretically but has some unappealing properties from the perspective of confidence interval construction: It is biased in finite samples, and estimates of the variance that treat \(\lambda\) as fixed can lead to misleading optimism. We provide complete proofs for all theoretical results in 7.

4.1 Warm Up: Gaussian Labels↩︎

We begin with the special case where \(Y\) and \(F\) are normally distributed, for two reasons. First, this special case allows for a unified analysis of both the  and the   estimators, and establishes the existence of settings where both estimators under-perform . Second, this special case provides simple expressions for the variance of each estimator that capture the fundamental “no free lunch” nature of the problem, yielding simple conditions where PPI++ outperforms the classical estimate. In the Gaussian setting, both estimators are unbiased, and so we consider their variance as a measure of performance.³

[theorem:GaussianMSE] shows the exact variance of the PPI++ variants in the Gaussian setting. We see that the variance is reduced by a term that depends on the correlation between \(F\) and \(Y\). However, the variance is also inflated by a multiplicative factor in the first term, and hence we require the correlation to be greater than a certain level for the overall variance to be lower than that of the classical estimator. 1 makes it easy to arrive at this required level, which we present in the below corollary:

[theorem:jointly-normal-mse-condition-all] provides a unifying perspective on the “no free lunch” phenomenon: If the correlation \(\rho_{yf} \mathrel{\vcenter{:}}= \sigma_{fy} / (\sigma_y \sigma_f)\) is non-zero, then pseudo-labels can improve our variance, but there is a minimum sample size at which this improvement kicks in.⁴ However, if the correlation is zero, then both PPI++ estimators always perform worse than the classical estimator. This conclusion follows from a direct application of [theorem:GaussianMSE], where plugging a covariance of zero \((\sigma_{fy} = 0)\) into 1 yields a variance of PPI++ that is \((1 + 1 / (c \cdot n - 1))\) times the variance of the classical estimator. Intuitively, this additional variance arises from estimation of \(\lambda\). If the covariance is zero, then \(\lambda^* = 0\), and this choice of \(\lambda\) would yield the same variance as the classical estimator. In practice, however, \(\hat{\lambda}\) is not exactly zero, leading to higher variance. As we will see in [theorem:cross-ppi-indpt-mse], the same gap arises for  under general distributions, and we discuss the intuition further in 4.2.1.

For the remainder of the paper, we consider the setting where \(N\) is taken to be arbitrarily large. However, we note that in the Gaussian setting, it is possible to give exact expressions even when \(N\) is finite.

4.2 Finite-Sample Variance of Cross-fit PPI++↩︎

Before we present our main results, we briefly note the unbiasedness of .

This is a trivial result included for completeness, and for contrast with [prop:bias95of95single95sample], where we will characterize the non-zero bias of . The unbiasedness of  follows by noting that the estimate on each fold after sample-splitting is unbiased since \(\lh\) is fixed within a fold, and so the simple average is unbiased by linearity of expectations. We now proceed to focus on the finite sample variance of  for general distributions. We first present our main result.

The first term is the variance of the classical estimator. The second term (Eff. Gain) captures the gain in efficiency from using a predictor \(f\) that is correlated with the true label \(Y\), and the third term (Eff. Loss) captures the loss of efficiency that arises from the expected error in estimating the covariance from finite samples. In [theorem:gen-covariance95estimation] we will characterize the loss term in 2 , which can be combined with [theorem:cross-ppi-general-mse] to give the exact variance of the  estimator. For now, 2 yields intuition for when  fails to provide benefit.

In other words, there is no free lunch, but rather an intuitive trade-off: If the error in estimating the covariance \(\sigma_{fy}\) is higher than the magnitude of \(\sigma_{fy}\), then  performs worse than using the \(n\) labeled examples alone.

In the following, we characterize the covariance estimation error \(\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]\). To build intuition, we first revisit the setting where \(Y, F\) are independent, and prove that (in general), PPI++ performs worse than classical in this setting. In 4.2.2 we then give an explicit characterization (in [theorem:gen-covariance95estimation]) of the covariance estimation error term.

4.2.1 Warm-up: Performance Gap with Random Pseudo-Labels↩︎

We revisit the case where \(Y\) and \(F\) are independent, but now allow for non-Gaussian distributions.

Note that this non-Gaussian result is identical to the analogous result in the Gaussian case (obtained by plugging in \(\sigma_{yf} = 0\) to 1 , where \(c = 1/2\) for ). [theorem:cross-ppi-indpt-mse] can help build intuition that bridges the non-asymptotic and asymptotic regimes: when pseudo-labels are independent of the true labels, the finite-sample variance of the  estimator is always greater than the variance of the classical estimator, even if the asymptotic variance is identical.

4.2.2 Finite-Sample Covariance Estimation Error with General Pseudo-Labels↩︎

Before introducing our main result, we introduce additional notation. For two variables (e.g., \(F\) and \(Y\)) and two integers \(a, b\), we use \(\sigma_{f^a y^b}\) to refer to the covariance between the random variable \(F^a\) and the random variable \(Y^b\). For a single variable \(Y\), we use \(\sigma^2_{Y^a}\) to denote the variance of \(Y^a\).

First, we characterize the error in covariance estimation in [theorem:gen-covariance95estimation], which, when plugged into [theorem:cross-ppi-general-mse], yields an analytical expression for the variance of in terms of terms involving the mean and covariance of \(Y, F\) and their higher-order moments.

Note that while here the covariance estimate is formed from \(n\) IID samples, forms this estimate from only \(n/2\) samples assuming \(n\) total labels are available.

4.2.3 Analytical Expression for the Finite-Sample Variance↩︎

Combining [theorem:gen-covariance95estimation] with [theorem:cross-ppi-general-mse] provides an analytical expression for the exact variance of . While we do not write the entire expression here due to space constraints, we make the following observation: Of the four components of the variance in 2 , the first two are \(O(1/n)\), and the last two are \(O(1 / n^2)\), since the covariance estimation error in [theorem:gen-covariance95estimation] is \(O(1/n)\) \[\begin{align} \mathrm{Var}(\thetah_{\text{CF-PPI++}}) =& \frac{1}{n} \left(\sigma_{y}^2 - \frac{\sigma_{fy}^2}{\sigma_f^2}\right) \\ & + \frac{1}{n} \left(\frac{1}{\sigma_f^2} \overbrace{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}^{O(1/n) \text{ by \ref{theorem:gen-covariance95estimation}}}\right) \\ & + \frac{1}{n^2}\left(\frac{2}{\sigma_f^4}(\sigma_{yf^2} - 2\sigma_{fy}\mu_f)^2\right) \\ =& \frac{1}{n} \left(\sigma_{y}^2 - \frac{\sigma_{fy}^2}{\sigma_f^2}\right) + O\left(\frac{1}{n^2}\right) \end{align}\] The full expression for the variance, expanding the covariance estimation error term, is a simple corollary of [theorem:cross-ppi-general-mse] and [theorem:gen-covariance95estimation], and is provided in 2 in the Appendix. The decomposition above provides further intuition regarding the mismatch between asymptotic results, as established in e.g., [4], which only consider the \(O(1/n)\) term since the \(O(1/n^2)\) terms vanishes asymptotically, and our finite-sample results, which take the \(O(1/n^2)\) terms into account.

4.3 Bias and Over-Optimistic Variance Estimation of Single-Sample PPI++↩︎

We now turn to some less desirable properties of the  estimator. Practitioners conducting statistical inference are often concerned with the construction of confidence intervals. Typically, we construct asymptotically valid confidence intervals using the variance of a consistent estimator, and hence lower variance yields tighter intervals. While such intervals are generally not guaranteed to maintain nominal coverage in finite samples, introduces two additional challenges. First, while asymptotically consistent, it is biased in finite sample, and second, it can produce overly optimistic variance estimates. We first characterize the exact bias⁵ in [prop:bias95of95single95sample], and in [prop:optimistic95variance95single95sample95ppi4343] we illustrate that under mild conditions, standard methods for estimating the variance of  will necessarily yield estimates that are lower than the classical variance, even in scenarios (as illustrated in 4.1) where we know the MSE of  is higher than that of the classical estimator.

While in 4.2, we exactly characterized the variance of  , we omit a similar analysis of , since the latter case is far less tractable to analyze, owing to complex dependencies that arise from estimating \(\lambda\) on the same data used to form the final estimate. We focus instead on the typical approach to estimating the variance that practitioners use to derive confidence intervals, and give the following result on its overly optimistic nature.

In [prop:optimistic95variance95single95sample95ppi4343] we consider a variance estimation strategy that treats \(\lambda\) as fixed, and uses plug-in estimators to estimate each component of the variance (namely, \(\hat{\sigma}_y, \hat{\sigma}_f,\) and \(\hat{\sigma}_{yf}\)). In the appendix, we present a similar result for another common method of variance estimation, that similarly treats the estimated \(\lambda\) as fixed, and uses the empirical variance of \(y - \lambda f\) in \(\cD_n\) and \(\lambda f\) in \(\cD'_N\).

As a result of [prop:optimistic95variance95single95sample95ppi4343], the asymptotic confidence intervals constructed for  will always be narrower than those of , even if the true variance is larger. In 5, we show empirically that these always tighter intervals can result in significant drops in coverage.

5 Experiments↩︎

We now illustrate our theory with experiments on the Alphafold [29] dataset used in [4], using a similar experimental setup where we use bootstrapping to simulate multiple draws from a common distribution, and use the empirical mean of the (larger) original dataset set for establishing ground truth. Note that we focus on the low sample size regime where n \(<\) 50. See 8 for more details. See 9 for additional results on the galaxies [4] dataset.

In 2, we show the difference between MSE of PPI++ and classical as a function of sample size \(n\). We use \(\MSE\) to reflect the fact that while is unbiased,   is not. For \(\MSE\) difference, the region below \(0\) marks improvement. We consider pseudo-labels with various correlation levels (0.81 is the original set in Alphafold). The results are in line with our theoretical developments for , and illustrate that similar trends hold for . Using pseudo-labels with varying quality, the MSE either (a) starts off worse than classical, but starts to improve over classical past a certain ‘tipping point’ sample size, (b) starts off worse than classical, but closes the gap as dataset size increases, or (c) starts off better than classical and stays there. For all non-zero correlation sets, there exists some \(n\) after which PPI++ has improved \(\MSE\) over classical. We note that for the pseudo-label models shown here,  tends to achieve lower \(\MSE\) than .

In 3, we show the coverage (left) and average widths of intervals (right) constructed via and over \(n\). Here we present the results with the original pseudo-label set in Alphafold with correlation \(0.81\), while a further-noised set is presented in 10. We note that the method produces intervals that are narrower than those from the . However, these substantively tighter intervals come at the cost of inadequate coverage. For instance, for the original pseudo-labels (top), at \(n = 20\), the realized coverage with is 87%. In 10,  6, we see that both and show wider intervals as the correlation drops, and while widens enough to match the coverage of , still falls short in this regard.

6 Discussion↩︎

Having presented our main theoretical results, we now step back to summarize what these results mean for applied use of prediction-powered methods.

A Cautionary Note (No Free Lunch): The practical appeal of is clear: when pseudo-labels are informative, they can increase the effective sample size. Our work seeks to establish the same spirit for , guiding practitioners away from the pitfall of believing that pseudo-labels always help, even when they are close to uninformative.

An Optimistic Note: Our results show that the required dependence between pseudo-labels and true labels is not fixed, but depends on the number of labeled samples, and is not necessarily an onerous requirement: Taking the Gaussian case as a general rule-of-thumb, the correlation lower-bound of \((n/2 - 2)^{-1/2}\) for Cross-fit PPI++ may be reasonable to achieve for moderate \(n\). The key is that some assumption on pseudo-label performance is required for improvement, but such assumptions may be quite reasonable.

A Practical Note: Our work provides more than just a “no free lunch” result, or a “rule-of-thumb” for gauging the benefit of pseudo-labels. [theorem:cross-ppi-general-mse], combined with [theorem:gen-covariance95estimation], can be used to derive the exact relative variance of . While the full form depends on some higher-order covariances (e.g., between \(Y\) and \(F^2\)), the full expression can be computed analytically in many common settings, and used to construct visualizations like the one shown in 1 for the binary setting. Thus, if the practitioner is willing to posit a reasonable assumption on the quality of the pseudo-labels, and the number of labels they have, our framework can determine if there is an upside to use or and the magnitude of improvement.

Overall, our hope is that this work helps practitioners use techniques like PPI++ with confidence, making transparent the implicit assumptions required to see practical benefit.

Impact Statement↩︎

Given the availability and ease of use of LLMs and VLMs, it has never been easier to construct some predictor that appears to plausibly have "good enough" performance, even without using labeled data. Thus, pseudo-labels are already a routine and central part of the decision-making processes within many high-impact applications. For instance, LLM-as-judge evaluations in industry production pipelines [2], [30]–[32], LLMs as proxies for human responses in social science experimentation [33]–[35], the use of foundation models in randomized controlled trials [36], [37], clinical trials [8], PPI-based content moderation [38], [39], LLMs-as-peer-reviewers [40], [41], etc. While procedures like PPI++ are important tools, the promise of these methods to always turn these synthetic signals, however noisy they may be, into statistically valid inferences might tempt practitioners to throw in “the kitchen sink” of black-box pseudo labels, confident that it cannot hurt. Given the already widespread use of LLM-labels, such misplaced optimism could prove very costly. Our theoretical findings narrate a “no-free lunch” style cautionary tale: proof that there are practical settings where PPI++ can hurt and PPI++ confidence intervals fail to meet advertised rates of coverage. On the other hand, our theory provides a quantitative framework that converts the assumptions that a practitioner is willing to make about the quality of their pseudo-labels into a quantification of the magnitude of the upside, if any.

7 Proofs↩︎

In this appendix section, we include formal proofs for all the mathematical statements in the main paper. We structure the section as follows. After giving the full expression for the variance of Cross-Fit PPI++, we begin with an introduction of some useful lemmas on properties of moments, variances and covariances. Some lemmas are interesting in their own right while other lemmas are introduced solely to establish some results that recur in the workings of other proofs. We then consider the bias of and . We show that is unbiased while is biased and we derive this bias that we presented in [prop:bias95of95single95sample]. We then present a full proof of the finite sample variance of the and all the statements in 4.2 including conditions where has higher variance than 1. Following this, we construct the proof for the covariance estimation error presented in [theorem:gen-covariance95estimation]. With these proofs in place, we are able to apply them to the Gaussian setting to complete the proofs in 4.1. Finally, we provide proofs for the optimistic variance estimate of that we presented in [prop:optimistic95variance95single95sample95ppi4343].

7.1 Full Expression for the Variance of Cross-Fit PPI++↩︎

7.2 Foreword and Useful Lemmas↩︎

While the following result is of interest in it’s own right (and makes up substantially the entire proof of [prop:bias95of95single95sample]), we give it here as a Lemma, as we also make use of it in the proof of [theorem:cross-ppi-general-mse].

A well-known result in statistics is that for Gaussian data, the sample mean and the sample covariance matrix are independent. Here we give the formal statement and include the proof.

7.3 is unbiased↩︎

While fairly simple to observe, we provide a proof for the unbiasedness of the estimator. This proof is intended to provide an intuition for the next section where we consider the bias of . We include this unbiasedness as a proposition

The key to the unbiasedness of is the independence of \(\lh\) from the data used to compute sample means \(\bar{f}_n\). Since this independence is not applicable to , we cannot factor \(\lh\) into its own expectation leading to a finite-sample bias as we shall prove next.

7.4 Proof of [prop:bias95of95single95sample]: Bias of↩︎

7.5 [theorem:cross-ppi-general-mse]: Finite-Sample Variance of Estimator↩︎

7.5.2 Variance of a single-fold estimate \(\hat{\theta}_1\)↩︎

Having expressed the variance in terms of the variance of a single-fold estimate, we now characterize the variance of this single-fold estimate.

Consider the estimate on a single-fold from 5, from which we have it that the variance can be decomposed as follows, (using \(E[\thetah_k] = \theta\))

\[\label{eq:ppi-split-mse} (\thetah_{k} - \theta) = \underbrace{(\bar{y}_{n/2} - \theta)}_{A} + \lh \underbrace{\left(\muf - \frac{1}{n/2} \sum_{i \in \cD'_{n/2}} f_i \right)}_{D} = A + \lh D\tag{7}\] Such that the variance is given by \[\begin{align} \mathrm{Var}(\thetah_{k}) & = \mathbb{E}[(\thetah_k - \theta)^2] \\ & = \mathbb{E}[(A + \lh D)^2] & \text{by \ref{eq:ppi-split-mse}} \\ & = \mathbb{E}[A^2 + 2A \lh D + {\lh}^{2} D^2] \\ & = \mathbb{E}[A^2] + 2\mathbb{E}[A \lh D] + \mathbb{E}[{\lh}^{2} D^2] & \text{by Linearity of Expectation} \end{align}\] We can observe that \[\mathbb{E}[A^2] = \mathbb{E}[(\bar{y}_{n/2} - \theta)^2] = \frac{\sigma_y^2}{n/2} = \frac{2\sigma_y^2}{n}\]

Consider the second term: \[\begin{align} 2\mathbb{E}[A \lh D] = 2\mathbb{E}[\lh (\bar{y}_{n/2} - \theta)(\mu_f - \bar{f}_{n/2})] \end{align}\] Recall that in each split-estimate, \(\lh\) is estimated on an independent split and is therefore independent of the other terms in the expectation. This allows us to apply [lemma:gen-cross-expect] to obtain \[\begin{align} 2\mathbb{E}[A \lh D] = 2\mathbb{E}[\lh (\bar{y}_{n/2} - \theta)(\mu_f - \bar{f}_{n/2})] = -2\frac{\sigma_{fy}^2}{(n/2)\sigma_f^2} = -4\frac{\sigma_{fy}^2}{n\sigma_f^2} \end{align}\] where we have accordingly applied \(n/2\) where the lemma uses \(n\). It is also useful to note that while \(\lh\) depends on the covariance estimate which also depends on the number of samples it is computed on, the expectation of the covariance estimate is independent of \(n\) and is just \(\sigma_{fy}\). Since only the expectation of \(\lh\) appears in the lemma, there is no need to adjust additionally for this.

Now consider the third term: Since \(\lh\) is independent of \(D\), we have it that \[\begin{align} \mathbb{E}[{\lh}^{2} D^2] & = \mathbb{E}[\lh^2]\mathbb{E}[(\mu_f - \bar{f}_{n/2})^2] \\ & = \frac{\mathbb{E}[\hat{\sigma}_{fy}^2]}{\sigma_f^4} \cdot \frac{\sigma_{f}^2}{n/2} & \lh & \mathrel{\vcenter{:}}= \frac{\hat{\sigma}_{fy}}{\varf} \\ & = \frac{2\mathbb{E}[\hat{\sigma}_{fy}^2]}{n \sigma_f^2} \end{align}\] where we use the definition of \(\hat{\lambda}\) and the fact that \(\mathbb{E}[(\bar{f} - \mu_f)^2] = \sigma_f^2 / (n/2)\) by definition of \(\bar{f}\). Putting these together, we get \[\begin{align} \mathrm{Var}(\thetah_k) & = \frac{2\sigma_y^2}{n} -4\frac{\sigma_{fy}^2}{n\sigma_f^2} + \frac{2\mathbb{E}[\hat{\sigma}_{fy}^2]}{n \sigma_f^2} \\ & = \frac{2\sigma_y^2}{n} -\frac{2\sigma_{fy}^2}{n\sigma_f^2} + \frac{2\mathbb{E}[\hat{\sigma}_{fy}^2]}{n\sigma_f^2} - \frac{2\sigma_{fy}^2}{n\sigma_f^2} \\ & = \frac{2\sigma_y^2}{n} -\frac{2\sigma_{fy}^2}{n\sigma_f^2} + \frac{2\mathbb{E}[\hat{\sigma}_{fy}^2 - \sigma_{fy}^2]}{n\sigma_f^2} \\ & = \frac{2\sigma_y^2}{n} -\frac{2\sigma_{fy}^2}{n\sigma_f^2} + \frac{2\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} \end{align}\] where the last line follows from the fact that \(\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2] = \mathbb{E}[\hat{\sigma}_{fy}^2 - 2\hat{\sigma}_{fy} \sigma_{fy} + \sigma_{fy}^2] = \mathbb{E}[\hat{\sigma}_{fy}^2 - \sigma_{fy}^2]\), from the fact that \(\mathbb{E}[\hat{\sigma}_{fy}] = \sigma_{fy}\).

7.5.3 Finite-Sample Variance of↩︎

Having characterized the variance of a single-fold estimate, we can plug this into the overall variance of the to complete the proof as follows:

\[\begin{align} \mathrm{Var}(\thetah_{\text{CF-PPI++}}) = \mathbb{E}\left[\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2} - \theta\right)^2\right] & = \frac{1}{2} \left(\mathrm{Var}(\hat{\theta}_1) + \frac{4}{n^2 \sigma_{f}^4} \left(\sigma_{yf^2} - 2 \sigma_{yf} \muf \right)^2\right) \end{align}\]

\[\begin{align} \mathrm{Var}(\thetah_{\text{CF-PPI++}}) & = \frac{1}{2} \left( \frac{2\sigma_y^2}{n} -\frac{2\sigma_{fy}^2}{n\sigma_f^2} + \frac{2\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} + \frac{4}{n^2 \sigma_{f}^4} \left(\sigma_{yf^2} - 2 \sigma_{yf} \muf \right)^2\right) \\ & = \frac{\sigma_y^2}{n} - \frac{\sigma_{fy}^2}{n\sigma_f^2} + \frac{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} + \frac{2}{n^2 \sigma_{f}^4} \left(\sigma_{yf^2} - 2 \sigma_{yf} \muf \right)^2 \end{align}\]

7.6 Proof of : Condition for Worse Performance of↩︎

With the above theorem, it is easy to prove 1 which we restate here.

From [theorem:cross-ppi-general-mse], we have \[\begin{align} \mathrm{Var}(\thetah_{\text{CF-PPI++}}) = \frac{\sigma_y^2}{n} - \frac{\sigma_{fy}^2}{n\sigma_f^2} + \frac{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} + \frac{2}{n^2 \sigma_{f}^4} \left(\sigma_{yf^2} - 2 \sigma_{yf} \muf \right)^2 \end{align}\]

Further, since \(\frac{2}{n^2 \sigma_{f}^4} \left(\sigma_{yf^2} - 2 \sigma_{yf} \muf \right)^2 \geq 0\), we have

\[\begin{align} \mathrm{Var}(\thetah_{\text{CF-PPI++}}) - \frac{\sigma_y^2}{n} \geq \frac{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} - \frac{\sigma_{fy}^2}{n\sigma_f^2} \end{align}\]

Therefore if \(\frac{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} > \frac{\sigma_{fy}^2}{n\sigma_f^2}\), we get \(\mathrm{Var}(\thetah_{\text{CF-PPI++}}) > \frac{\sigma_y^2}{n}\)

7.7 [theorem:cross-ppi-indpt-mse]: Variance of with Independent Pseudo-Labels↩︎

Motivated by the desire to build intuition, we first consider the case when \(Y,F\) are independent, and present a proof in that scenario. Using [lemma:indp-covariance95estimation] in combination with [theorem:cross-ppi-general-mse] allows us to prove [theorem:cross-ppi-indpt-mse].

With [lemma:indp-covariance95estimation] in hand, we now prove [theorem:cross-ppi-indpt-mse]. Note that we could have also written this proof using the more general result of [theorem:gen-covariance95estimation] instead of [lemma:indp-covariance95estimation].

7.8 [theorem:cross-ppi-general-mse]: Covariance Estimation Error General↩︎

7.9 4.1 Proofs for the Gaussian Setting↩︎

We begin with some notes on special properties of Gaussians. We then prove the results in 4.1 for and then follow it up with the proofs for the portion of the statements.

For jointly normal random variables \(X\) and \(Y\) with means \(\mu_x\), \(\mu_y\), variances \(\sigma_x^2\), \(\sigma_y^2\), and covariance \(\sigma_{xy}\), we have the following identities: \[\begin{align} \mathbb{E}[X^2Y^2] & = \mu_x^2\mu_y^2 + \sigma_x^2\mu_y^2 + \sigma_y^2\mu_x^2 + \sigma_x^2\sigma_y^2 + 2\sigma_{xy}^2 + 4\sigma_{xy}\mu_x\mu_y \\ \mathbb{E}[X^2Y] & = \mu_x^2\mu_y + \sigma_x^2\mu_y + 2\sigma_{xy}\mu_x \\ \mathbb{E}[XY^2] & = \mu_x\mu_y^2 + \sigma_y^2\mu_x + 2\sigma_{xy}\mu_y \end{align}\] Now consider the following expression for the expected value of the squared sample covariance obtained from [theorem:gen-covariance95estimation]: \[\begin{align} \mathbb{E}\left[\hat{\sigma}_{fy}^2\right] & = \frac{1}{n}\sigma_{f^2y^2} + \frac{1}{n-1}\sigma_f^2\sigma_y^2 + \left[\frac{n-1}{n} + \frac{1}{n(n-1)}\right]\sigma_{fy}^2 \\ & + \frac{1}{n}[-2\sigma_{y^2f}\mu_f - 2\sigma_{f^2y}\mu_y + 4\sigma_{fy}\mu_f\mu_y] \end{align}\] Using these identities, we can simplify the covariance terms in our expression: \[\begin{align} \sigma_{f^2y^2} & = \mathbb{E}[F^2Y^2] - \mathbb{E}[F^2]\mathbb{E}[Y^2] \\ & = (\mu_f^2\mu_y^2 + \sigma_f^2\mu_y^2 + \sigma_y^2\mu_f^2 + \sigma_f^2\sigma_y^2 + 2\sigma_{fy}^2 + 4\sigma_{fy}\mu_f\mu_y) \\ & \quad - (\mu_f^2 + \sigma_f^2)(\mu_y^2 + \sigma_y^2) \\ & = \mu_f^2\mu_y^2 + \sigma_f^2\mu_y^2 + \sigma_y^2\mu_f^2 + \sigma_f^2\sigma_y^2 + 2\sigma_{fy}^2 + 4\sigma_{fy}\mu_f\mu_y \\ & \quad - \mu_f^2\mu_y^2 - \mu_f^2\sigma_y^2 - \sigma_f^2\mu_y^2 - \sigma_f^2\sigma_y^2 \\ & = 2\sigma_{fy}^2 + 4\sigma_{fy}\mu_f\mu_y \end{align}\] Similarly for the other covariance terms: \[\begin{align} \label{eq:special-gaussian} \sigma_{f^2y} & = \mathbb{E}[F^2Y] - \mathbb{E}[F^2]\mathbb{E}[Y] \\ & = (\mu_f^2\mu_y + \sigma_f^2\mu_y + 2\sigma_{fy}\mu_f) - (\mu_f^2 + \sigma_f^2)\mu_y \\ & = \mu_f^2\mu_y + \sigma_f^2\mu_y + 2\sigma_{fy}\mu_f - \mu_f^2\mu_y - \sigma_f^2\mu_y \\ & = 2\sigma_{fy}\mu_f \end{align}\tag{16}\] and \[\begin{align} \sigma_{y^2f} & = \mathbb{E}[Y^2F] - \mathbb{E}[Y^2]\mathbb{E}[F] \\ & = (\mu_y^2\mu_f + \sigma_y^2\mu_f + 2\sigma_{fy}\mu_y) - (\mu_y^2 + \sigma_y^2)\mu_f \\ & = \mu_y^2\mu_f + \sigma_y^2\mu_f + 2\sigma_{fy}\mu_y - \mu_y^2\mu_f - \sigma_y^2\mu_f \\ & = 2\sigma_{fy}\mu_y \end{align}\] Now we substitute these expressions into our original equation: \[\begin{align} \mathbb{E}[\hat{\sigma}_{fy}^2] & = \frac{1}{n}(2\sigma_{fy}^2 + 4\sigma_{fy}\mu_f\mu_y) + \frac{1}{n-1}\sigma_f^2\sigma_y^2 \\ & \quad + \left[\frac{n-1}{n} + \frac{1}{n(n-1)}\right]\sigma_{fy}^2 \\ & \quad + \frac{1}{n}[-2(2\sigma_{fy}\mu_y)\mu_f - 2(2\sigma_{fy}\mu_f)\mu_y + 4\sigma_{fy}\mu_f\mu_y] \end{align}\] Simplifying the terms with \(\mu_f\) and \(\mu_y\): \[\begin{align} \frac{1}{n}(4\sigma_{fy}\mu_f\mu_y - 4\sigma_{fy}\mu_y\mu_f - 4\sigma_{fy}\mu_f\mu_y + 4\sigma_{fy}\mu_f\mu_y) = 0 \end{align}\] So these terms cancel out completely. Now we simplify the terms with \(\sigma_{fy}^2\): \[\begin{align} \frac{2\sigma_{fy}^2}{n} + \left[\frac{n-1}{n} + \frac{1}{n(n-1)}\right]\sigma_{fy}^2 & = \sigma_{fy}^2\left[\frac{2}{n} + \frac{n-1}{n} + \frac{1}{n(n-1)}\right] \\ & = \sigma_{fy}^2\left[\frac{n+1}{n} + \frac{1}{n(n-1)}\right] \\ & = \sigma_{fy}^2\left[\frac{(n+1)(n-1) + 1}{n(n-1)}\right] \\ & = \sigma_{fy}^2\left[\frac{n^2-1+1}{n(n-1)}\right] \\ & = \sigma_{fy}^2\left[\frac{n^2}{n(n-1)}\right] \\ & = \sigma_{fy}^2\left[\frac{n}{n-1}\right] \end{align}\] Therefore, for jointly normal random variables \(F\) and \(Y\), the Covariance Estimation Error becomes: \[\begin{align} \mathbb{E}[\hat{\sigma}_{fy}^2] - \sigma_{fy}^2 = \frac{n}{n-1}\sigma_{fy}^2 + \frac{1}{n-1}\sigma_f^2\sigma_y^2 - \sigma_{fy}^2 = \frac{1}{n-1}(\sigma_{fy}^2 + \sigma_f^2\sigma_y^2) \end{align}\] or \[\begin{align} \label{eq:gauss-cov-err} \mathbb{E}[(\hat{\sigma}_{fy}-\sigma_{fy})^2] = \frac{n}{n-1}\sigma_{fy}^2 + \frac{1}{n-1}\sigma_f^2\sigma_y^2 - \sigma_{fy}^2 = \frac{1}{n-1}(\sigma_{fy}^2 + \sigma_f^2\sigma_y^2) \end{align}\tag{17}\]

We are now ready to prove the results in 4.1. We begin with a proof for the Cross-fit PPI++ portion of the results. Then we treat the Single-sample case.

7.9.1 Proofs for the estimator, Gaussian↩︎

We now include an additional lemma for use,

That the variance between the folds is zero has the following consequences for

1. 1 becomes a necessary condition and not just a sufficient condition. It was previously sufficient since the variance between the folds added an additional positive term to the variance of the estimator. This term is now zero making the loss \(>\) gain for worse performance from a necessary condition.

2. This is now a reversible condition. That is, if \(\mathbb{E}[(\hat{\sigma}_{fy}-\sigma_{fy})^2] < \sigma_{fy}^2\) then performs better than 1.

We now prove the condition for improvement using in the Gaussian case.

Proof of [theorem:jointly-normal-mse-condition-all]: Condition for Performant Cross-fit PPI++, Gaussian

Proof of [theorem:GaussianMSE]: Variance of Cross-fit PPI++, Gaussian

Our covariance estimation error can be directly used in [theorem:cross-ppi-general-mse] to obtain the variance in the Gaussian case.

Finally, the variance when \(F, Y\) are independent Gaussians follows trivially from the above as follows.

Those complete the proofs for in the Gaussian case. We now turn to the proof of portion of the statements from 4.1.

7.9.2 Proofs for the estimator, Gaussian↩︎

Given that we don’t have a generalized expression for the variance of the , our proof involves deriving this variance from first principles. The reason we are able to do so in this Gaussian setting lies in [lemma:gaussian-indpt]. The independence of the sample covariance and the sample means results in \(\lh\) being independent of \((\mu_f - \bar{f}_n)\) which dramatically simplifies the analysis. The rest of the proof mechanics is quite similar to our proof of [theorem:cross-ppi-general-mse] in 7.5. Then we apply the conditions for improvement using this derived variance.

Consider the variance of the estimator: We have it that the errors in our estimator can be decomposed as follows \[(\thetahp - \theta) = \underbrace{(\bar{y}_{n} - \theta)}_{A} + \lh \underbrace{\left(\muf - \frac{1}{n} \sum_{i \in \cD_n} f_i \right)}_{D} = A + \lh D\] Such that the mean squared error is given by \[\begin{align} \MSE(\thetahp) & = \mathbb{E}[(\thetahp - \theta)^2] \\ & = \mathbb{E}[(A + \lh D)^2] \\ & = \mathbb{E}[A^2 + 2A \lh D + {\lh}^{2} D^2] \\ & = \mathbb{E}[A^2] + 2\mathbb{E}[A \lh D] + \mathbb{E}[{\lh}^{2} D^2] & \text{by Linearity of Expectation} \end{align}\] We can observe that \[\mathbb{E}[A^2] = \mathbb{E}[(\bar{y}_{n} - \theta)^2] = \frac{\sigma_y^2}{n}\] is \(\mathrm{Var}(\thetahc)\) the classical variance. Now we consider the second term: \[\begin{align} 2\mathbb{E}[A \lh D] = 2\mathbb{E}[\lh (\bar{y}_{n} - \theta)(\mu_f - \bar{f}_{n})] \end{align}\] From [lemma:gaussian-indpt] we have that since \(Y,F\) are gaussian, \(\lh\) is independent of \(\bar{y}_{n}, \bar{f}_{n}\). Further, since \(\theta, \mu_f\) are non-random, we have that \(\lh\) is also independent of them. Overall, this gets us that \(\lh\) is independent of \((\bar{y}_{n} - \theta)(\mu_f - \bar{f}_{n})\). This allows us to apply [lemma:gen-cross-expect] with \(n\) samples to obtain \[\begin{align} 2\mathbb{E}[A \lh D] = 2\mathbb{E}[\lh (\bar{y}_{n} - \theta)(\mu_f - \bar{f}_{n})] = -2\frac{\sigma_{fy}^2}{n\sigma_f^2} = -\frac{2\sigma_{fy}^2}{n\sigma_f^2} \end{align}\] Now consider the third term: \(\mathbb{E}[{\lh}^{2} D^2]\), once again using [lemma:gaussian-indpt], \(\lh\) is independent of \(D\), we get \(\mathbb{E}[{\lh}^{2} D^2] = \mathbb{E}[\lh^2]\mathbb{E}[(\mu_f - \bar{f}_{n})^2]\) \[\begin{align} \lh = \frac{\hat{\sigma}_{fy}}{\varf} \end{align}\] where \(\hat{\sigma}_{fy}\) is an unbiased estimate of the covariance \(\sigma_{fy}\). Therefore, \[\begin{align} \mathbb{E}[\lh^2] = \frac{\mathbb{E}[\hat{\sigma}_{fy}^2]}{\sigma_f^4} \end{align}\] Further, consider, \(\mathbb{E}[(\bar{f}_{n} - \mu_f)^2]\). Since \(\mu_f\) is the expected value of \(\bar{f}_{n}\), this term reduces to the variance of \(\bar{f}_{n}\) which is \(\frac{\sigma_f^2}{n}\). Putting these together, we get \[\begin{align} \MSE(\thetahp) & = \frac{\sigma_y^2}{n} - \frac{2\sigma_{fy}^2}{n\sigma_f^2} + \frac{\mathbb{E}[\hat{\sigma}_{fy}^2]}{\sigma_f^4}\cdot\frac{\sigma_f^2}{n} \nonumber \\ & = \frac{\sigma_y^2}{n} -\frac{\sigma_{fy}^2}{n\sigma_f^2} + \frac{\mathbb{E}[\hat{\sigma}_{fy}^2]}{n\sigma_f^2} - \frac{\sigma_{fy}^2}{n\sigma_f^2} \nonumber \\ & = \MSE(\thetahc) -\frac{\sigma_{fy}^2}{n\sigma_f^2} + \frac{\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]}{n\sigma_f^2} \label{eq:mse95gaussian95single95sample95interm} \end{align}\tag{20}\]

Therefore, the variance of in the Gaussian case takes exactly the same form as . However, one important difference is that uses all \(n\) available samples for estimating the covariance thus reducing \(\mathbb{E}[(\hat{\sigma}_{fy} - \sigma_{fy})^2]\).

With this variance derived, we can easily prove the required theorems by simply combining it with the covariance estimation error that we derived.

Proof of [theorem:GaussianMSE]: Variance of , Gaussian

Further, setting \(\sigma_{fy}\) to \(0\), we obtain \[\begin{align} \MSE(\thetahp) = \frac{\sigma_y^2}{n}\left(1 + \frac{1}{cn-1}\right) \end{align}\] which completes the proof for the estimator.

Finally, we prove the condition for improvement,

Proof of [theorem:jointly-normal-mse-condition-all], Condition for variance improvement estimator

7.9.3 Proof of the Finite-N Case↩︎

We now consider the setting where \(N\) is finite.

7.10 Variance Estimates of↩︎

We now consider the variance estimate of the 4 estimator in two methods. The first method is based on the idea of \(\lambda\) as a minimum variance linear combination, the spirit of PPI++. Here, we compute the theoretical variance of the PPI++ estimator assuming \(\lambda\) is independent of the data, and then use all the required plug-ins, including \(\lh\). In the second method, we naively compute the direct empirical plug-in in the estimator.

7.10.1 [prop:optimistic95variance95single95sample95ppi4343]: Data Independent Plug-in↩︎

7.10.2 Naive Plug-in↩︎

We now consider an alternate method for computing the empirical variance. Starting with the definition, assuming again that \(\lambda\) is fixed, we rewrite it as a sum of two independent terms.

\[\thetahp = (\bar{y} - \lambda\bar{f}_n) + \lambda\bar{f}_N\]

Now, consider the following standard method for estimating variance, by taking the empirical variance of \(\bar{y} - \lambda \bar{f}_n\) and the empirical variance of \(\lambda \bar{f}_N\), and adding up the empirical variance estimates, adjusted for sample size. The estimate of the variance of the former is given by \[\begin{align} \hat{\sigma}_{y - \lambda f}^2 & = \frac{1}{n-1} \sum_{i} ((y_i - \lambda f_i) - (\bar{y} - \lambda \bar{f}))^2 \\ & = \frac{1}{n-1} \sum_{i} ((y_i - \bar{y}) - (\lambda f_i - \lambda \bar{f}))^2 \\ & = \frac{1}{n-1} \sum_{i} (y_i - \bar{y})^2 - 2 \lambda (y_i - \bar{y})(f_i - \bar{f}) + \lambda^2 (f_i - \bar{f})^2 \\ & = \hat{\sigma}_y^2 + \lambda^2 \hat{\sigma}_f^2[n] - 2 \lambda \hat{\sigma}_{yf} \end{align}\] where \(\hat{\sigma}_y^2, \hat{\sigma}_{f}^2[n], \hat{\sigma}_{yf}\) are the empirical variances of \(Y\) and \(F\) and the empirical covariance respectively, where we note that \(\hat{\sigma}_f^2[n]\) is the empirical variance on \(n\) samples. Using the fact that \(\lambda \mathrel{\vcenter{:}}= \hat{\sigma}_{yf} / ((1 + n/N) \hat{\sigma}_f^2[N])\), we have it that \[\begin{align} \hat{\sigma}_{y - \lambda f}^2 & = \hat{\sigma}_y^2 + \lambda^2 \hat{\sigma}_f^2[n] - 2 \lambda \hat{\sigma}_{yf} \\ & = \hat{\sigma}_y^2 + \left(\frac{\hat{\sigma}_{yf}^2}{(1 + \frac{n}{N})^2 \hat{\sigma}_f^4[N]}\right) \hat{\sigma}_f^2[n] - 2 \left(\frac{\hat{\sigma}_{yf}}{(1 + \frac{n}{N}) \hat{\sigma}_f^2[N]}\right) \hat{\sigma}_{yf} \\ & = \hat{\sigma}_y^2 + \left(\frac{\hat{\sigma}_{yf}^2}{(1 + \frac{n}{N})^2 \hat{\sigma}_f^4[N]}\right)\hat{\sigma}_f^2[n] - 2 \left(\frac{\hat{\sigma}^2_{yf}}{(1 + \frac{n}{N}) \hat{\sigma}_f^2[N]}\right) \\ & = \hat{\sigma}_y^2 + \left(\frac{\hat{\sigma}_{yf}^2}{(1 + \frac{n}{N})^2 \hat{\sigma}_f^4[N]}\right)\hat{\sigma}_f^2[n] - 2 \hat{\sigma}_f^2[N] \left(1 - \frac{n}{N}\right) \left(\frac{\hat{\sigma}^2_{yf}}{(1 + \frac{n}{N})^2 \hat{\sigma}_f^4[N]}\right) \\ & = \hat{\sigma}_y^2 - \left(2 \hat{\sigma}_f^2[N] \left(1 - \frac{n}{N}\right) - \hat{\sigma}_f^2[n]\right)\left(\frac{\hat{\sigma}_{yf}^2}{(1 + \frac{n}{N})^2 \hat{\sigma}_f^4[N]}\right) \\ & = \hat{\sigma}_y^2 - \left(2 \hat{\sigma}_f^2[N] \left(1 - \frac{n}{N}\right) - \hat{\sigma}_f^2[n]\right) \hat{\lambda}^2 \end{align}\] Then the empirical variance of \(\hat{\sigma}_{\lambda f}^2\), computed on \(N\) samples, is given by \(\lambda^2 \hat{\sigma}_{f}^2[N]\), so that the total variance (dividing by \(n\) and \(N\) respectively, and then taking the sum), is given by \[\begin{align} \frac{\hat{\sigma}_{y - \lambda f}^2}{n} + \frac{\hat{\sigma}_{\lambda f}^2}{N} & = \frac{\hat{\sigma}_y^2}{n} - \frac{\left(2 \hat{\sigma}_f^2[N] \left(1 - \frac{n}{N}\right)\right) \hat{\lambda}^2}{n} + \frac{\hat{\lambda}^2\hat{\sigma}_f^2[n]}{n} + \frac{\hat{\lambda}^2 \hat{\sigma}_f^2[N]}{N} \end{align}\]

If \(N \to \infty\), then \(\frac{\hat{\sigma}_{\lambda f}^2}{N} \to 0\), \(1 - \frac{n}{N} \to 1\), \(\frac{\hat{\lambda}^2 \hat{\sigma}_f^2[N]}{N} \to 0\), and \(\sigma_f^2[N] \to \sigma_f^2\), and hence the empirical variance estimate becomes, \[\frac{\hat{\sigma}_y^2}{n} - \frac{2\hat{\lambda}^2\sigma_f^2}{n} + \frac{\hat{\lambda}^2\sigma_f^2[n]}{n}\] This is always less than the empirical estimate of the classical variance as long as the empirical variance of \(F\) on \(n\) samples is no greater than double the true variance.

8 Experiment Setup Details↩︎

In this section, we provide details of the experiment setup on Alphafold [29]. For both the \(\MSE\) as well as the coverage experiments, our results are bootstrapped over 50,000 draws. To closely match the theoretical intuition, we aim to study the setting where \(n\) is small and \(N\) is large. Therefore, after randomly sampling \(n\) (or \(2n\), for sample-splitting), we treat all the remaining data as unlabeled. Since the true labels are binary, we also use a binary noise model for the configurations that do not use the original predictor. This noise model is implemented by specifying \(\mathbb{P}(F = 1| Y = 1)\) and \(\mathbb{P}(F = 1 | Y = 0)\). Alternatively, this can be viewed as specifying the True Positive Rate and the False Positive Rate of the pseudo-labels. From a noisy label lens, this can be viewed as specifying the complete noise transition matrix. For each configuration, for each \(n\), we reset random states, ensuring full reproducibility. For all plug-in statistical quantities computed, we use ddof = 1 for unbiasedness.

9 Additional Experiments: Galaxies Dataset↩︎

In this section, we extend our experiments to the galaxies dataset used in [4]. Similar insights hold as in 5. Our experiment apparatus is the same as described in 8. We run \(50,000\) draws using small-regime of \(n\) or \(2n\) and all the remaining data as unlabeled. We use a similar binary noise model. We show our results in 4 5.

10 Additional Experiments: Coverage and Interval Width, Alphafold Noised↩︎

In this section, we present the coverage and interval width results on Alphafold, similar to 3. We note that in this lowered correlation setting, while and widen their interval width to improve coverage, matches the coverage of but still falls short. The experiment details are as presented in 5 and 8.

References↩︎

---

1. As discussed in [theorem:jointly-normal-mse-condition-all], this threshold is for an implementation without sample splitting. In the Gaussian setting with sample-splitting and cross-fitting, the threshold is slightly higher, at \((n/2 - 2)^{-1/2}\).↩︎

2. We use \(\theta\) here to align with notation in the PPI literature, but note that \(\theta\) and \(\mu_y\) are the same quantity.↩︎

3. The setting of Gaussian labels is unique: The empirical covariance of normally distributed random variables is independent of their sample means. This property reduces the dependence between the estimate of \(\lambda\) (which depends on the estimated covariance) and the estimated value itself. As a result,  is also unbiased in this setting. However, in the non-Gaussian case,  is biased, while the use of sample-splitting retains the unbiasedness of , as we will see in 4.3↩︎

4. Notably, the condition for  uses \(n/2\), the size of a single fold, while the condition for  uses \(n\), which illustrates a cost of sample splitting in the Gaussian setting.↩︎

5. It is generally known that  is biased in finite samples [27], [28], but we do not know of other results giving the exact form, so we produce it here. While [27] show that the bias of a similar estimator is \(O(1/n)\), they do not give the exact form.↩︎