1 Introduction↩︎

Knowledge distillation (KD) has become a fundamental technique in modern machine learning [1]–[3]. In its original formulation, KD transfers knowledge from a high-capacity teacher to a more compact student and allows the student to achieve comparable performance with fewer parameters. This paradigm has since been extended to self-distillation, in which a model is retrained using its own predictions while retaining the same architecture [4], [5]. Extensive empirical evidence suggests that self-distillation can improve generalization [6]–[9].

Self-distillation (SD) is typically studied when the student is retrained on the teacher’s original training data. In many practical deployments, however, those labeled data are no longer available: one can query a trained predictor but cannot access its training data or parameters. We instead assume access to a stream of fresh unlabeled covariates, which may be out-of-distribution (OOD) relative to the teacher’s training covariates. We refer to this regime as the prediction-only setting. It arises naturally in post-deployment adaptation and model reuse, particularly when the original training data cannot be retained or shared because of privacy, safety, or storage constraints [10], [11]. This setting raises a basic question:

A natural approach is to train a student on fresh unlabeled covariates using the teacher’s predictions as pseudo-labels. Because no ground-truth labels are used, this procedure yields a pure-distilled (PD) student [12]. Pure-distillation has been studied in several settings, including high-dimensional ridgeless regression [13], ridge regression [14], random-feature ridge regression [15], and Gaussian mixture models [16]–[18]. Perhaps surprisingly, the PD student can outperform its teacher. Such gains are not guaranteed, however, and the PD student can instead perform substantially worse; see, for example, 2.

Recent work on same-\(X\) ridge SD establishes strict-improvement guarantees [19]–[22]. In ridge regression, the optimally mixed student lies on the affine path between the teacher and the PD student; this observation suggests directly combining their predictions as the final output. We call this affine combination predictor the prediction-mixed self-distillation (PMSD) student. Its mixing weight is real-valued and need not lie in \([0,1]\), so the combination need not be convex. The method is illustrated in 1.

This approach is operationally attractive: it requires no access to the original labeled data and reduces adaptation to a one-dimensional post hoc aggregation problem, with no additional model fitting. Establishing an analogous strict-improvement guarantee in the fresh-\(X\) setting is theoretically nontrivial. The same-\(X\) result relies on geometric identities induced by the shared design, and these identities disappear when the student is trained on a fresh design independent of the teacher’s design. Before turning to theory, 2 provides an empirical preview of the fresh-\(X\) setting on the UCI Blog Feedback dataset. In both the in-distribution and out-of-distribution settings, the optimally mixed PMSD student outperforms the teacher and the PD student.

This empirical preview motivates three theoretical and practical questions:

1. Under what conditions can the PMSD student improve upon the teacher and the PD student?

2. How does its performance depend on the amount of fresh unlabeled data?

3. Can the optimal mixing weight be tuned when the teacher’s training data are unavailable?

Our main contributions, which also provide an outline of the paper, are as follows:

• High-dimensional characterization for ridge regression (3). We first derive exact oracle identities for the optimal PMSD mixing weight ([prop:oracle95affine]). Under proportional asymptotics, we then characterize the PMSD risk and optimal mixing weight under general anisotropic covariance and deterministic signal (1).

• Strict improvement and non-monotonicity (4). Using these deterministic equivalents, we show that the PMSD student strictly improves upon the teacher for almost every pair of regularization levels \((\lambda_t, \lambda_s)\) used by the teacher and the PD student ([prop:teacher_bad_set,prop:strict_improv_ood]). Thus, strict improvement is generic even when \(\lambda_t\) is unknown or \(\lambda_s\) is not optimally tuned, provided that the mixing weight is chosen optimally.

  In the same-\(\lambda\) isotropic setting, we also highlight a phenomenon absent from same-\(X\) self-distillation: at fixed regularization, the PMSD risk is generically non-monotone in the unlabeled aspect ratio, so more unlabeled data need not improve performance (4.3).

• Consistent data-dependent tuning (5). We show that the teacher’s outputs and fresh unlabeled covariates alone do not identify the oracle mixing weight. We then prove that a small independent labeled calibration sample enables consistent estimation in a single post-training step, without additional model fitting (2).

• Gains in logistic regression (6). Finally, we extend the analysis beyond regression under squared loss by studying binary logistic regression. We show that prediction mixing can strictly outperform both the teacher and the PD student ([thm:class_mixing,thm:class_mixing_2,thm:class_mixing_large_p]).

2 Related work↩︎

Same-\(X\) self-distillation. A recent line of work studies self-distillation (SD) in regression settings where the original labeled data are available and the student is retrained on the same design. [23] interpret repeated SD as a regularization mechanism in Hilbert space. [19] and [20] analyze one-step and repeated SD for linear models under fixed-design assumptions. Under random design, [21] establish pointwise strict improvement guarantees, proportional asymptotic results, and a tuning method for ridge regression. [22] further show that, for spiked covariance matrices with \(s\) spikes, \(s\) rounds of self-distillation attain optimal performance among spectral shrinkage estimators. Their results motivate using a different regularization level in each round of self-distillation. Our paper studies a different regime: the prediction-only fresh-\(X\) setting, where the original labeled sample is unavailable and the same-design geometric identities no longer apply. We instead ask whether prediction mixing can improve the teacher when its training data and regularization level \(\lambda_t\) are unknown and the student penalty \(\lambda_s > 0\) is arbitrary. Such pointwise guarantees are important because finding the optimal ridge penalty can be challenging. For example, [24] show that the leave-one-out cross-validation loss is generally neither convex nor quasi-convex as a function of the ridge penalty \(\lambda\).

Fresh-\(X\) pseudo-labeling and pure-distillation. Empirical work shows that training on unlabeled data using pseudo-labels generated by the teacher can improve performance [25], [26]. On the theoretical side, pure-distillation on fresh unlabeled covariates has been studied in high-dimensional ridgeless regression [13] and ridge regression [14]. [15] analyze scaling laws for the teacher and the PD student in random feature ridge regression. These works often consider a student that is stronger than the teacher, for example because it is more overparameterized or has a better regularization mechanism; this regime is known as “weak-to-strong” distillation [27]. Although the PD student can outperform the teacher, such gains are not guaranteed. For example, in high-dimensional ridge regression, [14] show that when the teacher is over-regularized, the PD student cannot improve upon it and may perform worse; see 2 for this phenomenon. These works focus primarily on the benefits of pure-distillation in teacher–student transfer, whereas we ask whether combining predictions from the teacher and the PD student can guarantee an improvement over the teacher. We also study how the PMSD risk depends on the fresh unlabeled sample size and how to calibrate the optimal mixing weight post hoc.

Theory of self-distillation in classification. A substantial literature studies the benefits of retraining on the teacher’s pseudo-labels in classification. This includes linear probing on frozen feature backbones [19], [28] and logistic regression on Gaussian mixture data [16]–[18], [29], [30]. In the Gaussian mixture setting, [30] show that retraining with the teacher’s hard labels can improve accuracy under certain conditions on the label noise rate and sample size. In a different direction, [16] show that training a student with the teacher’s hard labels on unlabeled data can improve robustness to adversarial examples. For binary and multiclass classification with frozen feature backbones (i.e., linear probing), [19], [28] show that the pure-distilled student, trained only on the teacher’s soft labels, can tolerate higher label noise rates and improve accuracy over the teacher. Our logistic regression analysis and linear probing experiments (see 6) are complementary: we study a prediction mixing mechanism that can improve upon both the teacher and the pure-distilled student.

3 Ridge regression↩︎

In this section, we introduce prediction-mixed distillation for ridge regression, derive its structural risk decomposition, and obtain deterministic equivalents under proportional asymptotics.

3.1 Setup and preliminaries↩︎

Let \(f_{\lambda_t}: \RR^p \to\R\) be the teacher ridge predictor trained on an unknown labeled dataset \(\cD_{\rm lab} := (X, y) \in \R^{n_t \times p} \times \R^{n_t}\). Here, \(X \in \mathbb{R}^{n_t \times p}\) is the design matrix with i.i.d. \(p\)-dimensional observations \(x_i \in \mathbb{R}^p\) as its rows, and \(y \in \mathbb{R}^{n_t}\) is the vector of corresponding scalar ground-truth labels \(y_i\), for \(i = 1, \ldots, n_t\). Denote the regularization parameter used by the teacher by \(\lambda_t > 0\) (potentially depending on \(\cD_{\rm lab}\)). That is, the teacher ridge predictor is the random function \[\label{eq:teacher} x \in \RR^p \mapsto f_{\lambda_t}(x) := x^\top \argmin_{\beta \in \RR^p}\big\{ \| y - X \beta \|_2^2 / n_t + \lambda_t \| \beta \|_2^2\big\}.\tag{1}\] In the prediction-only regime considered here, we can query the teacher at any input but need not know the explicit form of \(f_{\lambda_t}\) or its regularization parameter \(\lambda_t\). Thus, we observe only the teacher’s predictions at the queried points.

Given a fresh sample of unlabeled covariates \(\cD_{\rm unlab} := \tilde{X} \in \RR^{n_s \times p}\) consisting of \(n_s\) i.i.d.observations that may be out-of-distribution (OOD) relative to the teacher’s training distribution, we query the teacher on this unlabeled set and denote its predictions by \(\hat{y}_{\lambda_t} := f_{\lambda_t}(\tilde{X}) \in \RR^{n_s}\). The pure-distilled (PD) student \(f_{\pd, \lambda_s}\) is the ridge predictor trained on \((\tilde{X}, \hat{y}_{\lambda_t})\) with penalty \(\lambda_s > 0\)¹, \[\label{eq:pd95student} x \in \RR^p \mapsto f_{\pd, \lambda_s}(x) := x^\top \argmin_{\beta \in \RR^p}\big\{ \| \hat{y}_{\lambda_t} - \tilde{X} \beta \|_2^2 / n_s + \lambda_s \| \beta \|_2^2\big\}.\tag{2}\]

A natural approach to improving the student’s predictive performance is to combine the teacher’s queried predictions with the student’s own PD predictions. We focus on affine mixing strategies, in which the student deploys an affine combination of the available predictions. Specifically, for a mixing parameter \(\xi \in \RR\), the prediction-mixed self-distillation (PMSD) student predictor is given by \[\label{eq:affine95path} x \in \RR^p \mapsto f_{\mathsf{pmsd}, \lambda_t, \lambda_s, \xi}(x):=(1-\xi)f^{\te}_{\lambda_t}(x)+\xi f_{\pd, \lambda_s}(x).\tag{3}\] We consider affine rather than convex combinations because negative weights or weights exceeding one may be preferable. Indeed, empirical and theoretical work shows that the optimal mixing weight lies outside the unit interval in many settings [19]–[21].

Let \((x_0,y_0)\) denote an independent test point drawn from the teacher’s training distribution, corresponding to the out-of-sample in-distribution setting. For any predictor \(f\), define the conditional squared prediction risk \[\label{eq:freshX95pred95risk} R(f):=\E\big[(y_0-f(x_0))^2\mid \cD_{\rm lab},\cD_{\rm unlab}\big].\tag{4}\] For convenience, denote \(R_{\te}(\lambda_t):=R(f_{\lambda_t}^{\te})\), \(R_{\pd}(\lambda_s):=R(f_{\pd, \lambda_s})\), and \(R_{\mathsf{pmsd}}(\lambda_t, \lambda_s, \xi):=R(f_{\mathsf{pmsd}, \lambda_t, \lambda_s, \xi})\). When the regularization parameters are clear from context, we use the shorthand \(R_{\te}\), \(R_{\pd}\), and \(R_{\mathsf{pmsd}}\).

3.2 Optimal prediction-mixed risk decomposition↩︎

We first decompose the optimally mixed PMSD risk in terms of three key quantities: the teacher risk \(R_{\te}(\lambda_t)\), the PD student risk \(R_{\pd}(\lambda_s)\), and the teacher-student residual correlation \[C(\lambda_t, \lambda_s):=\E\!\bigl[(y_0-f^{\te}_{\lambda_t}(x_0))(y_0-f_{\pd, \lambda_s}(x_0))\mid \cD_{\rm lab},\cD_{\rm unlab}\bigr].\] Let \(\xi^\star(\lambda_t, \lambda_s) \in \argmin_{\xi \in \RR} R_{\mathsf{pmsd}}(\lambda_t, \lambda_s,\xi)\) denote the optimal mixing weight, and let \(R_{\mathsf{pmsd}}^\star(\lambda_t, \lambda_s) := R_{\mathsf{pmsd}}(\lambda_t, \lambda_s, \xi^{\star})\) denote the corresponding optimal PMSD risk.

The decomposition implies that \(R^{\star}_{\mathsf{pmsd}} \leq \min\{R_{\te}, R_{\pd}\}\). The gain over the teacher is determined by the magnitude of \(R_{\te}(\lambda_t)-C(\lambda_t, \lambda_s)\), while the sign of this difference determines the sign of \(\xi^\star\), which may therefore be negative. The denominator is the mean squared difference between the teacher and the PD student: \[D(\lambda_t, \lambda_s) = R_{\te}(\lambda_t) + R_{\pd}(\lambda_s) - 2 C(\lambda_t, \lambda_s) = \EE\big[(f_{\lambda_t}(x_0)-f_{\pd,\lambda_s}(x_0))^2\mid \cD_{\rm lab},\cD_{\rm unlab} \big] \geq 0. \nonumber\] It is zero if and only if \(f_{\lambda_t}\) and \(f_{\pd,\lambda_s}\) coincide almost surely on the test distribution, in which case \(R_{\mathsf{pmsd}} \equiv R_{\te} \equiv R_{\pd}\) for all \(\xi\), so any \(\xi\) is trivially optimal. Hence the assumption \(D > 0\) is mild.

3.3 Proportional asymptotics under general anisotropic covariance↩︎

In this subsection, we derive deterministic equivalents for the optimal mixing weight and the conditional PMSD risk in the proportional asymptotic regime, where \(n_t, n_s, p \to \infty\) at the same rate.

Throughout, we assume that the teacher’s training data \(\{(x_i,y_i)\}_{i=1}^{n_t}\) are i.i.d.samples from a distribution satisfying the standard assumptions used in the random matrix analysis of high-dimensional regression [31], [32]. We further assume that the fresh unlabeled covariates \(\{\tilde{x}_i\}_{i=1}^{n_s}\) are i.i.d.samples whose population covariance matrix is simultaneously diagonalizable with the population covariance matrix of the teacher’s covariates \(\{x_i\}_{i=1}^{n_t}\); equivalently, the two covariance matrices commute. This assumption of simultaneous diagonalizability across covariate sources has also been considered in the transfer learning literature [33], [34].

Key quantities and notation. We define the population projection and variance parameters as \(\beta:=\Sigma_t^{-1}\E[xy]\) and \(\sigma^2:=\Var(y-x^\top\beta)\), respectively. The signal energy is \(r^2 := \| \beta \|_2^2\) and \(\SNR:=r^2/\sigma^2\). Denote the labeled and unlabeled data aspect ratios \(\gamma_t := p/n_t\) and \(\gamma_s := p/n_s\), respectively. We introduce several scalar quantities to characterize the limiting risk. Let \(\otr(A):=\tr(A)/p\) denote the normalized trace. For any fixed \(\lambda_t > 0\) and \(\lambda_s > 0\), define \(\kappa_t = \kappa_t(\lambda_t) > 0\) and \(\kappa_s = \kappa_s(\lambda_s) > 0\) as the unique solutions to the corresponding fixed-point equations, \[\label{eq:kappa95fp95mainpaper} \kappa_t = \lambda_t + \gamma_t \kappa_t \,\otr\!\big(\Sigma_t (\Sigma_t +\kappa_t I_p)^{-1}\big) \quad \text{and} \quad \kappa_s = \lambda_s + \gamma_s \kappa_s \,\otr\!\big(\Sigma_s (\Sigma_s +\kappa_s I_p)^{-1}\big),\tag{6}\] respectively, and let \(G_t = (\Sigma_t + \kappa_t I_p)^{-1}\) and \(G_s = (\Sigma_s + \kappa_s I_p)^{-1}\) be the associated resolvents. For fixed \((\Sigma_t, \gamma_t)\) and \((\Sigma_s, \gamma_s)\), the maps \(\lambda_t \mapsto \kappa_t\) and \(\lambda_s \mapsto \kappa_s\) are both one-to-one on \((0,\infty)\). Finally, for \(i, j \geq 0\), define the mixed signal-covariance alignment and trace functionals as \[q_{i,j}:=\beta^\top G_t^i G_s^j \Sigma_t \,\beta, \qquad \tilde{q}_{i,j}:=\beta^\top G_t^i G_s^j \Sigma_s \,\beta, \qquad t_{i,j}:=\gamma_t \,\otr(\Sigma_t^2 G_t^i G_s^j),\] and set \(b_t := (1 - t_{2,0})^{-1}\) and \(b_s := \big(\gamma_s \otr(\Sigma_s \Sigma_t G_s^2) \big) / \big(1 - \gamma_s \otr(\Sigma_s^2 G_s^2) \big).\)

Our first result states that the three oracle ingredients in [prop:oracle95affine] have deterministic equivalents, in the sense defined in 10.5, under general anisotropic covariance and deterministic signal. The exact closed forms are more algebraically involved than those for same-\(X\) ridge distillation in [21] because they involve two coupled resolvents, but remain explicit in terms of finitely many traces of functions of \((G_t, G_s, \Sigma_t, \Sigma_s)\) and finitely many quadratic forms \(\beta^\top f(\Sigma_t, \Sigma_s)\beta\).

As shown in 3 (a), the theoretical predictions closely align with the empirical risks even at moderate values of \(n_t\), \(n_s\), and \(p\).

4 Strict improvement and non-monotonicity↩︎

Using the risk characterization from 3, we now study two qualitative questions: when the PMSD student strictly improves upon the teacher and the pure-distilled student, and how its optimal risk depends on the amount of fresh unlabeled data.

4.1 Strict improvement over the teacher↩︎

Leveraging the deterministic equivalents from 1, we study when the PMSD student strictly improves upon the teacher. Equation ?? shows that the PMSD risk is asymptotically strictly less than the teacher risk whenever \[\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s) \neq 0,\] and the two risks coincide otherwise. Thus, characterizing when the PMSD student fails to improve upon the teacher amounts to understanding when \(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s)\) vanishes.

Since \(\Sigma_t\) and \(\Sigma_s\) are simultaneously diagonalizable, let \(\Sigma_t = U^{\top} \operatorname{diag}(\sigma_1, \ldots, \sigma_p) U\) and \(\Sigma_s = U^{\top} \operatorname{diag}(\tilde{\sigma}_1, \ldots, \tilde{\sigma}_p) U\) be their eigendecompositions. Define the squared signal-covariance alignment by \(v_i := ((U\beta)_i)^2\) for \(i \in [p]\). Let \(m\) and \(\tilde{m}\) (\(1 \le m, \tilde{m} \le p\)) denote the number of distinct eigenvalues of \(\Sigma_t\) and \(\Sigma_s\), respectively. Write \(\{\tilde{\sigma}_{(1)}, \ldots, \tilde{\sigma}_{(\tilde{m})}\}\) for the corresponding set of distinct eigenvalues of \(\Sigma_s\), with \(\tilde{\sigma}_{(1)} \leq \cdots \leq \tilde{\sigma}_{(\tilde{m})}\). The term \(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s)\) can be rewritten as follows (see 11.2): \[\begin{align} \label{eq:R-C} \mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t,\lambda_s) &= \sum_{i=1}^{p} \frac{c_i}{\tilde{\sigma}_i+\kappa_s} = \sum_{j=1}^{\tilde{m}} \frac{d_j}{\tilde{\sigma}_{(j)}+\kappa_s}, \end{align}\tag{7}\] where \[\begin{align} c_i &:= \frac{\kappa_t \sigma_i^2}{(\sigma_i+\kappa_t)^2} \biggl( \frac{\kappa_t^2 b_t q_{2,0}+\sigma^2 b_t}{\kappa_t n_t} - v_i \biggr), \qquad i=1,\ldots,p, \tag{8} \\ d_j &:= \sum_{k:\,\tilde{\sigma}_k=\tilde{\sigma}_{(j)}} c_k, \qquad j=1,\ldots,\tilde{m}. \tag{9} \end{align}\] For a fixed and unknown \(\lambda_t > 0\) used by the teacher, we are interested in the choices of \(\lambda_s > 0\) for which the PMSD student strictly improves upon the teacher. Hence, we study the cardinality of the set of \(\lambda_s\) for which there is no strict improvement at a given \(\lambda_t\), which we call the “student-tie” set: \[\Lambda_{s}(\lambda_t) := \{ \lambda_s > 0: \mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s) = 0\}.\]

Fix \(\lambda_t\) and suppose that \(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s)\) is not identically zero as a function of \(\lambda_s\). Setting the right-hand side of 7 to zero and clearing the denominators gives \[\sum_{j=1}^{\tilde{m}} \, d_j \prod_{k \neq j} (\tilde{\sigma}_{(k)} + \kappa_s) = 0,\] which is a polynomial equation in \(\kappa_s\) of degree at most \(\tilde{m}-1\). The one-to-one correspondence between \(\lambda_s\) and \(\kappa_s\) in equation 6 implies that there are at most \(\tilde{m}-1\) values of \(\lambda_s\) for which \(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s) = 0\). Therefore, \(|\Lambda_{s}(\lambda_t)| \leq \tilde{m} - 1\) in this case.

On the other hand, if, for a given \(\lambda_t\), the gap \(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s)\) is identically zero as a function of \(\lambda_s\), then the PMSD student cannot improve upon the teacher for any choice of \(\lambda_s\). Equivalently, \(\Lambda_{s}(\lambda_t)=\RR^+\). We call this the “teacher-degeneracy” set: \[\Lambda_{t} := \{ \lambda_t > 0: \mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s) = 0 \text{ for all } \lambda_s>0\}.\]

Generally, this degeneracy condition requires \(\lambda_t\) to satisfy a specific system of \(\tilde{m}\) equations determined jointly by the covariance structure of \(\Sigma_t\), the signal-covariance alignment, and the noise variance \(\sigma^2\). The bound on \(|\Lambda_t|\) is crude, as it uses only the weaker necessary condition \[\sum_{j=1}^{\tilde{m}} d_j = 0,\] which holds whenever \(\lambda_t\in\Lambda_t\). When \(\lambda_t \notin \Lambda_{t}\), we further bound the size of the “student-tie” set \(\Lambda_{s}(\lambda_t)\) as follows.

Thus, under the mild nondegeneracy condition \(\lambda_t \notin \Lambda_t\), the deterministic equivalent of the optimal PMSD risk is strictly smaller than that of the teacher for almost every \(\lambda_s\). Equivalently, the two risks can tie only at finitely many values of \(\lambda_s\) for each fixed \(p\). When \(\Sigma_s\) has \(p\) distinct eigenvalues, \[d_j = c_j = \frac{\kappa_t \sigma_j^2}{(\sigma_j + \kappa_t)^2} \biggl( \frac{\kappa_t^2 b_t q_{2,0} + \sigma^2 b_t}{\kappa_t n_t} - v_j \biggr), \quad j = 1, \ldots, p,\] Thus, many sign changes in the sequence \(\{ d_1, \ldots, d_p \}\) can occur only if the sequence \(\{ v_1, \ldots, v_p\}\) oscillates repeatedly across the exact threshold \(\frac{\kappa_t^2 b_t q_{2,0} + \sigma^2 b_t}{\kappa_t n_t}\), which is typically a restrictive configuration. For common covariance-signal models, we obtain the following stronger guarantees.

4.1.1 In-distribution fresh unlabeled covariates↩︎

First, consider fresh unlabeled covariates drawn from the teacher’s training distribution, so that \(\Sigma_s = \Sigma_t\).

For example, in the isotropic design setting, the “teacher-degeneracy” set is the singleton \(\Lambda_t = \{\lambda_t^{\star}\}\), where \(\lambda_t^{\star}=\gamma_t\sigma^2/r^2\) is the ridge-optimal value. Consequently, the PMSD student does not strictly improve upon the optimally tuned teacher, but it does improve for every other \(\lambda_t>0\) and any choice of \(\lambda_s>0\). 2 (a) illustrates this property on the Blog Feedback dataset, using fresh unlabeled covariates taken from the same data pool as the teacher’s data (but with different samples).

As another consequence of 1, when the fresh sample size matches the teacher’s sample size and the same regularization is used, fresh-\(X\) prediction mixing recovers the tangent identity that drives strict improvement in same-\(X\), same-\(\lambda\) ridge self-distillation in [21].

The deterministic limiting risks and the sign rule for the mixing weight in this same-\(\lambda\) balanced regime are illustrated in 3 (b): (i) the PMSD and teacher risks coincide only at the teacher’s stationary point, and (ii) the sign of the optimal mixing weight is opposite to the sign of the derivative of the teacher risk curve.

4.1.2 Out-of-distribution fresh unlabeled covariates↩︎

In many settings, we may not know \(\Sigma_t\) or have access to fresh covariates from the same distribution as the teacher’s training data. A simple and practical alternative is to sample the fresh unlabeled covariates from an isotropic distribution, which commutes with any \(\Sigma_t\). In this case, we obtain the following strict improvement guarantee.

To see this, setting \(\Sigma_s = I_p\) in equation 7 gives \[\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s) = \frac{\sum_{i=1}^{p} c_i}{1 + \kappa_s}.\]

Thus, if \(d_1 = \sum_{i=1}^{p} c_i \neq 0\), strict improvement holds for every \(\lambda_s > 0\). If \(d_1 = 0\), the two risks tie for all \(\lambda_s\). For example, 1 shows that when \(\Sigma_t = I_p\), this degeneracy occurs only at the ridge-optimal value \(\lambda_t^{\star}\). More generally, the “teacher-degeneracy” condition can be rewritten as \[\begin{align} \label{eq:ood95degeneracy} \sum_{i=1}^{p} \frac{ \kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^2} \biggl(\frac{\kappa_t^2 b_t q_{2,0} + \sigma^2 b_t}{\kappa_t n_t} - v_i \biggr) = 0, \end{align}\tag{10}\]

which is a polynomial in \(\kappa_t\) with degree at most \(4m-1\) (see 11.1).

We now compare the “teacher-degeneracy” condition with the stationary points of the teacher risk function to see whether isotropic fresh covariates can improve upon a tuned teacher. By equation 26 , the stationary condition \(\sR'(\lambda_t)=0\) is equivalent to \[\begin{align} \label{eq:ood95R95prime} \sum_{i=1}^{p} \frac{\kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^3} \biggl(\frac{\kappa_t^2 b_t q_{2,0} + \sigma^2 b_t}{\kappa_t n_t} - v_i \biggr) = 0. \end{align}\tag{11}\]

Therefore, strict improvement over a teacher at its stationary point fails if and only if both equations 10 and 11 hold. Equivalently, \[\begin{align} \ddfrac{\sum_{i=1}^p \frac{\kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^2} v_i}{\sum_{i=1}^p \frac{\kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^2}} = \ddfrac{\sum_{i=1}^p \frac{\kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^3} v_i}{\sum_{i=1}^p \frac{\kappa_t \sigma_i^2}{(\sigma_i + \kappa_t)^3}} = \frac{\kappa_t^2 b_t q_{2,0} + \sigma^2 b_t}{\kappa_t n_t}. \end{align}\] In general, this imposes a strong alignment condition: two different weighted averages of the squared signal-alignment values \(\{v_i\}_{i=1}^p\) must equal the same threshold. Thus, we expect this simultaneous equality to be non-generic unless the covariance or signal alignment has special structure.

We illustrate the benefit of the PMSD student using isotropic fresh unlabeled covariates on real-world regression tasks in 4. Additional experiments appear in 16.1.

4.2 Strict improvement over the pure-distilled student↩︎

To compare the PMSD student with the pure-distilled student, we use a decomposition analogous to equation ?? : \[\begin{align} \sR^{\star}_{\mathsf{pmsd}}(\lambda_t, \lambda_s) = \mathcal{R}_{\pd}(\lambda_s) - \frac{\bigl(\mathcal{R}_{\pd}(\lambda_s)-\mathcal{C}(\lambda_t, \lambda_s)\bigr)^2}{\mathcal{R}_{\te}(\lambda_t)+\mathcal{R}_{\pd}(\lambda_s) - 2 \mathcal{C}(\lambda_t, \lambda_s)}. \end{align}\] Thus, the PMSD student strictly improves upon the PD student whenever \[\mathcal{R}_{\pd}(\lambda_s)-\mathcal{C}(\lambda_t, \lambda_s) \neq 0.\]

Since this condition is quite involved, we specialize to the isotropic design setting, where it admits a clean characterization.

Consequently, in the isotropic setting, the PMSD student strictly improves upon the PD student for all but at most two choices of \(\lambda_s > 0\).

4.3 Non-monotonicity in the amount of unlabeled data↩︎

A natural question is whether increasing the amount of fresh unlabeled data always improves the PMSD student. In the proportional regime, this amounts to fixing the labeled aspect ratio \(\gamma_t\) and studying the optimal PMSD risk as a function of \(\gamma_s\), where smaller \(\gamma_s\) corresponds to more unlabeled data. We specialize to the same-\(\lambda\) isotropic setting for this analysis. Surprisingly, for a fixed \(\lambda_t = \lambda_s = \lambda\), the optimal PMSD risk is not globally monotone in \(\gamma_s\). Non-monotone risk profiles as functions of aspect ratio are well documented in high-dimensional prediction, and subsampling- and aggregation-based procedures can sometimes restore monotonicity [35], [36]. The phenomenon here is different: the labeled training sample and teacher remain fixed, while only the amount of fresh unlabeled data varies. Even so, the optimal PMSD risk has a simple one-dimensional structure and is unimodal in \(\gamma_s\). See 15.1 for the formal result and 5 for illustrations from synthetic and real data experiments.

5 Data-dependent tuning↩︎

The results in 4 demonstrate the benefits of fresh-\(X\) distillation with optimal prediction mixing \(\xi^{\star}\). However, the deterministic equivalents in 1 depend on population quantities that are unknown in practice, such as the covariance spectrum and signal-covariance alignment. In this section, we study how to tune the mixing weight \(\xi\). We show that fresh unlabeled samples alone are insufficient, whereas an independent labeled calibration set facilitates consistent tuning.

Recall that the optimal oracle mixing weight from [prop:oracle95affine] depends on residual correlations with the unknown response. Thus, in the fully prediction-only regime, unlabeled covariates and teacher outputs alone are insufficient to identify \(\xi^\star\). This is an information-theoretic limitation, not a computational one. To see this, fix the observed teacher predictor \(f^{\te}\) and the induced fresh-\(X\) PD student \(f_{\pd}\). These predictors are observable in the prediction-only setting, while the response law is not. Consider two response models with the same covariate distribution: \(y=f^{\te}(x)+\varepsilon\) and \(y=f_{\pd}(x)+\varepsilon\), where \(\varepsilon\) is mean-zero noise independent of \(x\). Their oracle mixing weights are \(\xi^\star=0\) and \(\xi^\star=1\), respectively. Therefore, no estimator based only on unlabeled samples and teacher queries can be uniformly consistent for \(\xi^\star\). We give a formal non-identifiability statement in 15.2.

However, \(\xi^\star\) can be consistently estimated using an independent labeled calibration set \(\cD_{\rm cal}:=\{(x_i^{\rm cal},y_i^{\rm cal})\}_{i=1}^{n_{\rm cal}}\), which is used only for evaluation and not retraining. Conditional on the teacher predictor and fresh unlabeled data, both \(f^{\te}\) and \(f_{\pd}\) are fixed predictors. For an independent test point \((x,y)\) drawn from the target distribution, define the residuals \(e^{\te}:=y-f^{\te}(x)\) and \(e^{\pd}:=y-f_{\pd}(x)\). Define the calibration residuals \[\hat{e}_i^{\te}:=y_i^{\rm cal}-f^{\te}(x_i^{\rm cal}) \quad \text{and} \quad \hat{e}_i^{\pd}:=y_i^{\rm cal}-f_{\pd}(x_i^{\rm cal}),\] and estimate the three oracle quantities by \[\widehat R_{\te}^{\rm cal}:=\frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}(\hat{e}_i^{\te})^2, \qquad \widehat R_{\pd}^{\rm cal}:=\frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}(\hat{e}_i^{\pd})^2, \qquad \widehat C^{\rm cal}:=\frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}\hat{e}_i^{\te}\hat{e}_i^{\pd}.\] Define the one-shot plug-in calibration estimators of \(\xi^\star\) and \(R_{\mathsf{pmsd}}^\star\) by \[\label{eq:cal95xi} \widehat\xi^\star_{\rm cal} := \frac{\widehat R_{\te}^{\rm cal}-\widehat C^{\rm cal}}{\widehat R_{\te}^{\rm cal}+\widehat R_{\pd}^{\rm cal}-2\widehat C^{\rm cal}}, \quad \text{and} \quad \widehat R_{\mathsf{pmsd},{\rm cal}}^\star := \widehat R_{\te}^{\rm cal} - \frac{(\widehat R_{\te}^{\rm cal}-\widehat C^{\rm cal})^2}{\widehat R_{\te}^{\rm cal}+\widehat R_{\pd}^{\rm cal}-2\widehat C^{\rm cal}}.\tag{12}\] The next result establishes their consistency.

The assumptions required for consistency are minimal. Because the calibration set estimates only the risk and residual correlation of fixed predictors, no asymptotic condition involving \(p/n_{\rm cal}\) is required. For consistency, \(n_{\rm cal}\) need only diverge, regardless of the growth rate of \(p\). Computationally, tuning \(\xi\) is inexpensive: after training the PD student once, 12 requires only evaluating the two predictors on the calibration points and computing three empirical averages. The uniform finite-candidate statement also allows the same calibration set to choose among finitely many student penalties \(\lambda_s\) or unlabeled sample sizes \(n_s\). 6 illustrates calibration-based tuning with small labeled samples on real datasets. Reducing the label cost of this calibration step through alternative sampling or model-assisted evaluation schemes is a natural direction for future work [37].

6 Logistic regression↩︎

We now turn to the fresh-\(X\) prediction-only setting for logistic regression with cross-entropy loss. Our goal is to show that the PMSD student can strictly outperform both the teacher and the PD student.

Setting. A theoretical analysis of self-distillation for classification is more challenging than for regression. To streamline the presentation, we adopt the setting of [19], [28], which we review here. We study a binary classification problem in which each instance \(x \in \cX\) has a ground-truth label \(y(x) \in \{0,1\}\). Let \(\cP\) denote the underlying distribution on \(\cX\). We have access to an unlabeled dataset \(\cD_{\rm unlab} = \{x_i\}_{i=1}^{2n}\) consisting of \(2n\) i.i.d.draws from \(\cP\). The corresponding labels \(y_i:=y(x_i)\) are deterministic functions of the instances but are unobserved. We assume that the dataset is balanced and index the observations so that \(y_i = 1\) for \(i \in \mathcal{S}_1 := \{1, \ldots, n\}\) and \(y_i = 0\) for \(i \in \mathcal{S}_0 := \{n+1, \ldots, 2n\}\).

Teacher. We have access to a teacher classifier that returns a hard label \(\hat{y}(x) \in \{0,1\}\) for any \(x\in\cX\). Although its form is unknown, we can query its predictions. Suppose that its population classification error is the same within each class and equals an unknown \(\rho\in(0,1)\): \[\PP_{x\sim\cP}\bigl(\hat{y}(x)\neq y(x)\mid y(x)=0\bigr) = \PP_{x\sim\cP}\bigl(\hat{y}(x)\neq y(x)\mid y(x)=1\bigr) =\rho.\] Following [19], let \(\hat{n}:=\lfloor n\rho\rfloor\) and assume that the teacher makes \(\hat{n}\) errors in each class. After reindexing the observations within each class, write the teacher’s labels on \(\cD_{\rm unlab}\) as \[\hat{y}_i := \hat{y}(x_i) = \begin{cases} 1 - y_i & \text{for } i \in \underbrace{\{1, \ldots, \hat{n} \}}_{\mathcal{S}_{1,\text{bad}}} \, \cup \, \underbrace{\{n+1,\ldots,n+\hat{n}\}}_{\mathcal{S}_{0,\text{bad}}}, \\ y_i & \text{for } i \in \underbrace{\{\hat{n}+1,\ldots,n\}}_{\mathcal{S}_{1,\text{good}}} \, \cup \, \underbrace{\{n+\hat{n}+1,\ldots,2n\}}_{\mathcal{S}_{0,\text{good}}}. \end{cases}\]

Thus, \(|\mathcal{S}_{1,\text{bad}}|=|\mathcal{S}_{0,\text{bad}}|=\hat{n}\), and the fraction of incorrect labels in each class converges to \(\rho\) as \(n\to\infty\).

Students. We assume a feature map \(\phi:\cX\to\cH\), where \(\cH\) is a Hilbert space, and keep \(\phi\) fixed throughout. We train a linear classifier in \(\cH\) on the teacher’s hard pseudo-labels \(\{\hat{y}_i\}_{i=1}^{2n}\) using \(\ell_2\)-regularized logistic loss: \[\begin{align} \label{eq:class95training95loss} \theta_{\pd} := \argmin_{\theta} \frac{1}{2n} \sum_{i=1}^{2n} \text{CE}\big(\hat{y}_i, \sigma( \langle \theta, \phi(x_i) \rangle) \big) + \frac{\lambda}{2} \| \theta \|^2, \end{align}\tag{13}\] where \(\text{CE}(q, \hat{q}) := -q\log(\hat{q})-(1-q)\log(1-\hat{q})\) is the binary cross-entropy loss and \(\sigma(z):=(1+e^{-z})^{-1}\) is the sigmoid function. The resulting PD student’s soft prediction is \(y_{\pd}(x) := \sigma(\langle \theta_{\pd}, \phi(x) \rangle) \in (0,1)\). This setting corresponds to linear probing in neural networks, where a learnable linear layer and sigmoid activation are placed on top of a frozen feature backbone. Given a mixing weight \(\xi \in \RR\), the PMSD classifier is \[\label{eq:logistic95affine} y_{\mathsf{pmsd}, \xi}(x)=\II \bigl\{(1-\xi)\hat{y}(x)+\xi y_{\pd}(x)\ge 0.5 \bigr\}.\tag{14}\]

We make the following assumption on the feature map \(\phi\) and the unlabeled dataset \(\cD_{\rm unlab}\).

This assumption, formulated by [19], is technically convenient because it makes the data separable with respect to their unobserved ground-truth classes, thereby ensuring the existence of a perfect classifier. Moreover, [28] empirically validate this assumption on six real datasets: CIFAR-100, Caltech-101, Caltech-256, Flowers-102, Food-101, and Stanford Cars. They use features from a ResNet-34 pretrained on ImageNet; see their Figure 1 and Appendix D.2.

Goal. Given a soft prediction rule \(y_{\theta}:\cX\to[0,1]\), an instance \(x\) is classified correctly when \(y(x)=\II\{y_{\theta}(x)\geq 0.5\}\). By construction, the teacher’s accuracy on \(\cD_{\rm unlab}\) converges to \(100(1-\rho)\%\). We show in 3 that, when \(\rho\) is sufficiently close to \(0.5\) from below, the PD student trained by 13 also achieves only \(100(1-\rho)\%\) accuracy as \(n\to\infty\). Our goal is to construct a PMSD classifier as in 14 that achieves asymptotic \(100\%\) accuracy and thus perfectly recovers the true labels in the limit.

Prior work [19], [28] characterizes conditions on \(\rho\) and \(\lambda\) under which the PD student \(y_{\pd}\) itself achieves this goal. These works refer to the result as “\(100\%\) population accuracy” because the generalization gap vanishes for finite-dimensional linear classifiers as \(n\to\infty\) [38], [39].

Self-distillation has been observed to induce label averaging among highly correlated instances [28]. When \(\rho\) is small, the PD student can therefore correct mislabeled samples by aggregating labels from correctly classified neighbors, although incorrect neighbors also reduce its confidence on correctly labeled samples. Relative to the teacher, the student can thus trade confidence for accuracy. As \(\rho\) approaches \(0.5\) from below, however, incorrectly labeled neighbors increasingly mislead the student, and its predictions no longer cross the classification threshold on the mislabeled samples. Choosing \(\xi>1\) extrapolates beyond the PD student’s soft prediction and can amplify this corrective direction, although an excessively large \(\xi\) may hurt performance. We illustrate this behavior with synthetic experiments in [fig:synthetic_constant_small,fig:synthetic_uniform_small].

6.1 Soft teacher–student prediction mixing↩︎

3 mixes the teacher’s hard label with the PD student’s soft prediction and strictly improves upon both when \(\rho_0 < \rho < 0.5\). We now consider mixing soft predictions from both models. Specifically, we treat the PD student \(y_{\pd}(\cdot) = \sigma(\langle \theta_{\pd}, \phi(\cdot) \rangle)\) defined by 13 as a teacher that produces soft labels. Using its predictions on \(\cD_{\rm unlab}\), we train a second PD student \(y_{\pd}^{(2)}(\cdot) = \sigma(\langle \theta_{\pd}^{(2)}, \phi(\cdot) \rangle)\) with the same regularization parameter \(\lambda\): \[\begin{align} \label{eq:class95training95loss952} \theta_{\pd}^{(2)} &:= \argmin_{\theta} \frac{1}{2n} \sum_{i=1}^{2n} \text{CE} \big(y_{\pd}(x_i), \sigma( \langle \theta, \phi(x_i) \rangle) \big) + \frac{\lambda}{2} \| \theta \|^2. \end{align}\tag{15}\] For a mixing weight \(\xi\in\RR\), the resulting PMSD classifier thresholds the affine combination of the two soft predictions: \[\label{eq:logistic95affine952} y_{\mathsf{pmsd}, \xi}^{(2)}(x)=\II \bigl\{(1-\xi) y_{\pd}(x) + \xi y_{\pd}^{(2)}(x)\ge 0.5 \bigr\}.\tag{16}\]

Previous results (Theorem 5 of [19] and Theorem 4.1 of [28]) show that, as \(n\to\infty\), the second student \(y_{\pd}^{(2)}\) can tolerate a higher label noise rate \(\rho\) than its teacher \(y_{\pd}\), subject to \(\rho<0.5\). Specifically, there exists a threshold \(\rho_1\) with \(0 < \rho_0 < \rho_1 < 0.5\) such that (i) for \(\rho_0 < \rho < \rho_1\), the student \(y_{\pd}^{(2)}\) attains \(100\%\) accuracy while the teacher \(y_{\pd}\) has \(100(1-\rho)\%\) accuracy, and (ii) for \(\rho_1 < \rho < 0.5\), both achieve \(100(1-\rho)\%\) accuracy. In contrast, we show that, for every \(\rho<0.5\), an appropriate mixing parameter \(\xi\geq0\) allows the PMSD classifier \(y_{\mathsf{pmsd}, \xi}^{(2)}\) in 16 to attain \(100\%\) accuracy.

When \(\rho > 0.5\), incorrectly labeled neighbors outnumber correctly labeled ones. The resulting label averaging prevents the student from correcting misclassified samples and also causes it to misclassify samples that the teacher labeled correctly. In fact, both \(y_{\pd}\) and \(y_{\pd}^{(2)}\) achieve \(0\%\) population accuracy as \(n \to \infty\). Nevertheless, the PMSD student \(y_{\mathsf{pmsd}, \xi}^{(2)}\) can achieve asymptotic \(100\%\) population accuracy. See [fig:synthetic_constant_large,fig:synthetic_uniform_large] for synthetic illustrations with \(\rho > 0.5\).

In this regime, the first PD student \(y_{\pd}\) misclassifies both the samples that the original teacher labeled correctly and those it labeled incorrectly. Although the soft predictions of the second student \(y_{\pd}^{(2)}\) move in the correct direction relative to those of \(y_{\pd}\), the shift is insufficient to alter the induced hard predictions. With an appropriate mixing weight \(\xi > 1\), the PMSD student amplifies this corrective signal and extrapolates far enough to change the final predictions.

6.2 Linear probing experiments on real datasets↩︎

To illustrate the performance of PMSD, we consider multiclass classification with \(\ell_2\)-regularized cross-entropy loss on Caltech-101, Caltech-256 [40], and CIFAR-100 [41]. For each dataset, we train a linear classifier with a softmax output on frozen features from a ResNet-34 pretrained on ImageNet. We use this setup for both the teacher and the pure-distilled student: the teacher is trained on corrupted one-hot labels, whereas the student is trained on the teacher’s soft pseudo-labels using the same \(\lambda\). 1 reports the test accuracies of the teacher, PD student, and PMSD student. We estimate the mixing weight \(\xi\) on a small calibration set using a fine grid over \([-20, 20]\). Additional results across datasets and corruption levels appear in 16.2; experimental details are provided in 17.

7 Discussion↩︎

In this section, we discuss several possible extensions, compare the PMSD student with the same-\(X\) loss-mixed SD student, and examine the limitations of our proposed PMSD approach in the prediction-only regime, with a focus on high-dimensional ridge regression. This comparison clarifies both the advantages and the limitations of prediction-only distillation.

7.1 Negative regularization↩︎

Our analysis extends readily to negative regularization parameters \(\lambda_t\) and \(\lambda_s\), provided they are bounded below to prevent the risk from diverging. Negative regularization can be optimal on real datasets [42] and has since been studied in the high-dimensional ridge regression literature [43]–[45]. Under this extension, the strict improvement results in 4.1 remain essentially unchanged. In particular, [prop:teacher95bad95set] and [prop:strict95improv95ood] hold regardless of the signs of \(\lambda_t\) and \(\lambda_s\): the cardinality of the student-tie set is controlled by the number of real-valued solutions to a polynomial equation in \(\kappa_s\). Although the correspondence between \(\lambda_s\) and \(\kappa_s\) need not be one-to-one under negative regularization, each value of \(\kappa_s\) determines at most one value of \(\lambda_s\) through 6 .

7.2 Fresh-X prediction-mixed versus same-X loss-mixed SD↩︎

Fresh-\(X\) PMSD and the same-\(X\) loss-mixed SD methods studied by [19]–[22] operate with different information and therefore have different performance limits. Consider a teacher whose regularization parameter \(\lambda_t\) is mistuned for the target risk. A same-\(X\) student can reuse the original labeled sample to correct the teacher, whereas prediction-only PMSD can use only the teacher’s predictions on fresh unlabeled covariates. Consequently, fresh unlabeled data need not recover information lost in the teacher, regardless of the fresh sample size. 7 illustrates both the benefits and the limitations of PMSD relative to same-\(X\) loss mixing.

In the one-spike covariance model, [22] show that same-\(X\) loss-mixed SD, with jointly optimized \(\lambda_t\), \(\lambda_s\), and \(\xi \in \mathbb{R}\) and with negative regularization allowed, is optimal within the spectral shrinkage class given the labeled sample \(\mathcal{D}_{\text{lab}}=(X, y)\). Once additional fresh unlabeled data \(\mathcal{D}_{\text{unlab}}\) are available, the fresh-\(X\) PMSD student, which uses only the teacher’s ridge predictions, can nevertheless outperform this optimal same-\(X\) student; see 7 (a). The gains are especially pronounced in the overparameterized regime \(p>n_t\).

The comparison reverses in the isotropic design setting, where the optimally tuned teacher is optimal within the spectral shrinkage class. Because the same-\(X\) loss-mixed student is itself a spectral shrinkage estimator [22], it cannot outperform the optimally tuned teacher. If the given teacher is mistuned, however, the same-\(X\) student can attain the optimal ridge risk by setting \(\lambda_s=\lambda_t^\star\) and effectively ignoring the teacher’s predictions while using the original training data. In contrast, PMSD relies only on the mistuned teacher’s predictions on fresh unlabeled covariates. It can substantially improve upon this teacher but generally cannot attain the optimal ridge risk or outperform the optimally tuned teacher; see 7 (b). 17 presents a similar comparison for an AR1 design.

7.3 Optimal fresh-\(X\) covariance structure↩︎

Our framework also raises a natural design question: given the teacher predictor, from which distribution should the fresh unlabeled covariates be sampled to minimize the PMSD risk? Within the class of covariances \(\Sigma_s\) that are simultaneously diagonalizable with \(\Sigma_t\), this amounts to maximizing the teacher-to-PMSD risk reduction in 1, \[\label{eq:risk95reduction} \mathcal{R}_{\te}(\lambda_t) - \sR_{\mathsf{pmsd}}^{\star}(\lambda_t, \lambda_s) = \frac{\bigl(\mathcal{R}_{\te}(\lambda_t)-\mathcal{C}(\lambda_t, \lambda_s)\bigr)^2}{\mathcal{R}_{\te}(\lambda_t)+\mathcal{R}_{\pd}(\lambda_s) - 2 \mathcal{C}(\lambda_t, \lambda_s)}.\tag{17}\]

Because this quantity also depends on \(\lambda_t\) and \(\lambda_s\), the optimal choice \(\Sigma_s^{\star}\) generally varies with the regularization parameters. Jointly optimizing \(\Sigma_s\) and \(\lambda_s\) for a given teacher is considerably more involved, and we leave this problem for future work.

We instead consider the underparameterized regime \(\gamma_t, \gamma_s < 1\) and the post-asymptotic ridgeless limit \(\lambda_t, \lambda_s \to 0\). Recall that \(\Sigma_t = U^{\top} \operatorname{diag}(\sigma_1, \ldots, \sigma_p) U\), \(\Sigma_s = U^{\top} \operatorname{diag}(\tilde{\sigma}_1, \ldots, \tilde{\sigma}_p) U\), and \(v_i = ((U\beta)_i)^2\). For each \(i \in [p]\), define \(w_i = \tilde{\sigma}_i^{-1}\) and let \(w = (w_1, \ldots, w_p)^{\top}\). The limiting risk reduction in 17 is invariant under the rescaling \(\Sigma_s\mapsto c\Sigma_s\) for \(c > 0\), so we impose the normalization \(p^{-1}\sum_{i=1}^p w_i=1\). Define \[s_i := \sigma_i v_i + \frac{\sigma^2 \gamma_t}{p(1 - \gamma_t)}, \quad t_i := \frac{\gamma_s}{2p(1-\gamma_s)} \biggl( v_i + \frac{\sigma^2 \gamma_t}{p \sigma_i (1 - \gamma_t)} \biggr), \quad i = 1, \ldots, p,\] and \[S = \operatorname{diag}(s_1, \ldots, s_p), \quad t = (t_1, \ldots, t_p)^{\top}, \quad \theta = (\sigma_1, \ldots, \sigma_p)^{\top}.\]

This is a linearly constrained quadratic program in \(w\) and can be solved numerically using standard quadratic programming methods. In general, the spectrum of the optimal \(\Sigma_s^\star\) depends on the teacher covariance, the signal-covariance alignment, the noise level, and the data aspect ratios. In particular, matching the teacher covariance need not be optimal: \(\Sigma_s^\star\) need not be proportional to \(\Sigma_t\). This differs from a recent result on optimal labeled-data selection for ridgeless estimators [46], reflecting the different role of the fresh sample in our prediction-only setting.

Together with our calibration and logistic regression results, these observations outline the scope of PMSD: prediction mixing can exploit fresh unlabeled data to improve a fixed teacher, and a small labeled sample can make this improvement operational, but PMSD cannot in general recover the information in the original labeled sample. Optimizing the fresh covariate distribution beyond the simultaneously diagonalizable ridgeless setting is a natural direction for future work.

Acknowledgments↩︎

We thank Sujay Sanghavi, Adel Javanmard, Peter Müeller, Antonio Linero, Edward George, and participants in the Best of Statistical Science (“BOSS”) Workshop for helpful discussions.

This supplement accompanies the paper “Prediction-Only Distillation in Linear and Logistic Regression”. It contains additional details, illustrations, and proofs of all the theoretical results. The supplement is organized as follows.

8 Notation↩︎

General. We denote vectors by lowercase letters and matrices by uppercase letters. We use blackboard letters for standard number systems: \(\NN\) denotes the positive integers, \(\RR\) the real numbers, and \(\overline{\RR}\) the extended real line.

For a positive integer \(n\), we write \([n]:=\{1,\ldots,n\}\). For \(x,y\in\RR\), we write \(x \wedge y:=\min\{x,y\}\) and \(x \vee y:=\max\{x,y\}\). For an event or set \(A\), \(\ind_A\) denotes its indicator.

Linear algebra. For a vector \(x\), \(\|x\|_2\) denotes its Euclidean norm; when the subscript is omitted for simplicity, \(\|x\|\) has the same meaning. For a matrix \(X \in \RR^{n \times p}\), \(X^\top \in \RR^{p \times n}\) denotes its transpose and \(X^{\dagger} \in \RR^{p \times n}\) its Moore–Penrose pseudoinverse. For a square matrix \(A \in \RR^{p \times p}\), \(\tr(A)\) denotes its trace and \(\otr(A):=\tr(A)/p\) its normalized trace. If \(A\) is invertible, \(A^{-1}\) denotes its inverse. For a symmetric positive semidefinite matrix \(\Sigma\), \(\Sigma^{1/2}\) denotes its principal square root. The \(p \times p\) identity matrix is denoted by \(I_p\), or simply by \(I\) when its dimension is clear. For a matrix \(X\), \(\|X\|_{\mathrm{op}}\) denotes the operator norm induced by the Euclidean vector norm, and \(\|X\|_F\) denotes the Frobenius norm. For a matrix \(M\), \(\|M\|_{\tr}\) denotes its trace norm, equal to the sum of its singular values.

Asymptotics. For \(Y\geq 0\), we write \(X=O_\alpha(Y)\) if \(|X|\leq C_\alpha Y\), where \(C_\alpha\) may depend on the ambient parameter \(\alpha\) but not on the other parameters under consideration. We use \(O_p\) for the corresponding probabilistic order notation. The notation \(\Theta(1)\) denotes a quantity whose absolute value is bounded above and away from zero by constants independent of the asymptotic index. We write \(X_n\pto X\) for convergence in probability and \(X_n\asto X\) for almost sure convergence. When the mode of convergence is clear from context, we write \(X_n \to Y_n\) to mean \(|X_n-Y_n|\to 0\) in that mode. We use \(C\) and \(C'\) for generic positive absolute constants whose values may change from line to line.

9 Further related work↩︎

Retraining on synthetic labels and model collapse. Beyond a single round of pure-distillation, recent work studies how recursively training on model-generated data can degrade performance, a phenomenon known as model collapse [47]–[52]. [53] and [54] study infinitely repeated schemes for ridge and ridgeless regression that mix ground-truth and synthetic labels with a fixed weight \(w \in [0,1]\) to prevent such degradation. [55] studies an extension of this iterative training framework to the next-token prediction setting, using a mixture of real and synthetic data at the population level. [52] use the term “strong model collapse” when a model trained on synthetic data has worse population risk than one trained only on ground-truth data. Our setting differs in two respects: we study a fixed, finite distillation procedure using fresh unlabeled covariates, and the final predictor mixes predictions rather than recursively replacing ground-truth data with synthetic data. Accordingly, our results concern the benefits of post hoc prediction mixing, rather than the long-run stability of repeated synthetic training; see [sec:strict_nonmonotonicity,sec:logistic].

Risk estimation and tuning. Our calibration analysis in 5 connects to the literature on out-of-sample risk estimation for high-dimensional ridge regression and related models, including analyses based on leave-one-out and generalized cross-validation [44], [56]–[58]. Risk estimation has also been studied for finite ensembles of penalized estimators and subagged regularized \(M\)-estimators [59], [60]. Unlike those settings, PMSD combines two predictors linked through pseudo-labeling, and our goal is to estimate their residual correlation and optimal mixing weight using a small independent labeled calibration sample, without retraining either model.

10 Proofs in Section 3↩︎

10.1 Preliminaries↩︎

Define the empirical covariance matrices for the teacher’s training covariates and the student’s fresh covariates, together with their resolvents, by \[\begin{align} \widehat\Sigma:=\frac{1}{n_t} X^\top X,\qquad \widetilde{\Sigma}:=\frac{1}{n_s} \tilde{X}^\top\tilde{X}, \qquad Q_{\lambda_t}:=(\widehat\Sigma+\lambda_t I_p)^{-1},\qquad \widetilde{Q}_{\lambda_s}:=(\widetilde{\Sigma}+\lambda_s I_p)^{-1}. \nonumber \end{align}\]

The teacher coefficient is \(\beta_{\lambda_t}:=Q_{\lambda_t} X^\top y/n_t\). The fresh-\(X\) pure-distilled (PD) coefficient is \[\beta_{\pd,\lambda_s} :=\widetilde{Q}_{\lambda_s} \,\widetilde{\Sigma}\,\beta_{\lambda_t} =\big(I_p - \lambda_s \widetilde{Q}_{\lambda_s} \big)\beta_{\lambda_t}.\] The prediction-mixed student family is \[f_{\mathsf{pmsd},\lambda_t, \lambda_s, \xi}(x)=(1-\xi)f_{\lambda_t}(x)+\xi f_{\pd,\lambda_s}(x), \qquad f_{\lambda_t}(x)=x^\top\beta_{\lambda_t},\;\;f_{\pd,\lambda_s}(x)=x^\top\beta_{\pd,\lambda_s}.\] Recall the in-distribution risks and correlation \[\begin{align} R_{\te}(\lambda_t) &:=\E[(y_0-f_{\lambda_t}(x_0))^2\mid \cD_{\rm lab}], \qquad R_{\pd}(\lambda_s):=\E[(y_0-f_{\pd,\lambda_s}(x_0))^2 \mid \cD_{\rm lab}, \cD_{\rm unlab}], \nonumber \\ C(\lambda_t, \lambda_s)&:=\E[(y_0 - f_{\lambda_t}(x_0)) (y_0 -f_{\pd,\lambda_s}(x_0))\mid\cD_{\rm lab}, \cD_{\rm unlab}], \nonumber \\ R_{\mathsf{pmsd}}(\lambda_t, \lambda_s, \xi) &= \E[(y_0-f_{\mathsf{pmsd},\lambda_t, \lambda_s, \xi}(x_0))^2 \mid \cD_{\rm lab}, \cD_{\rm unlab}], \nonumber \end{align}\] Denote the deterministic equivalents of the first three quantities by \(\sR(\lambda_t)\), \(\sR_{\pd}(\lambda_s)\), and \(\sC(\lambda_t, \lambda_s)\), respectively. Also define \(D(\lambda_t, \lambda_s) := R_{\te}(\lambda_t) + R_{\pd}(\lambda_s) - 2C(\lambda_t, \lambda_s) \geq 0\).

Recall the random-design model in 1. The population \(L_2\)-projection coefficient, residual, and residual variance are \[\beta := \E[xx^\top]^{-1}\E[xy] = \Sigma_t^{-1}\E[xy], \qquad \varepsilon := y - x^\top\beta, \qquad \sigma^2 := \E[\varepsilon^2].\] Then \(\E[x\varepsilon]=0\) (equivalently, \(\E[\bz\,\varepsilon]=0\)) and \(\E[\varepsilon]=0\). Throughout, we consider in-distribution prediction: the test pair \((x_0,y_0)\) is an independent copy of \((x,y)\), so \(y_0=x_0^\top\beta+\varepsilon_0\), with \(\E[x_0\varepsilon_0]=0\) and \(\E[\varepsilon_0^2]=\sigma^2\).

10.2 Proof of [prop:oracle95affine]↩︎

10.3 Deterministic equivalents for the proof of 1↩︎

Fix \(\lambda_t > 0\). Let \(\kappa_t=\kappa_t(\lambda_t)>0\) denote the teacher fixed-point solution \[\kappa_t = \lambda_t +\gamma_t \kappa_t \,\otr\!\big(\Sigma_t (\Sigma_t +\kappa_t I_p)^{-1}\big),\] and define \[G_t:=(\Sigma_t +\kappa_t I_p)^{-1}.\] Likewise, let \(\kappa_s=\kappa_s(\lambda_s)>0\) denote the PD student fixed-point solution \[\kappa_s = \lambda_s + \gamma_s \kappa_s\,\otr\!\big(\Sigma_s (\Sigma_s +\kappa_s I_p)^{-1}\big),\] and define \[G_s:=(\Sigma_s +\kappa_s I_p)^{-1}.\]

For \(i, j \geq 0\), define the mixed signal-covariance alignment and trace functionals by \[q_{i,j}:=\beta^\top G_t^i G_s^j \Sigma_t \,\beta, \qquad \tilde{q}_{i,j}:=\beta^\top G_t^i G_s^j \Sigma_s \,\beta, \qquad t_{i,j}:=\gamma_t \,\otr(\Sigma_t^2 G_t^i G_s^j),\] and set \[b_t := (1 - t_{2,0})^{-1}, \quad b_s := \frac{\gamma_s \otr(\Sigma_s \Sigma_t G_s^2) }{1 - \gamma_s \otr(\Sigma_s^2 G_s^2) }.\]

10.4 Proof of 1↩︎

10.5 Background on deterministic equivalents↩︎

We use an anisotropic (bilinear-form) notion of deterministic equivalent.

This notion implies trace convergence whenever operator norms are uniformly bounded. In particular, if \(A\asymp \bar A\) and \(\|A\|_{\mathrm{op}},\|\bar A\|_{\mathrm{op}}=O_{\PP}(1)\), then \(\otr(A)-\otr(\bar A)\pto 0\).

We will also use that uniform deterministic equivalents can be differentiated.

For the next two lemmas, let \(\widehat\Sigma=X^\top X/n\) be a generic sample covariance matrix with population covariance \(\Sigma\) and aspect ratio \(\gamma=p/n\), and let \(Q_\lambda=(\widehat\Sigma+\lambda I_p)^{-1}\). A standard anisotropic local law for sample-covariance resolvents gives the following deterministic equivalent; see, e.g., [61]–[63].

Define \[t_2(\lambda):=\gamma \otr(\Sigma^2 G^2), \qquad b(\lambda):=\frac{1}{1-t_2(\lambda)}.\]

11 Proofs in Section 4↩︎

11.1 Proof of [prop:teacher95bad95set]↩︎

11.2 Proof of [prop:strict95improv95ood]↩︎

11.3 Proof of 1↩︎

11.4 Proof of 2↩︎

11.5 Proof of 3↩︎

11.6 Proof of [prop:strict95improv95pd]↩︎

12 Proofs in Section 5↩︎

12.1 Proof of 2↩︎

We prove a slightly more general finite-candidate version of 2. This version is useful when the same calibration set is used to choose among finitely many student regularization parameters \(\lambda_s\), unlabeled sample sizes \(n_s\), or other finite hyperparameter choices.

The argument is straightforward: once the teacher and fresh-\(X\) pure-distilled student are fixed, calibration estimates only the risks and residual correlation of two fixed predictors. Thus, no high-dimensional condition on \(p/n_{\rm cal}\) is needed.

Let \(\mathcal{G}\) denote the sigma-field generated by the deployed teacher, the fresh unlabeled covariates, and all fitted pure-distilled students under consideration. Conditionally on \(\mathcal{G}\), the teacher and student predictors are fixed functions. For a single candidate \(f_{\pd}\), let \((X_i^{\rm cal},Y_i^{\rm cal})_{i=1}^{n_{\rm cal}}\) be an independent labeled calibration sample drawn from the target test distribution. Define \[e_i^{\te}:=Y_i^{\rm cal}-f^{\te}(X_i^{\rm cal}), \qquad e_i^{\pd}:=Y_i^{\rm cal}-f_{\pd}(X_i^{\rm cal}).\] The oracle quantities are \[R_{\te}:=\E[(e^{\te})^2\mid \mathcal{G}], \qquad R_{\pd}:=\E[(e^{\pd})^2\mid \mathcal{G}], \qquad C:=\E[e^{\te}e^{\pd}\mid \mathcal{G}],\] where \((e^{\te},e^{\pd})\) denotes an independent test residual pair. The calibration estimates are \[\widehat R_{\te}^{\rm cal} := \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}(e_i^{\te})^2, \qquad \widehat R_{\pd}^{\rm cal} := \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}(e_i^{\pd})^2, \qquad \widehat C^{\rm cal} := \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}}e_i^{\te}e_i^{\pd}.\]

We first prove the following finite-candidate result.

This finite-candidate consistency statement immediately implies that calibration can also be used for finite hyperparameter selection.

12.2 Vector-valued predictors↩︎

The same calibration method extends directly to vector-valued prediction, which is used in our squared-loss linear-probing experiments. Suppose \(Y\in\mathbb{R}^K\) and predictors \(f:\mathcal{X}\to\mathbb{R}^K\) are evaluated by squared risk \[R(f):=\E\big[\|Y-f(X)\|_2^2\big].\] For two fixed predictors \(f^{\te}\) and \(f_{\pd}\), define vector residuals \[e^{\te}:=Y-f^{\te}(X), \qquad e^{\pd}:=Y-f_{\pd}(X),\] and set \[R_{\te}:=\E[\|e^{\te}\|_2^2], \qquad R_{\pd}:=\E[\|e^{\pd}\|_2^2], \qquad C:=\E[\langle e^{\te},e^{\pd}\rangle].\] Then the same affine-risk expansion gives \[\xi^\star = \frac{R_{\te}-C}{R_{\te}+R_{\pd}-2C}, \qquad R_{\mathsf{pmsd}}^{\star} = R_{\te} - \frac{(R_{\te}-C)^2}{R_{\te}+R_{\pd}-2C}.\] The calibration estimates are obtained by replacing scalar squares and products with squared Euclidean norms and inner products: \[\widehat R_{\te}^{\rm cal} = \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}} \|e_i^{\te}\|_2^2, \quad \widehat R_{\pd}^{\rm cal} = \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}} \|e_i^{\pd}\|_2^2, \quad \widehat C^{\rm cal} = \frac{1}{n_{\rm cal}}\sum_{i=1}^{n_{\rm cal}} \langle e_i^{\te},e_i^{\pd}\rangle.\] The proof of [thm:calibration95consistency95finite95candidates] applies directly under the corresponding finite-variance assumptions on \[\|e^{\te}\|_2^2, \qquad \|e^{\pd}\|_2^2, \qquad \langle e^{\te},e^{\pd}\rangle.\]

A note on the computational cost: The calibration estimator is “one-shot” in the following sense. For a fixed candidate student \(f_{\pd}\), after training \(f_{\pd}\) once on fresh pseudo-labels, tuning \(\xi\) requires only \[\big(f^{\te}(x_i^{\rm cal}), f_{\pd}(x_i^{\rm cal}), y_i^{\rm cal}\big)_{i=1}^{n_{\rm cal}},\] followed by three empirical averages and the closed-form plug-in formula. There is no retraining over candidate mixing weights \(\xi\). If one considers finitely many candidate students, such as different student penalties \(\lambda_s\) or different fresh unlabeled sample sizes \(n_s\), each student must be trained once, but the same calibration set can evaluate all candidates by 4.

13 Proofs in Section 6↩︎

13.1 Preliminaries↩︎

For readability, we write \(\rho\) in the finite sample systems below. When \(n\rho\) is not an integer, this notation denotes \(\rho_n:=\lfloor n\rho\rfloor/n\); since \(\rho_n\to\rho\), this does not affect the limiting conclusions.

Recall the training objective for the student classifier based on the teacher’s hard labels: \[\begin{align} \label{eq:class95training95loss95appendix} \theta_{\pd} := \argmin_{\theta} \underbrace{\frac{1}{2n} \sum_{i=1}^{2n} \text{CE}(\hat{y}_i, \sigma( \langle \theta, \phi(x_i) \rangle)) + \frac{\lambda}{2} \| \theta \|^2}_{\mathcal{L}_1(\theta)}. \end{align}\tag{29}\]

The loss can be rewritten as \[\begin{align} \mathcal{L}_1(\theta) &= \frac{1}{2n} \sum_{i=1}^{2n} \Bigl( \log \big(1 + e^{\langle \theta, \phi(x_i) \rangle} \big) - \hat{y}_i \langle \theta, \phi(x_i) \rangle \Bigr) + \frac{\lambda}{2} \| \theta \|^2, \\ \nabla \mathcal{L}_1(\theta) &= \frac{1}{2n} \sum_{i=1}^{2n} \Big( \sigma(\langle \theta, \phi(x_i) \rangle) - \hat{y}_i \Big) \phi(x_i) + \lambda \theta. \end{align}\]

Setting \(\nabla \mathcal{L}_1(\theta)=0\) gives \[\theta_{\pd} = \sum_{i=1}^{2n} \underbrace{\frac{1}{2n \lambda} \Big( \hat{y}_i - \sigma ( \langle \theta_{\pd}, \phi(x_i) \rangle ) \Big) }_{:=\alpha(x_i, \hat{y}_i)} \phi(x_i) = \sum_{i=1}^{2n} \alpha(x_i, \hat{y}_i) \phi(x_i).\]

It follows that \[\begin{align} {y_{\pd}(x_i) = \sigma ( \langle \theta_{\pd}, \phi(x_i) \rangle ) = \hat{y}_i - 2 n \lambda \alpha(x_i, \hat{y}_i).} \end{align}\]

The following result characterizes these coefficients.

By 7, the student predictions on the training set are \[\label{eq:pd95prediction} y_{\pd}(x_i) = \begin{cases} \bar{\lambda}_n \hat{\alpha}_n & \text{for } i \in \mathcal{S}_{1,\mathrm{bad}}, \\ 1 - \bar{\lambda}_n \alpha_n & \text{for } i \in \mathcal{S}_{1,\mathrm{good}}, \\ 1 - \bar{\lambda}_n \hat{\alpha}_n & \text{for } i \in \mathcal{S}_{0,\mathrm{bad}}, \\ \bar{\lambda}_n \alpha_n & \text{for } i \in \mathcal{S}_{0,\mathrm{good}}. \end{cases}\tag{30}\]

For \(y_{\pd}\) to achieve \(100\%\) training accuracy with respect to the ground-truth labels, we require \(\bar{\lambda}_n\hat{\alpha}_n>0.5\) and \(\bar{\lambda}_n\alpha_n<0.5\).

In 6.1, we train a second student on the first student’s soft labels \(y_{\pd}(\cdot)\) over the same unlabeled set \(\cD_{\rm unlab}\). Its training objective is \[\begin{align} \label{eq:class95training95loss95295appendix} \theta_{\pd}^{(2)} &:= \argmin_{\theta} \underbrace{\frac{1}{2n} \sum_{i=1}^{2n} \text{CE}(y_{\pd}(x_i), \sigma( \langle \theta, \phi(x_i) \rangle)) + \frac{\lambda}{2} \| \theta \|^2}_{\mathcal{L}_2(\theta)}. \end{align}\tag{31}\]

Setting \(\nabla\mathcal{L}_2(\theta)=0\) gives

\[\theta_{\pd}^{(2)} = \sum_{i=1}^{2n} \underbrace{\frac{1}{2n \lambda} \Big( y_{\pd}(x_i) - \sigma ( \langle \theta_{\pd}^{(2)}, \phi(x_i) \rangle ) \Big) }_{:=\beta_i} \phi(x_i) = \sum_{i=1}^{2n} \beta_i \phi(x_i).\]

The following lemma characterizes the coefficients \(\beta_i\).

By 8, the second student’s predictions on the training set are \[\label{eq:pd95295prediction} y_{\pd}^{(2)}(x_i) = \begin{cases} \bar{\lambda}_n \hat{\alpha}_n + \bar{\lambda}_n \hat{\beta}_n & \text{for } i \in \mathcal{S}_{1,\text{bad}}, \\ 1 - \bar{\lambda}_n \alpha_n - \bar{\lambda}_n \beta_n & \text{for } i \in \mathcal{S}_{1,\text{good}}, \\ 1 - \bar{\lambda}_n \hat{\alpha}_n - \bar{\lambda}_n \hat{\beta}_n & \text{for } i \in \mathcal{S}_{0,\text{bad}}, \\ \bar{\lambda}_n \alpha_n + \bar{\lambda}_n \beta_n & \text{for } i \in \mathcal{S}_{0,\text{good}}. \end{cases}\tag{32}\]

For \(y_{\pd}^{(2)}\) to achieve \(100\%\) training accuracy, we require \(\bar{\lambda}_n(\hat{\alpha}_n+\hat{\beta}_n)>0.5\) and \(\bar{\lambda}_n(\alpha_n+\beta_n)<0.5\).

To distinguish the affine scores from the thresholded classifiers in 14 and 16 , define \[s_{\mathsf{pmsd},\xi}(x):=(1-\xi)\hat{y}(x)+\xi y_{\pd}(x), \qquad s_{\mathsf{pmsd},\xi}^{(2)}(x):=(1-\xi)y_{\pd}(x)+\xi y_{\pd}^{(2)}(x).\]

13.2 Proof of 3↩︎

13.3 Proof of 4↩︎

13.4 Proof of 5↩︎

14 Proof in 7↩︎

14.1 Proof of [prop:optimize95Sigma95s]↩︎

15 Additional theoretical results↩︎

15.1 Non-monotonicity in the fresh unlabeled sample size↩︎

In this subsection, we consider the isotropic and same-\(\lambda\) setting, i.e., \(\lambda_t = \lambda_s = \lambda\). We demonstrate that the optimal fresh-\(X\) PMSD risk is not globally monotone in \(\gamma_s\). Instead, it has a simple one-dimensional structure and is unimodal in \(\gamma_s\).

Assume \(\Sigma_t = \Sigma_s = I_p\) and \(\beta\sim\cN(0,(r^2/p)I_p)\), where \(r^2>0\), and let \[\lambda^\star:=\gamma_t\frac{\sigma^2}{r^2}.\] For the labeled sample, let \(\kappa_t=\kappa_t(\lambda)\) be the unique positive solution to \[\kappa_t = \lambda+\frac{\gamma_t\kappa_t}{1+\kappa_t},\] and set \[b_t:=\Big(1-\frac{\gamma_t}{(1+\kappa_t)^2}\Big)^{-1}.\] For the fresh unlabeled sample, let \(\kappa_s=\kappa_s(\lambda)\) be the unique positive solution to \[\kappa_s = \lambda+\frac{\gamma_s\kappa_s}{1+\kappa_s},\] and define \[c_s:=\Big(1-\frac{\gamma_s}{(1+\kappa_s)^2}\Big)^{-1}.\]

Let \(\bar \sR_{\te}(\lambda):=\sR_{\te}(\lambda)-\sigma^2\) denote the excess teacher risk. Define the excess PD-student risk \(\bar \sR_{\pd}\) and the excess correlation \(\bar \sC\) similarly.

In the isotropic model, the limiting teacher risk in 18 simplifies to \[\label{eq:barR95teacher95iso95again} \bar \sR_{\te}(\lambda) = \frac{b_t}{(1+\kappa_t)^2} \big(r^2\kappa_t^2+\sigma^2\gamma_t\big).\tag{52}\] Define also \[\label{eq:Theta95T95labeled} \Theta_t(\lambda) := \bar \sR_{\te}(\lambda)-r^2\frac{\kappa_t}{1+\kappa_t}, \qquad T_t(\lambda) := \bar \sR_{\te}(\lambda)+r^2\frac{1-\kappa_t}{1+\kappa_t}.\tag{53}\]

We are now ready to state the key result in this subsection.

Since smaller \(\gamma_s=p/n_s\) corresponds to more unlabeled data, [prop:freshX95unlabeled95effect95iso] shows that adding more fresh unlabeled data is not uniformly beneficial. For fixed \(\lambda\neq\lambda^\star\):

• if \(\gamma_s<1+\lambda\), then decreasing \(\gamma_s\) (adding more unlabeled data) decreases the optimal PMSD risk;

• if \(\gamma_s>1+\lambda\), then decreasing \(\gamma_s\) initially increases the optimal PMSD risk, until the turning point \(\gamma_s=1+\lambda\) is reached.

Thus, the fresh-\(X\) PMSD risk is unimodal, not monotone, in the amount of unlabeled data.

15.2 Non-identifiability of tuning without labeled data↩︎

Recall that, for squared prediction risk with test covariance \(\Sigma\succ 0\), the oracle weight is \[\xi^\star = \frac{R_{\te}-C}{D} = \frac{\langle \beta^{\te}_{\lambda_t}-\beta,\;\beta^{\te}_{\lambda_t}-\beta_{\pd, \lambda_s}\rangle_{\Sigma}}{\|\beta^{\te}_{\lambda_t}-\beta_{\pd, \lambda_s}\|_{\Sigma}^2},\] provided \(D(\lambda_t, \lambda_s)>0\). Thus, for known \(\Sigma\), the denominator depends only on the observable pair \((\beta^{\te}_{\lambda_t},\beta_{\pd, \lambda_s})\), whereas the numerator still depends on the unknown target parameter \(\beta\).

As an immediate corollary, there is no uniformly consistent estimator of \(\xi^\star\) based only on the teacher and unlabeled sample. Indeed, any measurable rule \(\hat{\xi}=\hat{\xi}(\beta^{\te}_{\lambda_t},\tilde{X})\) takes the same value for the setting in [prop:no95label95tuning95impossible], whereas the target \(\xi^\star_{\eta}(\lambda_t,\lambda_s)=\eta\) can be changed arbitrarily.

16 Additional experiments↩︎

Code for reproducing the results is available at https://github.com/hhd357/prediction_only_mixing_distillation.

16.1 Real-world regression tasks with different student regularization levels↩︎

We report additional ridge regression results on Communities and Crime and Blog Feedback. For each dataset, we compare fresh covariates drawn from the teacher’s training distribution with fresh covariates drawn from an isotropic Gaussian distribution, fixing or tuning \(\lambda_s\) as in the captions.

16.1.1 Communities and Crime↩︎

16.1.2 Blog Feedback↩︎

16.2 Linear probing experiments on real datasets↩︎

The following tables report test classification accuracy and the estimated optimal mixing weight across regularization levels. In every experiment, the mixing weight is selected using a calibration set over the grid \([-20, 20]\); see 17 for the dataset and training details.

16.2.1 Caltech-101↩︎

16.2.2 Caltech-256↩︎

16.2.3 CIFAR-100 (random corruption)↩︎

16.2.4 CIFAR-100 (hierarchical corruption)↩︎

16.3 Synthetic experiments for ridge regression↩︎

We next compare the empirical performance of PMSD with its theoretical predictions under several covariance and regularization settings.

16.3.1 Empirical versus theoretical predictions↩︎

16.3.2 Comparison with the same-\(X\) SD student↩︎

16.4 Synthetic experiments for logistic regression↩︎

16.4.1 Constant feature correlation↩︎

16.4.2 Uniform feature correlation↩︎

17 Experiment details↩︎

17.1 Ridge regression experiments↩︎

For the real-world ridge regression experiments in the left panel of 2 and the middle and right panels of 6, we use UCI Communities and Crime [64], UCI Blog Feedback [65], and UCI Airfoil [66]. We divide each dataset into a teacher training set, a fresh unlabeled set, a calibration set, and a test set, with the sizes specified below or in the corresponding figure captions. After handling missing values as described below, we center and standardize all variables using the means and standard deviations computed on the teacher training set.

UCI Communities and Crime. The UCI Communities and Crime dataset [64] combines socioeconomic data from multiple sources. It consists of 1,994 samples with 127 covariates. The response, “ViolentCrimesPerPop,” is the rate of violent crimes per 100,000 population. We remove 5 nonpredictive identifiers and 23 covariates from the LEMAS survey that contain 1,675 missing values across the 1,994 samples. The remaining \(99\) covariates have no missing values and are used in our experiments.

UCI Blog Feedback. The UCI Blog Feedback dataset [65] was constructed from crawled and processed HTML documents of blog posts. We use its training split, which contains 52,397 samples, and reserve 30% of these samples for testing. The sizes of the teacher training, fresh unlabeled, and calibration sets are given in the corresponding figure captions.

UCI Airfoil. The UCI Airfoil Self-Noise dataset [66] contains aerodynamic and acoustic measurements from NASA tests of two- and three-dimensional airfoil blade sections. The continuous response is the sound pressure level, measured in decibels. The dataset contains \(n = 1503\) samples and \(p = 5\) covariates: frequency, attack-angle, free-stream-velocity, chord-length, and suction-side-displacement-thickness. We reserve 30% of the samples for testing. The sizes of the teacher training, fresh unlabeled, and calibration sets are given in the corresponding figure captions.

CIFAR-10. The CIFAR-10 dataset [41] consists of \(32 \times 32\) color images from 10 classes, with 50,000 training images and 10,000 test images. We use subsets of the training and test splits, with sizes specified in 6.

For the CIFAR-10 ridge regression experiment in the left panel of 6, we extract 512-dimensional features using a ResNet-18 [67] pretrained on ImageNet and available in PyTorch. We perform no additional data preprocessing and fit ridge regression to these frozen representations. The resulting predictor is a vector-valued function \(f:\RR^{512}\to\RR^{10}\). Using the one-hot label vector \(y\in\RR^{10}\) as the response, we define the aggregate squared risk by summing over the 10 output dimensions: \[\begin{align} R(f) = \EE_{(x,y)}\biggl[ \sum_{k=1}^{10} (y_k-f_k(x))^2 \biggr], \end{align}\] where \(f_k:\RR^{512}\to\RR\) is the ridge predictor for the \(k\)th component of the one-hot response.

17.2 Linear probing experiments↩︎

For the linear probing experiments on Caltech-101, Caltech-256, and CIFAR-100, we extract features using a ResNet-34 [67] pretrained on ImageNet and available in PyTorch. We then train a linear classifier with a softmax output on top of these frozen features. Results appear in the right panel of 2, 1, and 16.2.

Training details. The teacher is trained by minimizing cross-entropy against the corrupted one-hot labels. The PD student is trained by minimizing cross-entropy against the teacher’s softmax predictions. Each linear classifier is trained for 200 epochs using stochastic gradient descent with momentum 0.9 and a learning-rate decay factor of 0.98. The initial learning rates are \(0.001\) for the teacher and \(0.05\) for the student. We perform no data preprocessing, and all experiments are run on an NVIDIA T4 GPU.

Caltech-101. Caltech-101 [40] is an object recognition dataset with 101 object categories plus a background category. It contains approximately 9,000 images, with 40 to 800 images per category, and is therefore highly imbalanced. We draw disjoint subsets containing \(n_t = 1500\) teacher training samples, \(n_s = 4500\) fresh unlabeled samples, \(n_{\mathrm{cal}} = 500\) calibration samples, and 2,000 test samples.

Caltech-256. Caltech-256 [40] contains 30,607 images spanning 256 object categories. We draw disjoint subsets containing \(n_t = 5000\) teacher training samples, \(n_s = 15000\) fresh unlabeled samples, \(n_{\mathrm{cal}} = 1000\) calibration samples, and 6,000 test samples.

CIFAR-100. CIFAR-100 [41] contains 60,000 images from 100 classes. These classes are grouped into 20 superclasses, each containing five related fine-grained classes; for example, beaver, dolphin, otter, seal, and whale belong to the same superclass. We draw disjoint subsets containing \(n_t = 5000\) teacher training samples, \(n_s = 15000\) fresh unlabeled samples, \(n_{\mathrm{cal}} = 1000\) calibration samples, and 9,000 test samples.

Random corruption. Under random label corruption with rate \(\rho\), we select a fraction \(\rho\) of the teacher’s training samples uniformly at random and replace each label with one drawn uniformly from the remaining classes.

Hierarchical corruption. This scheme models errors between semantically similar classes. Because only CIFAR-100 provides the required superclass structure, we use hierarchical corruption only for this dataset. At corruption rate \(\rho\), we select a fraction \(\rho\) of the teacher’s training samples uniformly at random and replace each label with one drawn uniformly from the other classes in the same superclass.

17.3 Synthetic experiments↩︎

17.3.1 Ridge regression↩︎

For the synthetic ridge experiments in 3 and the additional experiments in 16.3, we use the following covariance and signal models.

• Isotropic covariance: \(\Sigma = I_p\).

• One-spike covariance: \(\Sigma = I_p + 5vv^{\top}\), where \(v=g/\|g\|_2\) and \(g\sim\mathcal{N}(0,I_p)\).

• AR1 covariance: \(\Sigma_{ij}=\rho^{|i-j|}\) for all \(i,j\), with correlation parameter \(\rho=0.25\).

• Isotropic signal: \(\beta\sim\mathcal{N}(0,(r^2/p)I_p)\).

• Top-aligned signal: let \(m\in(0,1)\) be the alignment proportion, \(a\in(0,1)\) the alignment factor, and \(k=\lfloor mp\rfloor\), where \(1\leq k\leq p-1\). If \(V=[v_1,\ldots,v_p]\) contains the eigenvectors of \(\Sigma\) in decreasing order of their eigenvalues, define \[\Sigma_\beta :=pV\operatorname{diag}\!\Bigl( \underbrace{\frac{a}{k},\ldots,\frac{a}{k}}_{k}, \underbrace{\frac{1-a}{p-k},\ldots,\frac{1-a}{p-k}}_{p-k} \Bigr)V^\top, \qquad \beta\sim\mathcal{N}\!\Bigl(0,\frac{r^2}{p}\Sigma_\beta\Bigr).\] Thus, a proportion \(a\) of the expected signal energy lies in the top \(k\) eigendirections. In 3, we set \(m=0.1\) and \(a=0.9\).

The empirical curves in 3 are averaged over 30 independent simulations.

17.3.2 Logistic regression↩︎

For the synthetic logistic regression experiments, we follow [19] and construct a block-diagonal kernel matrix \[K:=\operatorname{diag}(K_1,K_0)\in\mathbb{R}^{2n\times 2n},\] where the two blocks correspond to the ground-truth classes and have zero cross-class correlation. Each block has unit diagonal. In the constant feature-correlation setting, every off-diagonal entry equals \(c=0.25\). In the uniform feature-correlation setting, the off-diagonal entries are taken from \((1/n)ZZ^\top\), where the entries of \(Z\in\mathbb{R}^{n\times n}\) are drawn independently from the uniform distribution on \([0.3,0.7]\). The second block is constructed analogously. Both constructions yield a positive semidefinite kernel matrix.

We train kernel logistic regression in the dual parameterization. For pseudo-labels \(q=(q_1,\ldots,q_{2n})\in[0,1]^{2n}\) and dual variables \(\alpha\in\mathbb{R}^{2n}\), let \(f=K\alpha\) and apply the sigmoid componentwise to obtain \(\hat{p}=\sigma(f)\). We minimize the regularized binary cross-entropy objective \[\mathcal{L}_q(\alpha) \;=\; -\frac{1}{2n}\sum_{i=1}^{2n} \Bigl[ q_i\log \hat{p}_i + (1-q_i)\log(1-\hat{p}_i) \Bigr] +\frac{\lambda}{2}\,\alpha^\top K\alpha.\] For the first PD student, \(q_i=\hat{y}_i\) is the teacher’s hard pseudo-label; for the second PD student, \(q_i=y_{\pd}(x_i)\) is the first student’s soft prediction. This is the dual form of the training objectives in [eq:class_training_loss,eq:class_training_loss_2]. We minimize it using L-BFGS-B [68] with analytic gradients and initialize at \(\alpha=0\).

References↩︎

---

1. For brevity, we omit the dependence on \(\lambda_t\) in the notation for the PD student when the context is clear.↩︎