1 INTRODUCTION↩︎

Knowledge distillation (KD) [1], [2] transfers the predictive function of a large teacher model into a smaller student model by training the student on the teacher’s returned probability vector outputs. This very same mechanism that makes possible the compression of large models into smaller, efficient networks  [3]–[6], also enables model extraction wherein an adversary with black-box API access systematically queries a deployed classifier, collects the returned probability vectors, and trains a student model that replicates the teacher’s behavior at a fraction of the original cost [7], [8]. As recent events have highlighted, this threat is not just hypothetical. DeepSeek’s 2025 release demonstrated that near-frontier capability may be obtained largely through distillation from proprietary models [9]. Additionally, Anthropic recently disclosed an industrial-scale extraction campaign against Claude that produced over 16 million exchanges via approximately 24,000 fraudulent accounts [10]. Beyond direct intellectual-property loss [11], [12], extracted models facilitate downstream membership-inference [13], [14], evasion [15]–[17], and privacy attacks against the underlying training distribution [18]–[20], raising the stakes especially in regulated domains such as medicine, and finance where machine learning applications are gaining traction  [21], [22]. Beyond these settings, the implications of model extraction also extend to CPS as KD is increasingly used in resource-constrained robotic platforms [23]. Distillation-based extraction may also expose safety-critical control behavior in the physical world, raising security concerns for CPS deployments.

The attacker consumes the returned probability vector, also known as the soft labels, in contrast to hard labels, where the output contains only the index of the predicted class. The inter-class structure of these soft labels, i.e., the relative probabilities of the classes other than the top class, constitutes the dark knowledge of the given model [2]. This dark knowledge makes soft-label distillation markedly more effective than training on hard labels alone. A naive defense against such distillation attempts would be to always return uninformative output, but that would make the deployed model useless for regular users. The natural question is therefore how to corrupt the dark knowledge a model deployed through an API serves without destroying the utility of the given model for legitimate users. Every inference-time defense implicitly negotiates this trade-off, and we argue the trade-off rate should be measured against an explicit break-even: a defense that removes one pp of attacker accuracy per pp of teacher accuracy, a 1:1 trade-off, is no better than deploying a worse model. A useful defense must beat the 1:1 trade-off, ideally by a wide margin in favor of the model being defended.

ADS [24] is a compelling starting point. Proposed for autoregressive language models, ADS perturbs each next-token distribution with an input-dependent, gradient-directed penalty, computed from a hidden proxy student, that estimates how much serving each token would help a distilling student. Unlike static perturbation defenses, which apply a fixed transformation that an attacker can invert or meta-learn [25], [26], the ADS perturbation is recomputed per query from hidden state. Whether this idea transfers to classification, which is among the most broadly exposed extraction surfaces in practice, has remained open. The question is not merely one of engineering: in a classification API the distilling student ingests the entire served vector through its KL objective, whereas LLM sampling collapses the perturbation onto a single emitted token, so per unit of perturbation ADS should be more potent in classification, not less.

This paper adapts ADS to classification, analyzes the mechanism that governs its behavior there, and repairs its ineffectiveness with a composition rule whose utility cost is provably zero. Our contributions are as follows.

1. Adaptation. We instantiate the ADS penalty in the classification setting, preserving its derivation, and identify the structural differences from the LLM setting (Sec. 4).

2. Diagnosis. We show that the behavior of the transferred defense is determined by the distribution of per-input flip thresholds, whose cumulative distribution, before any attack or defense is run, accurately predicts the defended teacher’s accuracy cost (Sec. 5). Classifier overconfidence [27] renders the defense inert at low strengths and unfavorable at high strengths; the perturbation itself is faithful and correctly aimed at dark knowledge, but margin-blind.

3. Analysis of temperature softening. We show that softening the served distribution rescales the phase transition in closed form, and confirm the prediction in trained students: softening amplifies the perturbation at low strengths while collapsing the teacher at moderate ones, and no temperature configuration improves the trade-off (Sec. 6).

4. ADS-C. We compose the ADS perturbation under a closed-form, per-input, margin-aware budget: each input receives the largest penalty strength that provably keeps its served top-1 prediction unchanged (Proposition 1), so the defended teacher’s accuracy equals the undefended teacher’s identically. The surviving perturbation still degrades the distilled student substantially, by \(17.4\,\mathrm{pp}\) on CIFAR-100, \(29.6\,\mathrm{pp}\) on CIFAR-10, and \(13.3\,\mathrm{pp}\) on Tiny-ImageNet at zero utility cost (Secs. 7–8). Against the complementary hard-label attacker the defended soft output proves strictly worse as a distillation signal than the argmax it accompanies. The soft-label student falls \(17.1\,\mathrm{pp}\) and \(29.7\,\mathrm{pp}\) below what the hard-label attacker achieves through distillation (Sec. 8.5).

5. Relaxations of the guarantee. We introduce two mechanisms with closed-form-predictable utility cost: first, stochastic enforcement of the budget; second, a minimum served strength. Both trade a controlled amount of the guarantee for attacker uncertainty. We price each relaxation against the attacker’s best response between soft- and hard-label extraction, and report a negative result on margin-ranked sacrifice that further clarifies the damage mechanism (Sec. 9).

6. Evaluation methodology and reproducibility. We argue that once utility cost is zero, defenses should be compared at matched served fidelity rather than matched penalty strength, and show that the two comparisons can disagree (Sec. 8.4). We release a library and every experiment driver: https://github.com/the-kas-lab/ADS-C.

2 RELATED WORK↩︎

Distillation and dark knowledge. Distillation trains a student on a teacher’s output distributions [1], [2], [28], [29]; its advantage over hard-label training is attributed to the inter-class structure of those distributions [2], [30], [31]. Furlanello et al. [30] showed diagnostically that destroying non-target structure destroys most of distillation’s benefit, providing evidence that dark knowledge is the asset a defense should target.

Model extraction. Extraction attacks replicate a deployed model from query access [7], [32]–[34]; surveys [8], [35]–[37] document their practicality. We consider the distillation-based extraction attacker, who trains on returned soft labels which is the strongest and most common variant when probability vectors are exposed.

Extraction defenses. Training-time defenses harden the teacher’s weights [38]–[41] but require retraining and access to the original corpus, which is often undesirable or even infeasible at deployment scale. Detection and monitoring [42]–[44] flag anomalous query streams, but rely on out-of-distribution assumptions that prove futile against attackers with in-distribution data [41] and are circumvented by Sybil accounts [45], [46]. Watermarking [47]–[50] supports post-hoc attribution but erects no barrier against the extraction itself. Perturbation defenses are closest to our approach. These defenses modify the served probabilities: Reverse Sigmoid [25], prediction poisoning [51], ModelGuard [52], noise-transition learning [53], and information-theoretic output compression [54]. All of these pay for protection with utility, and the static members of the family being fixed functions of the input are invertible by meta-learned recovery attacks such as D-DAE [26]. None offers a per-query guarantee on the served top-1 prediction.

ADS. Antidistillation Sampling [24] is input-dependent and gradient-directed wherein a hidden proxy student defines, per query, a direction in output space that maximally harms a distilling student. It was derived and evaluated for LLM reasoning traces. Two questions are open: whether it transfers to classification, and what it costs the defender there. We answer both: the transfer’s behavior is gated entirely by classifier overconfidence, and the repair (a margin budget at composition) makes the cost exactly zero.

3 THREAT MODEL AND PROBLEM SETUP↩︎

Figure 1 summarizes the setting. A defender trains a classifier \(\mathcal{T}\colon\mathcal{X}\to\Delta^{K-1}\) mapping an input space \(\mathcal{X}\) to the probability simplex over \(K\) classes and deploys it behind a query API that returns, for each query \(\mathbf{x}\in\mathcal{X}\), a served probability vector \(\hat{\mathbf{p}}(\mathbf{x})\in\Delta^{K-1}\).

Adversary. The adversary holds (i) black-box access to the API with an arbitrary but bounded query budget; (ii) an unlabeled query corpus drawn from the same distribution \(\mathcal{D}\) over \(\mathcal{X}\) as the teacher’s training data—we make no out-of-distribution assumption, which disables detection-style defenses by construction; and (iii) a student architecture \(\mathcal{S}\) of its choosing. The adversary lacks the teacher’s parameters, and ground-truth labels. With no labels, the attacker’s objective reduces from the standard hard-plus-soft KD loss to pure soft-label matching, \[\label{eq:ukd} \mathcal{L}_{\mathrm{UKD}} \;=\; D_{KL}\bigl(\hat{\mathbf{p}}(\mathbf{x})\,\big\|\,\mathbf{p}_\mathcal{S}(\mathbf{x})\bigr),\tag{1}\] evaluated at temperature \(1\): the served vector is the attacker’s sole supervisory signal, and hence the natural locus of defense.

Why the soft-label attacker. We scope the threat to the dark-knowledge distiller, and this scoping is principled rather than convenient. The attacker’s advantage over training on public hard labels is the served soft-label structure; an attacker who reads only the served argmax gains nothing that a labeling function of equal top-1 accuracy would not provide. Our defense will, by construction, never alter the served argmax (Sec. 7), so hard-label distillation against the defended API yields exactly what it would yield against the undefended API (verified empirically in Sec. 8.5): the defense removes the soft-label advantage. The exploitation of the guarantee’s determinism itself is treated in Sec. 9.

Defender. The defender owns the teacher, a modest labeled holdout set, and a compact proxy student \(\mathcal{P}\) that stands in for the unknown attacker student; it cannot retrain the teacher and must act at inference time only. The defender’s goals are: (a) utility: legitimate users, who consume the top-1 prediction, must be unharmed; (b) distillation resistance: a student trained via 1 on the served outputs should be substantially degraded.

Metrics. (A summary of notation is provided in Appendix 11.1.) We report three quantities. Teacher top-1: accuracy of the served argmax. Student top-1: accuracy of the distilled student. Served fidelity: \(D_{KL}(\mathbf{p}_\mathcal{T}\,\|\, \hat{\mathbf{p}})\) averaged over the evaluation set, the total distributional perturbation the defense spends; this axis quantifies the defense’s “corruption budget” and becomes essential once utility cost is zero (Sec. 8.4).

4 ANTIDISTILLATION SAMPLING FOR CLASSIFICATION↩︎

4.1 Preliminaries↩︎

A pretrained teacher with parameters \(\theta_\mathcal{T}\) produces logits \(\mathbf{z}_\mathcal{T}=\mathcal{T}(\mathbf{x};\theta_\mathcal{T})\in\mathbb{R}^{K}\); the softmax \(\sigma(\mathbf{z})_i = \exp(z_i)/\sum_j \exp(z_j)\) yields probabilities \(\mathbf{p}_\mathcal{T}\). Only logit differences matter: adding a constant to \(\mathbf{z}\) leaves \(\sigma(\mathbf{z})\) unchanged, so the margin \(z_t - z_j\) between the top class \(t\) and a rival \(j\) is the invariant quantity that decides the argmax. Distillation trains a student to minimize 1 against the served distribution.

4.2 The ADS penalty↩︎

ADS asks, per query and per class: if the student came to believe class \(y\) is slightly more likely on this input, would that make the student worse? Formally, let \(\ell(\theta_\mathcal{P})\) be a downstream loss of the proxy on the defender’s labeled holdout. We take \(\ell(\theta_\mathcal{P}) = \frac{1}{N}\sum_{i=1}^{N}D_{KL}\bigl(\mathbf{p}_\mathcal{T}(\mathbf{x}_i)\,\|\,\mathbf{p}_\mathcal{P}(\mathbf{x}_i)\bigr)\), the proxy’s mean disagreement with the teacher over the holdout, following [24], and let \(g = \nabla\ell(\theta_\mathcal{P})\) be its gradient. One SGD step of distillation on class \(y\) moves \(\theta_\mathcal{P}\) along \(\nabla_{\theta}\log p(y\mid\mathbf{x};\theta_\mathcal{P})\); the resulting change in \(\ell\) is, to first order, the inner product of that update with \(\nabla\ell\). Exactly as in the LLM derivation, symmetry of the directional derivative converts this into a finite difference over two frozen clones of the proxy, \(\theta_{\mathcal{P}\pm} = \theta_\mathcal{P}\pm \varepsilon g\): \[\label{eq:fd} \widehat{\Delta}(y\mid\mathbf{x}) \;=\; \frac{\log p(y\mid\mathbf{x};\theta_{\mathcal{P}+}) - \log p(y\mid\mathbf{x};\theta_{\mathcal{P}-})}{2\varepsilon}.\tag{2}\] A positive \(\widehat{\Delta}(y\mid\mathbf{x})\) means that serving mass on class \(y\) raises the proxy’s holdout loss, which is good for the defender. The defended distribution adds the penalty to the teacher’s log-probabilities: \[\label{eq:ads95compose} \hat{\mathbf{p}}(\cdot\mid\mathbf{x}) \;=\; \sigma\bigl(\log\mathbf{p}_\mathcal{T}(\cdot\mid\mathbf{x}) + \lambda\,\widehat{\Delta}(\cdot\mid\mathbf{x})\bigr),\tag{3}\] with \(\lambda \ge 0\) the defense strength; \(\lambda=0\) recovers the undefended API.

Parameter choices. (i) Defining \(\ell\) as a per-sample mean fixes the scale of \(g\) independently of the holdout size; empirically \(\lVert g\rVert\) is of order one (\(1.00\) on CIFAR-100, \(0.77\) on CIFAR-10, \(1.37\) on Tiny-ImageNet), so \(\lambda\) operates on a comparable scale across datasets. We sweep \(\lambda\in[0,3]\), extended to \(\lambda=10\) for unmodified ADS to map the saturation regime (Sec. 5.4). (ii) The step \(\varepsilon\) is calibrated, not tuned: we select \(\varepsilon\) to minimize the discrepancy between 2 and the exact Jacobian–vector product it approximates, obtaining \(\varepsilon=3\times10^{-3}\) on both CIFAR datasets and \(\varepsilon=10^{-2}\) on Tiny-ImageNet, each with a magnitude ratio of \(\approx\!1.00\) (Appendix 11.2); the finite difference is a faithful estimate, which will matter for the diagnosis of Sec. 5. (iii) The proxy (ResNet-20 on the CIFAR datasets, ResNet-18 on Tiny-ImageNet) is substantially smaller than the teacher and is trained on the teacher’s training split, with a disjoint \(10\%\) held out for computing \(g\); the two clones are built once, offline, and frozen, so a defended query costs two small forward passes and no gradient computation. (iv) The margin floor \(m\) introduced in Sec. 7 is analyzed in Appendix 11.3.

4.3 ADS Case for Classification↩︎

The penalty 2 is structurally identical to the next-token formulation of [24]; what differs is downstream consumption. In autoregressive generation the perturbation tilts the next-token distribution, but the attacker observes only a single sampled token per step: a one-hot realization whose expectation carries the tilted distribution, so the signal on the non-sampled coordinates of \(\widehat{\Delta}\) reaches the student only in expectation, across many draws. In classification distillation, the attacker observes the served vector itself: the KL objective 1 ingests every coordinate of \(\widehat{\Delta}\) exactly, on every query. The same perturbation is thus delivered through a noiseless channel rather than estimated through a stochastic one, and per unit of perturbation ADS should be more potent in classification, not less. The next section shows that, despite this, the direct transfer is inert over a wide regime, then trades at an unfavorable rate, and finally saturates with both parties near their accuracy floors. In addition, it locates the cause of this behavior not in the penalty but in its composition with the teacher.

5 DIAGNOSIS OF ADS INEFFECTIVENESS↩︎

Swept over \(\lambda\), the unmodified ADS exhibits three behaviors in sequence that make it undesirable for deployment in the classification domain. For \(\lambda \lesssim 0.2\) it is inert: on both CIFAR datasets, student and teacher accuracy each change by at most \(1\)–\(2\,\mathrm{pp}\), within seed noise. The defense measurably affects neither party. Beyond that, it becomes active at an unfavorable rate: on CIFAR-100 at \(\lambda=0.5\), the distilled student drops \(5.9\,\mathrm{pp}\) while the teacher drops \(12.2\,\mathrm{pp}\), a trade-off rate of \(0.48\), half of break-even; on CIFAR-10 the rate is \(0.85\) (\(9.6\,\mathrm{pp}\) student for \(11.3\,\mathrm{pp}\) teacher). Finally, for \(\lambda \gtrsim 2\) the sweep saturates: the trade-off remains sub-linear while both models degrade toward their floors, and by \(\lambda=3\) the defended teacher’s accuracy has fallen to the level of the very student it defends against (Fig. 10, diagonal curves). This section explains all three behaviors through a single mechanism and derives the design requirement for the remedy.

5.1 Overconfidence of Classifiers↩︎

Modern, well-trained networks are systematically overconfident [27], [55]: their top-class probability \(p_{\max}\) concentrates near 1. Table 1 quantifies this for our teachers. On CIFAR-10 the median served \(p_{\max}\) exceeds \(0.99999\). The median non-target mass, the entire surface on which dark knowledge lives and on which an additive penalty can visibly act, is below \(10^{-6}\). CIFAR-100 is less extreme (median non-target mass \(\approx 6\times10^{-3}\)), and Tiny-ImageNet sits between the two (\(4\times10^{-4}\)) despite having twice CIFAR-100’s classes. The comparison is deliberate: the Tiny-ImageNet teacher is a strong ImageNet-pretrained ResNet-50 fine-tune, whereas the CIFAR teachers are trained from scratch, and the pretrained model’s strength overrides the intuition that more classes produce more diffuse outputs. We find that overconfidence is determined by model quality, not class count, and strong pretrained models are precisely what production APIs deploy, so the overconfident regime studied here is the realistic one rather than a small-benchmark artifact. (At the extreme of output dimension, LLMs over \(>\)​50k-token vocabularies operate in a naturally diffuse regime which is one reason the original ADS formulation could presuppose sufficient non-target mass.)

The overconfidence is also heterogeneous, and this is the central observation. The top-logit margin \(z_t - z_{(2)}\) (top minus runner-up) spans two orders of magnitude within a single dataset: on CIFAR-100 its 10th/50th/90th percentiles are \(0.7\)/\(5.5\)/\(15.0\) log-units; on Tiny-ImageNet, \(1.1\)/\(8.3\)/\(17.0\); on CIFAR-10, \(3.9\)/\(14.4\)/\(21.0\). Most inputs have margins far too large for a moderate output perturbation to overcome, while a persistent minority — \(13.4\%\), \(9.4\%\), and \(2.4\%\) of inputs, respectively, have margins below one log-unit—lie close to a prediction flip. Any defense that treats all inputs identically will therefore be simultaneously too weak on the head of this distribution and too strong on its tail. To our knowledge, no prior antidistillation work has examined how this confidence structure interacts with an output-perturbing defense; the remainder of this section shows the interaction is decisive.

5.2 Soundness of ADS Penalty↩︎

Before locating the futility at composition we rule out the penalty. Two verifications (Fig. 3):

Faithfulness. At the calibrated \(\varepsilon\), the finite-difference \(\widehat{\Delta}\) matches the exact directional derivative with magnitude ratio \(\approx\!1.00\) on both datasets (Appendix 11.2). The poison is not a numerical artifact.

Aim. \(\widehat{\Delta}\) concentrates exactly where dark knowledge lives. Averaged over inputs, its value at the teacher’s top class is near zero (mean \(-1.1\) on CIFAR-100, \(-0.4\) on CIFAR-10) while deep non-target ranks receive large positive pushes (up to \(+5.6\) and \(+12.3\) respectively); the class each input’s poison boosts hardest sits at teacher rank \(\ge 5\) for \(98\%\) of CIFAR-100 and \(63\%\) of CIFAR-10 inputs. ADS corrupts the inter-class structure beneath the prediction.

5.3 Margin Blindness↩︎

Whether the served argmax survives 3 is a per-input race between two numbers. For each rival \(j\neq t\), let \(c_j = \widehat{\Delta}_j - \widehat{\Delta}_t\) be the rate (per unit \(\lambda\)) at which the poison closes the margin to \(j\); we define the poison’s contrast on input \(\mathbf{x}\) as the largest such rate, \[\label{eq:contrast} c(\mathbf{x}) \;=\; \max_{j\neq t}\; c_j,\tag{4}\] the rate at which the strongest rival gains on the top class \(t\). The served top-1 flips exactly when \(\lambda\) exceeds the flip threshold \[\label{eq:flip} \lambda^\ast(\mathbf{x}) \;=\; \min_{j\neq t:\,c_j>0}\;\frac{z_t - z_j}{c_j},\tag{5}\] with \(\lambda^\ast(\mathbf{x})=+\infty\) when no rival satisfies \(c_j>0\). Two measurements now explain the unfavorable exchange. First, the contrast is large: its median is \(17.4\) (CIFAR-100) and \(16.3\) (CIFAR-10) log-units per unit \(\lambda\)—comparable to the 90th-percentile margin and \(25\times\) the 10th-percentile margin. Second, and critically, the contrast is margin-blind: its Spearman correlation with the margin is \(\rho = +0.12\) (CIFAR-100) and \(+0.18\) (CIFAR-10). The poison’s per-input strength is unrelated to the margin it must respect. Nothing in the objective \(\ell\) ever sees the teacher’s margins, so nothing in \(\widehat{\Delta}\) respects them.

The consequence follows directly from 5 and the thin tail of Fig. 2: as \(\lambda\) grows, served predictions flip in order of their margins, beginning with the thin-margin inputs. At \(\lambda=0.5\), a global penalty has already flipped \(26.1\%\) of CIFAR-100 and \(14.8\%\) of CIFAR-10 served predictions (Fig. 4). These flips constitute the entirety of the teacher’s cost, and they are an inefficient attack, for a reason the two parties’ metrics make precise. The teacher’s cost is discontinuous in the perturbation: the moment the top two entries cross, a flip registers as a full served error. The student’s supervision is the served vector, which is continuous: a flip just past its threshold crosses at near-parity, so the training signal on that row is almost unchanged. Moreover, low-\(\lambda\) flips land exclusively on thin-margin inputs, whose near-uniform outputs carry little of the inter-class structure a distilling student exploits (Sec. 9.2 confirms this independently: corruption concentrated on thin-margin rows barely moves the student). The defender thus pays full price for perturbations the student scarcely notices—at \(\lambda=0.5\) on CIFAR-100 the flip component costs the teacher \(12.2\,\mathrm{pp}\) while accounting for only \(3.1\,\mathrm{pp}\) of student damage—whereas the argmax-safe corruption underneath is what a distilling student cannot discount; we quantify this decomposition in Sec. 8.3.

5.4 The inert window and the phase transition↩︎

In addition to explaining the flips, Equations 4 –5 make the entire teacher-cost curve predictable in closed form, before any student is trained. Let \[\label{eq:flipcdf} F(\lambda) \;=\; \Pr\bigl[\lambda^\ast(\mathbf{x}) \le \lambda\bigr]\tag{6}\] be the cumulative distribution of the flip threshold over the evaluation set, computable from the teacher’s and the proxy’s outputs in a single forward pass. The defended teacher’s accuracy drop under unmodified ADS is then the fraction of correctly classified inputs whose threshold has been crossed. This prediction tracks the measured, trained teacher drop to within \({\approx}1\)–\(2\,\mathrm{pp}\) at every \(\lambda\) on both datasets (CIFAR-100 at \(\lambda=1\): predicted \(34.9\) vs.measured \(33.3\,\mathrm{pp}\); CIFAR-10 at \(\lambda=2\): \(69.8\) vs.\(69.0\,\mathrm{pp}\)). The teacher cost of unmodified ADS is thus fully determined by the flip-threshold distribution.

\(F\) inherits its shape from the margin heterogeneity of Sec. 5.1, and that shape organizes the sweep into three regimes (Fig. 5):

• Inert window (\(\lambda\) below the 5th-percentile flip threshold: \(\lambda<0.12\) on CIFAR-100, \(\lambda<0.30\) on CIFAR-10). \(F\approx0\), and—because the same overconfidence leaves almost no non-target mass for the poison to reweight—the student is unaffected as well: measured student and teacher effects are both within \(1\)–\(2\,\mathrm{pp}\) of the undefended anchors, indistinguishable from seed noise. In this regime the defense has no measurable effect on either party.

• Phase transition (\(0.35\lesssim\lambda\lesssim1\)). \(F\) rises through its steep section; teacher cost and student damage both grow from noise level to tens of pp within about \(1.5\) octaves of \(\lambda\)—a narrow band of a control parameter that we sweep over four orders of magnitude. We define the transition threshold \(\lambda_c\) as the largest \(\lambda\) with \(F(\lambda)\le1\%\) (\(\lambda_c\approx0.02\) on CIFAR-100, \(0.09\) on CIFAR-10 by this strict definition; the teacher-cost onset—drop exceeding \(1\,\mathrm{pp}\)—lies at \(0.066\) and \(0.135\), respectively). \(\lambda^\ast\) is approximately log-normal, so \(F\) is a smooth sigmoid in \(\log\lambda\) whose inflection lies at the median flip threshold (\(0.91\)/\(1.09\)).

• Saturation (\(\lambda\gtrsim2\)). Both parties approach their floors and the sweep exhausts itself. Per-unit-\(\lambda\) student damage falls by two orders of magnitude (\(20.3\,\mathrm{pp}\) near \(\lambda=1\) to \(0.2\,\mathrm{pp}\) at \(\lambda=10\) on CIFAR-100; \(33.6\) to \(0.3\,\mathrm{pp}\) on CIFAR-10) as the student completes its fall to—and slightly through—the chance floor (\(0.6\%\) vs.\(1\%\) chance; \(6.3\%\) vs.\(10\%\)): deeply poisoned vectors are systematically wrong, not merely uninformative. The teacher’s tail is equally exhausted (\(64.0\to70.2\,\mathrm{pp}\) and \(77.6\to86.7\,\mathrm{pp}\) of drop over \(\lambda=3\to10\)), and 6 predicts every measured tail point to within \(0.6\,\mathrm{pp}\). By \(\lambda=3\) the defended teacher’s accuracy has fallen to the level of the very student it defends against (\(7.2\%\) vs.\(4.2\%\); \(15.1\%\) vs.\(12.9\%\)), and by \(\lambda=10\) on CIFAR-10 below it (\(6.0\%\) served vs.\(6.3\%\) distilled): the direct transfer does not merely trade unfavorably in this regime—it eliminates, and finally inverts, the accuracy advantage it exists to protect.

Note the ordering: CIFAR-10, the more overconfident teacher, has the wider inert window showing that inertness tracks overconfidence, as the mechanism predicts. The same computation places Tiny-ImageNet’s transition threshold at \(\lambda_c\approx0.06\), between the two CIFAR values, consistent with its intermediate overconfidence (Table 1); and its closed-form \(F(\lambda)\) predicts the measured teacher cost of the unmodified sweep to within \(2\,\mathrm{pp}\) at every \(\lambda\) (e.g., \(33.2\,\mathrm{pp}\) predicted versus \(31.4\,\mathrm{pp}\) measured at \(\lambda=2\)), validating the diagnosis on a third teacher whose confidence arises from ImageNet pretraining rather than from a narrow benchmark.

One further property of 6 is used in the next section. Serving the teacher at temperature \(\tau\) divides every logit gap by \(\tau\) while leaving the poison contrast unchanged, so every flip threshold is divided by exactly \(\tau\). A threshold of \(\lambda^\ast/\tau\) is crossed by a given \(\lambda\) precisely when \(\lambda^\ast\) is crossed by \(\tau\lambda\), so the flipped fraction at strength \(\lambda\) under temperature \(\tau\) equals the unsoftened fraction at \(\tau\lambda\): \[\label{eq:ftau} F_\tau(\lambda) \;=\; F(\tau\lambda),\tag{7}\] a parameter-free prediction (Fig. 6): temperature shifts the entire transition curve toward smaller \(\lambda\) by the factor \(1/\tau\). Softening does not alter the mechanism; it rescales the \(\lambda\) axis on which the mechanism operates.

The design requirement is now explicit. The penalty direction is sound; what is missing is margin-awareness, and margin-awareness cannot be injected through the poison pipeline. The one free choice inside that pipeline is where the proxy sits, and we tested it: even a proxy distilled directly from the teacher’s served probabilities, corresponding to the state a distilling attacker’s student would be in during distillation rather than a proxy trained conventionally on labeled data, produces a poison whose contrast–margin correlation remains \(\rho \approx 0\) (CIFAR-100). No placement of the proxy makes \(\widehat{\Delta}\) margin-aware, because the objective \(\ell\) it derives from never sees a margin. Margin-awareness must therefore be enforced at composition, the only point where the margin and the poison are simultaneously visible. That is ADS-C (Sec. 7). First, however, we examine the natural alternative, whose failure the theory above already predicts and whose failure mode is itself informative.

6 TEMPERATURE SOFTENING↩︎

Table 1 suggests an obvious remedy, that is, if overconfidence starves ADS of non-target mass, soften the served distribution, with a constant distillation temperature [2], [27] or a confidence-conditioned variant [56]–[58], to create dark knowledge for the poison to corrupt. This intuition is partially correct, but what it yields is a demonstration of the mechanism rather than a viable defense. Softening relaxes both of the constraints that teacher confidence imposes at once. First, it inflates the non-target mass visible to the attacker’s KL loss (activating the poison), and secondly, it shrinks every served margin by the same factor (activating the flips, per 7 ). We serve \(\sigma(\mathbf{z}/T)\) for \(T\in\{2,4\}\), alone and composed with the raw ADS penalty, over \(\lambda\in[0,0.5]\), on both datasets with three seeds, extending the poison-free sweep on CIFAR-10 to \(T\in\{6,8,10,12\}\). (Throughout, \(\tau\) in 7 and \(T\) denote the same served temperature; we write \(T\) when quoting experimental settings.) We report four findings.

6.1 Softening alone can improve the attacker’s student↩︎

Softened teacher outputs are the classic recipe for knowledge distillation [2]. However, whether softening assists the adversary is governed by how much dark knowledge the teacher has to reveal.

On CIFAR-100, with no poison at all (\(\lambda=0\)), serving at \(T=4\) (median served confidence \(0.42\)) improves the distilled student by \(+4.7\,\mathrm{pp}\) over the undefended baseline (\(T=2\): \(+0.9\,\mathrm{pp}\)). On CIFAR-10 the benefit is absent: pushing to higher temperatures only degrades the student, monotonically, from \(-1.0\,\mathrm{pp}\) at \(T=4\) to \(-6.5\,\mathrm{pp}\) at \(T=12\). At \(T=10\), CIFAR-10 reproduces CIFAR-100’s \(T=4\) operating point (served median confidence \(0.42\) on both), yet CIFAR-100 gains \(+4.7\,\mathrm{pp}\) where CIFAR-10 loses \(-5.2\,\mathrm{pp}\). Tiny-ImageNet interpolates between the two exactly as this account predicts: its non-target mass is intermediate (Table 1), and so is the benefit gained by temperature scaling, that is, \(+1.6\,\mathrm{pp}\) at \(T=4\), which lies between CIFAR-10’s absent benefit and CIFAR-100’s \(+4.7\,\mathrm{pp}\). At matched served confidence the sign of the effect is therefore set by how much usable non-target structure the teacher has to reveal, not by the magnitude of the temperature parameter or by the class count. Wherever dark knowledge exists, revealing it assists the adversary; where it does not, softening only dilutes the supervision the attacker already had. Note that constant-\(T\) scaling preserves the argmax, so this assistance is not even purchased with teacher accuracy: its only cost is served fidelity, which is the axis on which Sec. 6.4 prices every softening variant.

6.2 Softening activates ADS inside the inert window↩︎

Composed with the poison, softening behaves exactly as 7 predicts: it shifts the entire transition curve into the formerly inert window. At \(\lambda=0.2\)—inert under plain serving—the poison’s marginal effect on the student (measured relative to the same serving at \(\lambda=0\), which nets out the shift of Sec. 6.1) grows from \(-1.1\,\mathrm{pp}\) (unsoftened) to \(-3.0\,\mathrm{pp}\) (\(T=2\)) to \(-17.8\,\mathrm{pp}\) (\(T=4\)) on CIFAR-100, and from \(-0.4\) to \(-5.8\) to \(-27.7\,\mathrm{pp}\) on CIFAR-10 (Fig. 7a): the same small \(\lambda\) becomes \(17\times\)–\(69\times\) more potent. This provides direct causal evidence that the availability of non-target mass, not the penalty strength, is what limits the poison at low \(\lambda\). CIFAR-10 sharpens the point: the same \(T=4\) serving that conferred no benefit on a clean student (Sec. 6.1) multiplies the poison’s potency \(69\times\)—what softening manufactures is probability mass for the poison to reweight, whether or not that mass encodes anything a student could exploit. The teacher-side cost agrees with the theory quantitatively: the pre-registered prediction \(F(\tau\lambda)\) placed the \(T=4\), \(\lambda=0.2\) teacher cost at \(26.9\,\mathrm{pp}\) (CIFAR-100) and \(30.1\,\mathrm{pp}\) (CIFAR-10); the trained runs measured \(25.1\) and \(29.1\,\mathrm{pp}\).

6.3 The same rescaling collapses the teacher↩︎

The teacher-side cost just quoted is the other half of the same mechanism: softening activates the poison by shrinking the very margins the poison then crosses, so enablement and collapse are one phenomenon, not two. On the student-versus-teacher-cost plane (Fig. 7b), every constant-temperature variant (\(T=1\), \(2\), \(4\), on both datasets) collapses onto a single trade-off curve. Temperature never improves the exchange rate rather it only moves the operating point further along it. A worked example makes the severity concrete. A typical softened CIFAR-100 row has served margin \(0.38\) against poison contrast \(16.8\), so its prediction flips at \(\lambda^\ast \approx 0.022\); measured end-to-end, \(T=4\) serving loses \(25.1\)/\(29.1\,\mathrm{pp}\) of teacher accuracy (CIFAR-100/CIFAR-10) at \(\lambda=0.2\), a strength that is inert under plain serving, and \(45.0\)/\(56.3\,\mathrm{pp}\) at \(\lambda=0.35\) (Fig. 7). Softening manufactures precisely the thin margins on which a margin-blind poison is most destructive because the two mechanisms compete for the same resource, since the softener consumes the margin the poison needs in order to act safely.

6.4 At matched served fidelity, softening is dominated↩︎

Constant temperature is the bluntest possible softener, so its failure does not by itself close the question. We therefore construct the strongest softening variant we could: a gated operator, inspired by [59], that lowers the confidence of over-confident inputs only, and does so by lowering the top logit alone, leaving all non-target structure intact for the dark-knowledge channel (Appendix 11.4). It is composed with the margin budget of Sec. 7, so that its teacher cost is identically zero. Section 8.4 evaluates this combination across five gates and the full \(\lambda\) grid against poison-only ADS-C on the axis a defender actually spends, served fidelity: the softener pays a large fidelity cost up front, before any poison flows, and at every matched fidelity level in the operating regime poison-only ADS-C damages the student more (Fig. 12). We therefore include no softening in the proposed defense: softening serves in this paper as the probe that establishes dark knowledge as the resource the poison consumes, and as a defense component it is dominated.

7 ADS-C↩︎

The diagnosis of Sec. 5 dictates the design requirement: retain the poison, but enforce margin-awareness at composition, per input. ADS-C implements this requirement in closed form.

7.1 The margin budget↩︎

Fix an input \(\mathbf{x}\) with teacher logits \(\mathbf{z}\), top class \(t=\arg\max_k z_k\), and poison \(\widehat{\Delta}\). For every rival \(j\neq t\), let \(\mathrm{gap}_j = z_t - z_j\) denote the served margin to that rival and recall from 4 the rate \(c_j = \widehat{\Delta}_j - \widehat{\Delta}_t\) at which the poison closes it. Given the requested strength \(\lambda\) and a margin floor \(m>0\), the per-input budget is \[\label{eq:budget} \lambda(\mathbf{x}) \;=\; \operatorname{clip}\Bigl(\;\min_{j:\,c_j>0}\;\frac{\mathrm{gap}_j - m}{c_j}\;,\;0,\;\lambda\Bigr),\tag{8}\] with the convention that a minimum over an empty set is \(+\infty\), and the API serves \(\hat{\mathbf{p}} = \sigma\bigl(\log \mathbf{p}_\mathcal{T}+ \lambda(\mathbf{x})\,\widehat{\Delta}\bigr)\) (Algorithm 8). The budget is the flip threshold 5 computed with \(m\) log-units of every margin held in reserve, clipped to the feasible range, and its behavior at the extremes is the intended one: rivals the poison pushes down (\(c_j\le0\)) never bind, and an input none of whose rivals gain passes the full requested strength. Conversely, an input whose margin is already at or below the floor receives no poison rather than a flip. Figure 9 traces one query through the full pipeline.

7.2 The guarantee↩︎

7.3 Design discussion↩︎

The floor \(m\). The floor is the service level the defender guarantees on served margins: every served prediction retains at least \(\min(m,\text{original margin})\) log-units of separation. We use \(m=0.05\) throughout; the guarantee itself is \(m\)-independent (teacher drop is zero for every \(m\)), and Appendix 11.3 shows the poison passed through the budget is insensitive over \(m\in[0.01,0.2]\) (served poison-KL varies by \({\approx}10\%\) across the range).

Overconfidence cuts both ways. The same heterogeneity that made the global penalty fatal makes the budget cheap: most inputs sit far from the floor, so most of the requested strength survives. At \(\lambda=0.5\) the mean effective strength is \(0.44\) (CIFAR-100) and \(0.47\) (CIFAR-10), corresponding to \(88\)–\(95\%\) of the request, while the thin-margin tail, which a global \(\lambda\) would flip first, is individually protected. The budget spends the poison precisely where the confidence structure of Fig. 2 has room for it.

Deployment properties. ADS-C is inference-time only. Per query, beyond the teacher’s forward pass, it costs two forward passes through the frozen proxy clones, each substantially smaller than the teacher, and \(O(K)\) arithmetic for 8 . The served output remains a valid probability vector with unchanged argmax. Like ADS, and unlike static perturbation defenses [25], [26], the perturbation is input-dependent and computed from state the attacker cannot observe (the proxy and its gradient \(g\)), so there is no fixed transformation to invert. Additionally, unlike ADS, its utility cost is identically zero.

8 EXPERIMENTS↩︎

8.1 Setup↩︎

Benchmarks. CIFAR-100 (teacher ResNet-110 \(\to\) student ResNet-20), CIFAR-10 (ResNet-56 \(\to\) ResNet-20), and Tiny-ImageNet (\(K=200\), \(64\times64\); teacher ResNet-50, ImageNet-pretrained and fine-tuned with a small-input stem \(\to\) student ResNet-18). The proxy always matches the student architecture; Appendix 11.5 reports training recipes. Undefended anchors: CIFAR-100 teacher \(71.23\%\), distilled student \(47.34\pm0.37\%\); CIFAR-10 teacher \(92.70\%\), student \(81.44\pm0.43\%\); Tiny-ImageNet teacher \(80.07\%\), student \(47.84\pm0.59\%\).

Attacker. The fixed distillation attacker of Sec. 3: it queries the API once per image with the full unlabeled training split (\(50\,000\) queries on the CIFAR datasets, \(100\,000\) on Tiny-ImageNet), unaugmented and at native resolution, then trains the student for 50 epochs on the returned (image, served-vector) pairs with the pure soft-label loss 1 at temperature 1 (full optimizer settings in Appendix 11.5). Every configuration is run over three seeds (\(\{92, 786, 1337\}\)); we report mean \(\pm\) std.

Defense grid. \(\lambda \in \{0, 0.1, 0.2, 0.35, 0.5, 0.75, 1, 1.5, 2, 3\}\); margin floor \(m=0.05\); \(\varepsilon = 3\times10^{-3}\) (CIFAR) and \(10^{-2}\) (Tiny-ImageNet; Appendix 11.2). ADS denotes the unmodified transfer 3 ; ADS-C denotes Algorithm 8; ADS-C + softener denotes the ablation of Sec. 8.4; the guarantee relaxations \(q\) and \(\lambda_{\min}\) (Sec. 9) default to \(q=1\), \(\lambda_{\min}=0\).

8.2 Main result: the trade-off rate before and after budgeting↩︎

Figure 10 plots student accuracy against teacher accuracy for both methods on all three benchmarks; Table 2 reports representative operating points.

Unmodified ADS. The direct transfer degrades the student only by degrading the teacher comparably or faster. On CIFAR-100 the sweep passes through \((-12.2\,\mathrm{pp}\text{ teacher}, -5.9\,\mathrm{pp}\text{ student})\) at \(\lambda=0.5\) and ends at \((-64.0\,\mathrm{pp}, -43.1\,\mathrm{pp})\) at \(\lambda=3\). The teacher suffers more than the student. Tiny-ImageNet exhibits the same shape shifted by its wider inert window (Sec. 5.4). Its exchange is still near-inert at \(\lambda=0.5\) (\(-1.7\,\mathrm{pp}\) teacher, \(0.0\,\mathrm{pp}\) student), first becomes active at \(\lambda=1\) (\(-9.0\,\mathrm{pp}\) teacher for \(-4.6\,\mathrm{pp}\) student, rate \(0.51\)), and reaches \((-46.7\,\mathrm{pp}, -28.8\,\mathrm{pp})\) at \(\lambda=3\).

ADS-C. With the budget active, teacher top-1 drop is \(0.00\,\mathrm{pp}\) at every cell, while the student falls monotonically with \(\lambda\). It falls to \(29.95\pm0.23\%\) on CIFAR-100 (\(-17.4\,\mathrm{pp}\), \(37\%\) relative), \(51.80\pm0.38\%\) on CIFAR-10 (\(-29.6\,\mathrm{pp}\), \(36\%\) relative), and \(34.51\pm0.28\%\) on Tiny-ImageNet (\(-13.3\,\mathrm{pp}\), \(28\%\) relative) at \(\lambda=3\).

Matched-damage comparison. For unmodified ADS to inflict the student damage that ADS-C delivers at zero cost, it must spend \(27.5\,\mathrm{pp}\) (CIFAR-100), \(32.9\,\mathrm{pp}\) (CIFAR-10), and \(22.2\,\mathrm{pp}\) (Tiny-ImageNet) of teacher accuracy. The margin budget converts an unfavorable trade into a free one.

8.3 Decomposition of the unbudgeted poison’s damage↩︎

The budget also functions as an analytical instrument. It separates the unbudgeted poison’s student damage into an argmax-safe component, which survives the budget, and flip collateral, which the budget removes (Fig. 11). At matched \(\lambda=0.5\) on CIFAR-100, the unbudgeted poison’s \(5.9\,\mathrm{pp}\) student drop decomposes into \(2.8\,\mathrm{pp}\) of argmax-safe corruption (retained at zero teacher cost) and \(3.1\,\mathrm{pp}\) of flip collateral (purchased at \(12.2\,\mathrm{pp}\) of teacher accuracy); on CIFAR-10, \(5.0\,\mathrm{pp}\) of \(9.6\,\mathrm{pp}\) survives (\(52\%\)). At smaller strengths the argmax-safe share is larger still (\(87\%\) at \(\lambda=0.2\) on CIFAR-100). Two conclusions follow. First, roughly half of the student damage that unmodified ADS purchased with teacher accuracy was genuine dark-knowledge corruption that required no payment at all. Second, because the cost ceiling is removed, ADS-C can increase \(\lambda\) far past the point where unmodified ADS becomes unusable, more than recovering the remainder: its zero-cost \(-17.4\,\mathrm{pp}\) (CIFAR-100) / \(-29.6\,\mathrm{pp}\) (CIFAR-10) at \(\lambda=3\) exceeds the unbudgeted poison’s total damage at any teacher cost below \(27\)–\(33\,\mathrm{pp}\). This decomposition is, to our knowledge, the first quantitative account of how ADS corrupts dark knowledge in classification: the durable damage is corruption of non-target structure, not label noise. This conclusion is further corroborated by Sec. 9.2 from an independent direction.

8.4 Ablation: softening front-ends at matched served fidelity↩︎

Finally we place the strongest softening variant, namely the gated top-logit reduction of Sec. 6.4, composed with the ADS-C against ADS-C with no temperature scaling. We sweep the softener’s gate over five settings and \(\lambda\) over the full grid (\(360\) runs total across both datasets; teacher drop \(\equiv 0\) everywhere, as guaranteed).

At matched \(\lambda\) the comparison is misleading as the softener is stronger at low \(\lambda\) (by up to \(6.3\,\mathrm{pp}\) on CIFAR-100 and \(22.2\,\mathrm{pp}\) on CIFAR-10) and weaker at high \(\lambda\). But \(\lambda\) is free. The axes on which a defender actually pays are teacher drop (identically zero for both variants) and served fidelity. Recomputed on the fidelity axis (Fig. 12), the comparison inverts as the softener spends a large fidelity budget up front (\(D_{KL}= 0.49\) at \(\lambda=0\) for its most aggressive gate) before any poison flows, while ADS-C spends fidelity only as \(\lambda\) grows. At matched served-KL, ADS-C without any softening damages the student more everywhere in the operating regime. Its advantage widens with fidelity spend, from \(+1.0\,\mathrm{pp}\) at KL \(0.5\) to \(+3.8\,\mathrm{pp}\) at KL \(0.9\) on CIFAR-100, and \(+1.0\) to \(+2.2\,\mathrm{pp}\) across KL \(0.4\)–\(0.7\) on CIFAR-10. ADS-C’s deepest zero-cost damage (\(29.95\)/\(51.80\)) is unmatched by any softener configuration on either dataset. The softener retains a \(\le1\,\mathrm{pp}\) advantage only in the low-damage corner (KL \(\le 0.3\)) that is undesirable for defense. The matched-\(\lambda\) crossover is thus an artifact of comparing on an axis that costs nothing; at matched served fidelity, softening is dominated. This ablation also answers the natural question of why ADS-C includes no confidence-scaling component: the strongest such component we constructed is dominated by not including one.

8.5 Controls↩︎

The hard-label attacker. We train the attacker of Sec. 3 on the served argmax alone (cross-entropy; implemented as KL against the one-hot served label, which is identical). Under ADS-C the served argmax equals the clean argmax on every query. We verify zero deviations over both \(50\,000\)-query corpora at \(\lambda\in\{2,3\}\). Therefore this attacker receives identical supervision from the defended and undefended APIs. Two observations follow. First, these overconfident teachers serve vectors so nearly one-hot that, without softening, dark knowledge confers almost no advantage to begin with, consistent with Table 1, and complementary to Sec. 6.1, where the advantage appears (\(+4.7\,\mathrm{pp}\)) as soon as softening reveals the non-target structure. Second, and decisive for the defense: ADS-C drives the soft-label student accuracy down by \(17.1\,\mathrm{pp}\) (CIFAR-100, \(\lambda=3\)) and \(29.7\,\mathrm{pp}\) (CIFAR-10). The defended soft output is strictly worse distillation signal than the argmax it accompanies. The defense does not merely remove the incentive to distill from served probabilities, it reverses it, while the argmax channel remains exactly as informative as any accurate API must be. Sec. 9 addresses the attacker who exploits knowledge of the guarantee itself.

8.6 Fidelity of the stolen copy↩︎

Top-1 accuracy understates the damage to an attacker whose goal is a faithful replica rather than a merely accurate one (e.g., to mount downstream evasion or membership-inference attacks [13], [15]). Agreement between student and teacher predictions falls in step with student accuracy (CIFAR-100: \(0.49\) undefended \(\to\) \(0.31\) at \(\lambda=3\); CIFAR-10: \(0.83\to0.52\)): the extracted model is not only worse, it is a worse copy.

9 TRADING THE GUARANTEE FOR ATTACKER UNCERTAINTY↩︎

Proposition 1 is deliberately absolute, and absoluteness is itself information: an attacker who knows the defense knows that every served top-1 is honest, and that inputs whose margin lies at or below the threshold are served identically to the clean teacher. Those provably clean rows are, moreover, exactly the thin-margin inputs where near-boundary class structure, corresponding to the richest dark knowledge, resides. A defender may therefore wish to spend a small, controlled amount of the guarantee to remove these certainties. We provide two such relaxations, each with closed-form-predictable utility cost and each disabled by default; ADS-C with \(q=1\) and \(\lambda_{\min}=0\) remains the proposed method. All results below are trained-student measurements (3 seeds, both CIFAR datasets).

9.1 Stochastic enforcement of the budget↩︎

The first relaxation enforces the budget 8 on each input with probability \(q\); with probability \(1-q\) the input receives the full requested \(\lambda\) (raw poison). The endpoints recover the two extremes—\(q=1\) is ADS-C, \(q=0\) is unmodified ADS—and the expected teacher cost follows in closed form from Sec. 5.4: \((1-q)\) times the unmodified-ADS teacher cost at \(\lambda\) predicted by 6 , so the defender can price the relaxation from cached outputs before deploying it. The enforcement mask is drawn once per input from a fixed seed, not per query (Sec. 9.3).

Figure 13 shows the resulting family of trade-off curves, which interpolates between the ADS-C vertical and the ADS diagonal. The trade-off is ordered exactly by \(q\) on both datasets: at every level of teacher cost, the best achievable student damage is obtained by the largest \(q\) whose curve reaches that cost, so no configuration with smaller \(q\) is ever preferable at equal spend. The exchange below the zero-cost floor is far better than unmodified ADS offers: relaxing to \(q=0.9\) at \(\lambda=3\) yields a further \(-4.1\,\mathrm{pp}\) of student damage (CIFAR-100, to \(25.9\%\)) for \(6.4\,\mathrm{pp}\) of teacher cost, where unmodified ADS would require \(32.8\,\mathrm{pp}\) to match—approximately a \(5\times\) better exchange; on CIFAR-10, \(-6.0\,\mathrm{pp}\) for \(7.9\,\mathrm{pp}\) against \(39.3\,\mathrm{pp}\) (\(5.0\times\)). Extending the sweep to \(\lambda=10\) on CIFAR-100 strengthens this to a uniform statement: the relaxed configurations saturate (\(q=0.9\) reaches student accuracy \(22.7\%\) at \(6.9\,\mathrm{pp}\), more than one student-pp per teacher-pp below the floor), and unmodified ADS becomes dominated at every point of the trade-off—its deepest operating point (student \(4.2\%\) at \(64.0\,\mathrm{pp}\)) is surpassed by \(q=0.3\) at \(\lambda=10\) (student \(3.5\%\) at \(49.1\,\mathrm{pp}\)).

The hard-label measurements of Sec. 8.5 add an important qualification (Fig. 14). Once \(q<1\), the two attacker channels separate, and at every \(q\) the hard-label student is the stronger one (\(38.5\%\) versus \(25.9\%\) at \(q=0.9\) on CIFAR-100; \(73.1\) versus \(45.8\%\) on CIFAR-10): a defense-aware attacker responds to the relaxation by distilling from the served argmax, whose label noise—the flipped rows—arrives at the closed-form rate \((1-q)F(\lambda)\). Priced against this best response, the exchange below the floor is \(1.35\), \(1.02\), \(0.89\), and \(0.81\) student-pp per teacher-pp at \(q=0.9\), \(0.7\), \(0.5\), and \(0.3\) on CIFAR-100 (\(1.07\), \(1.02\), \(0.99\), and \(0.97\) on CIFAR-10): approximately break-even near the guarantee, drifting below it at deep relaxation, and in every case far from the \(5\times\) available against an attacker committed to soft labels. Three observations delimit the relaxation’s value. First, the best-responding attacker still degrades below the floor at every \(q\)—the deliberate flips are systematic, input-consistent noise that the student learns rather than averages away—so no channel recovers the undefended leakage. Second, the exchange rates place the defensible operating points at shallow relaxation (\(q\gtrsim0.7\)), where the trade stays at or above break-even against either channel; deeper relaxation is justified only against an attacker known to consume soft labels. Third, the guarantee point \(q=1\) remains uniquely favorable: it is the only setting at which the defender pays nothing while the attacker’s best response yields exactly the floor.

9.2 Defensive flipping with a minimum \(\lambda\)↩︎

The second relaxation addresses the clean-row exposure directly: serve every input with strength at least \(\lambda_{\min}\), i.e., \(\lambda_{\mathrm{eff}}(\mathbf{x})=\max(\lambda_{\min}, \lambda(\mathbf{x}))\). Inputs whose flip threshold lies below \(\lambda_{\min}\) are deliberately flipped, which we refer to as defensive flipping, concentrated by construction on the thinnest-margin rows, where flips are least costly. The gross cost is again available in closed form from the transition curve, \(F(\lambda_{\min})\), and the net cost is smaller, for a geometric reason: thin-margin rows are disproportionately already misclassified, and on such rows the true class is often the runner-up (\(68\%\) of flipped misclassified rows on CIFAR-10, \(33\%\) on CIFAR-100), so a fraction of defensive flips land on the true label and partially offset the gross cost (net cost \(\approx{\tfrac{1}{2}}\) gross on CIFAR-100; the offset weakens with \(K\)). This is a consequence of margin geometry rather than of any corrective aim in the poison, and we report it as such.

Measured end-to-end, the minimum-strength floor serves a single purpose, at near-zero cost: at \(\lambda_{\min}=0.05\), \(\lambda=3\), no served row is clean (the clean-row fraction falls from \(0.4\%\) to \({\approx}0\)) at a teacher cost of \(0.30\,\mathrm{pp}\) (CIFAR-100) / \(0.16\,\mathrm{pp}\) (CIFAR-10), with student accuracy statistically indistinguishable from the ADS-C floor (\(30.1\) vs.\(29.95\); \(51.4\) vs.\(51.80\)). It is a hardening mechanism—it removes the attacker’s certainty about which rows are unperturbed—not a means of increasing damage. Nor does it open a side channel: the hard-label attacker under \(\lambda_{\min}\in\{0.05,0.2\}\) remains within seed noise of its leakage floor (\(46.7\)/\(47.3\) vs.\(47.0\%\) on CIFAR-100; \(81.3\)/\(81.5\) vs.\(81.6\%\) on CIFAR-10), as the \(\le0.2\%\) label-noise rate predicts.

A stronger variant is conceivable: sacrificing the thin-margin rows entirely (serving them the full unbudgeted \(\lambda\)) is the margin-ranked analogue of stochastic enforcement, and on served-distribution corruption (poison-KL) it appeared to dominate stochastic sacrifice by a factor of \({\sim}2\) in our training-free pre-analysis. The trained students overturn this prediction (Fig. 15): margin-ranked sacrifice cannot push the student meaningfully below the ADS-C floor (best case \(29.15\%\) vs.floor \(29.95\%\) on CIFAR-100, at \(6.0\,\mathrm{pp}\) teacher cost), whereas stochastic sacrifice at the same \({\sim}6\,\mathrm{pp}\) reaches \(25.9\%\) and continues to \(8.3\%\) at higher spend. Concentrating heavy poison on near-boundary rows inflates the served KL divergence while teaching the attacker’s student little it would not have learned regardless: those rows carry little informative dark knowledge relative to their contribution to the KL divergence. We report this negative result because it provides independent, converging evidence for the mechanism identified in Sec. 8.3: what damages a distilling student is corruption of the informative inter-class structure on representative inputs, not maximization of distributional distortion where distortion is inexpensive.

9.3 Determinism and repeated queries↩︎

Any defense that randomizes per query is invertible by repetition: querying the same input many times and aggregating the responses recovers the clean signal. The mechanisms above are immune to this attack by construction. The base ADS-C perturbation is a deterministic function of the input (teacher logits and frozen proxy clones), so repeated queries return the identical perturbed vector; the enforcement mask of Sec. 9.1 is drawn per input from a fixed seed, not per query, so a sacrificed input is consistently sacrificed; and \(\lambda_{\min}\) is inherently per-input deterministic. Repeated queries therefore provide the attacker no additional information. The residual cost of determinism is a persistent error set under \(\lambda_{\min}\) (the same flipped inputs are always flipped)—for a deployed API arguably preferable to nondeterministic answers—and identifying that set would require access to the clean teacher, which is the asset under protection. The utility statement weakens in form but remains strong: from “accuracy identical” to “accuracy drop at most \(F(\lambda_{\min})\), known in closed form before deployment, and concentrated on inputs the teacher was already uncertain about.”

10 DISCUSSION AND LIMITATIONS↩︎

What the guarantee does and does not cover. Proposition 1 protects the served argmax—the quantity consumed by classification API users. Applications that consume the full served distribution (e.g., downstream risk calibration) pay the served-fidelity cost, which we quantify explicitly: it is the horizontal axis of Fig. 12, and at the recommended operating point amounts to a served KL of \(0.8\)–\(0.9\) (CIFAR-100). In the dark-knowledge threat model this cost falls almost entirely on the attacker: the honest consumer of a classification API reads the label.

Attacker adaptivity. Our main results use the standard fixed distillation attacker. The hard-label attacker is reduced to the distillation floor every classification API shares. Repetition attacks are addressed by per-input determinism (Sec. 9.3). Attacks that estimate and subtract the poison would need to reconstruct a hidden gradient direction through a black-box interface; we view a formal analysis of this inversion problem as valuable future work.

Scale and generality. Our evidence is CIFAR-10, CIFAR-100, and Tiny-ImageNet. Nothing in ADS-C is architecture- or scale-specific, and the Tiny-ImageNet teacher already covers the deployment-realistic case of a strong pretrained model, whose overconfidence (Table 1) shows the phenomenon we exploit is a property of model quality rather than of small benchmarks. Validation on large-scale production teachers and modern architectures (e.g., vision transformers) remains future work; the transition-curve analysis of Sec. 5.4 provides a tool, computable from teacher and proxy outputs alone, to predict, before any training, how a new teacher will behave.

The trade-off-rate lens. We encourage future inference-time defenses to report (i) utility cost with the axes made explicit, and (ii) frontiers at matched served fidelity rather than matched internal knob settings; Sec. 8.4 shows the latter comparison can invert the former.

11 CONCLUSION↩︎

We adapted Antidistillation Sampling to classification and showed that its behavior there is governed by teacher overconfidence: an inert window in which the defense measurably affects neither party, a transition curve that predicts the defended teacher’s cost across the entire sweep, and a saturating regime in which the direct transfer trades below break-even. The perturbation itself is faithful and correctly aimed at dark knowledge; the failure is margin-blindness at composition, on teachers whose confidence is extremely heterogeneous. Temperature softening rescales the transition curve exactly as the theory predicts, which makes it a valuable probe and demonstrates that the availability of dark knowledge is what limits the perturbation at low strengths; but it is not a remedy: the same rescaling collapses the teacher, and every temperature configuration lies on the same unfavorable trade-off curve. ADS-C enforces the missing margin-awareness where both the margin and the perturbation are visible—at composition—with a closed-form per-input budget that provably preserves every served prediction. The result is, to our knowledge, the first antidistillation defense for classification with exactly zero utility cost: \(17.4\,\mathrm{pp}\) (CIFAR-100), \(29.6\,\mathrm{pp}\) (CIFAR-10), and \(13.3\,\mathrm{pp}\) (Tiny-ImageNet) of distilled-student degradation at a teacher cost of \(0.00\,\mathrm{pp}\). Measured against the complementary channel, the guarantee reverses the economics of extraction: an attacker restricted to the served argmax obtains exactly the distillation floor that every accurate classification API shares, while the defended soft output trains a student well below that floor. The served probabilities become strictly worse supervision than the labels they accompany. Around the guarantee we provide two relaxations with closed-form-predictable cost, stochastic enforcement and defensive flipping, that allow a defender to exchange a known amount of utility for attacker uncertainty, priced throughout against the attacker’s best response across both supervision channels, together with a negative result showing that stochastic sacrifice outperforms margin-ranked sacrifice, further evidence that the durable damage resides in informative dark knowledge. A defense that costs nothing changes the deployment calculus as there is no longer a utility argument for serving undefended soft labels.

ACKNOWLEDGMENTS↩︎

The authors thank the MS Artificial Intelligence Program at the Rochester Institute of Technology for computational resources, and the Fulbright Program for scholarship support to the first author. M. J. Khojasteh acknowledges support from the Gleason Endowment and Provost’s Learning Innovation Grant at RIT. The authors acknowledge Research Computing at the Rochester Institute of Technology for providing computational resources and support that have contributed to the research results reported in this publication [60].

APPENDIX↩︎

11.1 Notation↩︎

Table 3 summarizes the symbols used throughout the paper, grouped by role.

4pt

11.2 Calibration of the finite-difference step \(\varepsilon\)↩︎

Following the ADS paper [24], the penalty 2 approximates the exact directional derivative \(\langle \nabla_{\theta}\log p(y\mid\mathbf{x};\theta_\mathcal{P}),\, g\rangle\), which we compute directly as a Jacobian–vector product (JVP) on a reference batch. We select \(\varepsilon\) by sweeping over a log-grid and scoring the per-coordinate magnitude ratio and angular agreement between 2 and the JVP. Too-small \(\varepsilon\) loses the signal to floating-point cancellation; too-large \(\varepsilon\) leaves the linear regime (Fig. 16). The selected steps, \(\varepsilon = 3\times10^{-3}\) on both CIFAR datasets and \(\varepsilon = 10^{-2}\) on Tiny-ImageNet, each achieve a magnitude ratio of \(\approx1.00\) under our GPU (TF32) inference stack; the calibration is hardware-arithmetic-specific and should be re-run if the serving stack changes numerics.

11.3 Sensitivity to the margin floor \(m\)↩︎

The guarantee is \(m\)-independent: recomputing the full serving pipeline over \(m\in\{0.01,0.05,0.1,0.2\}\) at \(\lambda\in\{1,2\}\), the defended teacher’s top-1 drop is exactly \(0.00\,\mathrm{pp}\) at every setting on both datasets, as Proposition 1 requires. What \(m\) modulates is only how much poison the budget passes, and mildly: at \(\lambda=2\) on CIFAR-100 the mean effective strength moves from \(1.028\) (\(m=0.01\)) to \(1.009\) (\(m=0.2\)) and the served poison-KL from \(0.842\) to \(0.757\) (\({\approx}10\%\) across a \(20\times\) range of \(m\)); CIFAR-10 behaves identically (\(0.702\to0.627\)). At \(\lambda=1\) the spread is smaller still. Conclusions are therefore insensitive to \(m\); we fix \(m=0.05\) as a mid-range served-margin service level. (Rows whose original margin is below \(m\) receive zero poison and keep their original margin, per the \(\min(m,\text{original})\) guarantee—hence the minimum achieved served margin can lie below \(m\).) A student-side confirmation (distilled students at \(m\in\{0.01,0.05,0.1,0.2\}\), \(\lambda\in\{1,2\}\), three seeds per cell) shows the same insensitivity where it matters (Fig. 17): student top-1 varies by at most \(1.7\,\mathrm{pp}\) across the \(20\times\) range of \(m\) on CIFAR-100 (\(\lambda=2\): \(32.3\%\) at \(m=0.01\) versus \(33.9\%\) at \(m=0.2\)) and by at most \(2.9\,\mathrm{pp}\) on CIFAR-10 (\(53.7\%\) versus \(56.6\%\)), against a poison-induced degradation of \(10\)–\(27\,\mathrm{pp}\) at these \(\lambda\). Larger floors admit marginally less poison, as expected, and the defended teacher’s top-1 drop remains exactly zero at every cell.

11.4 Softening operators used in the ablation↩︎

The ablation of Sec. 8.4 uses a gated, argmax-preserving softener defined in log-odds space. Let \(G = z_t - \operatorname{logsumexp}_{j\neq t} z_j\), which equals \(\operatorname{logit}(p_{\max})\) exactly. Given a gate \(p_g\) with \(G_g = \operatorname{logit}(p_g)\), inputs with \(G \le G_g\) are untouched; over-confident inputs are mapped to the served confidence \(G' = G_g\) by lowering the top logit only (\(z_t \leftarrow G' + \operatorname{logsumexp}_{j\neq t} z_j\)), leaving all non-target structure intact for the dark-knowledge channel. We sweep \(p_g \in \{0.5, 0.6, 0.7, 0.8, 0.9\}\); the constant-temperature comparison is Sec. 6 (E-CT). The operator is inspired in spirit by asymmetric temperature scaling [59], which applies separate temperatures to the correct and wrong classes so that wrong-class discriminability survives softening; ours is the gated, serving-side limiting case in which the wrong-class logits are left exactly unchanged. Additionally, where asymmetric temperature scaling selects a fixed pair of temperatures by grid search over a candidate space, our operator adapts the effective softening per query.

11.5 Reproducibility↩︎

Table 4 reports the full training recipes. All models are trained with SGD (momentum \(0.9\), Nesterov, weight decay \(10^{-4}\)) under cosine annealing; teachers and proxies use a 5-epoch linear warmup, plain cross-entropy, and no label smoothing. Teachers and proxies train on the augmented training split (random crop with 4-pixel padding at \(32\times32\), 8-pixel at \(64\times64\), plus horizontal flip); the Tiny-ImageNet teacher is an ImageNet-pretrained ResNet-50 fine-tuned end to end with the stem adapted to \(64\times64\) (first-conv stride \(2\to1\), max-pool removed, fresh 200-way head). The clone gradient \(g\) is computed on a held-out \(10\%\) of the training split. The attacker’s student trains on the cached (image, served-vector) pairs with no augmentation. Every figure is generated by a script that emits a sibling metrics JSON with the underlying numbers, and aggregates for every (variant, \(\lambda\), seed) cell are archived. Code, configuration files, and exact command lines: https://github.com/the-kas-lab/ADS-C.

4pt

References↩︎