Improving Certified Robustness via Adversarial Distillation
June 30, 2026
Abstract
Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points.
1 Introduction↩︎
While deep neural networks have seen great success across various disciplines, adversarial examples [1]–[3] raise important questions about their adversarial robustness, i.e., a network’s ability to preserve its prediction under small input perturbations. This is of particular
concern in safety-critical domains such as autonomous driving and medical diagnostics. Motivated by this, neural network verification methods [4], [5] aim to provide formal certificates of robustness for a given network.
Neural network verifiers broadly fall into two categories, complete verifiers [4], [6], which
compute exact bounds but have an exponential worst-case runtime, and incomplete verifiers [7], [8], which rely on convex relaxations to obtain approximate bounds. Modern state-of-the-art verifiers [9]–[12] typically combine the two, using convex relaxations to accelerate complete verification within a branch-and-bound framework
[13].
Since verification only certifies robustness after training, robust training methods are needed to produce networks that are both accurate and certifiable. Training neural networks for adversarial robustness is typically approached either empirically,
through adversarial examples, or through certified training, using over-approximations of the network’s reachable set over a perturbation region. Adversarial training [14] is an empirical approach that aims to improve a network’s adversarial robustness by augmenting the training objective with adversarial examples, thereby optimising a
lower bound on the worst-case loss. However, adversarially trained networks are hard to formally verify and often fall short when faced with stronger or adaptive attack strategies [15], [16]. In contrast, certified training directly optimises an upper bound on the worst-case loss, producing networks that are more amenable to
verification, often at the cost of standard accuracy. While earlier certified training methods [17]–[20] trained networks against sound upper bounds of the robust loss, recent state-of-the-art methods [21]–[24] effectively optimise unsound approximations of the
worst-case loss by combining adversarial training with loose IBP-based over-approximations [18]. Due to IBP’s favourable optimisation properties [25], [26] and adversarial training’s empirical strengths, such certified training methods have
been shown to provide state-of-the-art standard-certified accuracy trade-offs in recent benchmarks [27].
Motivated by the success of robust objectives that combine adversarial and certified training, we hypothesise that transferring adversarial knowledge from an empirically robust teacher to a certified student using knowledge distillation could further
improve certified performance when paired with IBP bounds. This leads to the following contributions:
We introduce Adversarial Distillation for Certification (AD-CERT) (§3.1), a novel certified training objective that combines adversarial logit-level distillation from an empirically robust teacher with IBP bounds (depicted in Figure 1).
We show that AD-CERT is a scalarisation between a teacher-guided lower bound surrogate and a certified IBP upper bound on the robust loss (§3.2).
We present extensive experimental evaluations of AD-CERT across standard certified training benchmarks (§4.2), showing that it achieves state-of-the-art certified accuracy against current certified training methods.
We provide a systematic analysis of multiple knowledge distillation approaches for empirical endpoints (§4.3.0.1), showing that adversarial logit-level distillation preserves teacher robustness more effectively than clean or feature-space distillation over the same architecture, and translates best into certified training when combined with an IBP loss (§4.2).
2 Background↩︎
Here, we discuss some necessary background for this work. Let \(f_\theta\) denote a multi-class neural network classifier parametrised by \(\theta\) such that \(f_\theta: \mathbb{R}^{d} \mapsto{\mathbb{R}^C}\) maps an input \(\mathbf{x} \in \mathcal{X} \subseteq \mathbb{R}^d\) to a numerical prediction score \([f_\theta(\mathbf{x})]_i\) for each class \(i \in \{1, 2, \dots, C\}\). Let \((\mathbf{x}, y) \sim \mathcal{D}\) denote a sample input and true label from a data distribution \(\mathcal{D}\) and let \(\mathcal{B}_{\varepsilon_p}(\mathbf{x})\) denote a perturbation set based on an \(\ell_p\) norm threat model. In this work, as standard with certified training [18], [21]–[24], [28], [29], we only consider the \(\ell_\infty\) case, and therefore put \(\mathcal{B}_\varepsilon(\mathbf{x}) := \{\mathbf{x}' : \|\mathbf{x}' - \mathbf{x}\|_\infty \le \varepsilon\}\), where \(\mathbf{x}'\) denotes the perturbed input and \(\varepsilon\) is the perturbation radius.
2.1 Neural Network Verification↩︎
A network is said to be adversarially robust for an input \(\mathbf{x}\) if for all \(\mathbf{x'} \in \mathcal{B}_\varepsilon(\mathbf{x})\), we have \([f_\theta(\mathbf{x'})]_i < [f_\theta(\mathbf{x'})]_y \;\forall \;i \ne y\). Let \(\mathbf{z}^\Delta_\theta \in \mathbb{R}^{C-1}\) denote the logit-difference between the true class and all other classes, i.e., \(\mathbf{z}^\Delta_\theta(\mathbf{x}, y) := ([f_\theta(\mathbf{x})]_y - [f_\theta(\mathbf{x})]_i)_{i\ne y}\). Then, a network’s adversarial robustness can be verified by solving: \[\min_{\mathbf{x'} \in \mathcal{B}_\varepsilon(\mathbf{x})}\text{ }\min_{i \ne y}{[\mathbf{z}^\Delta_\theta(\mathbf{x'}, y)]}_i, \label{eq:rob95ver}\tag{1}\] and checking if the solution to Equation 1 is greater than zero. Finding an exact solution to Equation 1 was shown to be NP-hard [4], thus, neural network verifiers either approximate the bounds [7], [8], or perform complete verification at a larger computational cost [4], [6], [9]–[12].
2.2 Training for Robustness↩︎
Standard training of neural network classifiers optimises network parameters \(\theta\) by minimising the expected cross-entropy loss, i.e., \[\min_\theta \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ \mathcal{L}_{\mathrm{CE}} \left(f_\theta ( \mathbf{x}), y \right) \right], \label{eq:train95std}\tag{2}\] where \(\mathcal{L}_{\mathrm{CE}}(f_\theta(\mathbf{x}), y) = \log (1 + \sum_{i\ne y}^C\exp ([f_\theta(\mathbf{x})]_i - [f_\theta(\mathbf{x})]_y))\). In contrast, training for robustness can be viewed as the following min-max objective: \[\min_\theta \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ \max_{\mathbf{x}' \in \mathcal{B}_\varepsilon( \mathbf{x})} \mathcal{L}_{\mathrm{CE}} \left(f_\theta ( \mathbf{x}'), y \right) \right]. \label{eq:train95rob}\tag{3}\] Concretely, for an input \(\mathbf{x}\), the inner maximisation finds the most malicious perturbation \(\mathbf{x}' \in \mathcal{B}_\varepsilon(\mathbf{x})\) that maximises the loss. The outer minimisation seeks the optimal network parameters \(\theta\) such that these threats are less effective against the network predictions. Since the inner maximisation problem is non-convex, we generally solve this by under- or over-approximating the worst-case loss. In general, if we let \(\mathcal{L}^*_\mathrm{rob}(f_\theta(\mathbf{x}),y) := \displaystyle \max_{\mathbf{x}' \in \mathcal{B}_\varepsilon( \mathbf{x})} \mathcal{L} \left(f_\theta ( \mathbf{x}'), y \right)\) denote a network objective solved for the optimisation problem in Equation 3 , then we have: \[\underline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}),y) \le \mathcal{L}^*_\mathrm{rob}(f_\theta(\mathbf{x}),y) \le \overline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}),y), \label{eq:rob-inequality}\tag{4}\] with \(\underline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}),y)\) and \(\overline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}),y)\) corresponding to adversarial and certified training, respectively, as described below.
2.2.0.1 Adversarial Training (AT)
AT first approximates a solution to the inner maximisation problem of Equation 3 by performing an adversarial attack, such as projected gradient descent (PGD) [14], within \(\mathcal{B}_\varepsilon(\mathbf{x})\) for each input \(\mathbf{x}\). Following this, the outer minimisation problem of Equation 3 is solved by augmenting the network’s training objective with the adversarial example \(\mathbf{x_{adv}} \in \mathcal{B}_\varepsilon(\mathbf{x})\) obtained by the attack. In its standard form, AT aims to minimise: \[\mathcal{L}_{\mathrm{AT}} := \mathcal{L}_{\mathrm{CE}} \left(f_\theta ( \mathbf{x_{adv}}), y \right).\] This can be seen as optimising a lower bound of the worst-case loss, i.e., \(\underline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}), y)\) from Equation 4 . While AT leads to strong empirical robustness, it produces complex networks that are difficult to formally verify and remain susceptible to stronger attacks [15], [16].
2.2.0.2 Certified Training and Bound Propagation
Certified training optimises an upper bound of the inner-maximisation problem in Equation 3 by over-approximating the worst-case loss. A simple approach is Interval Bound Propagation (IBP) [18], leveraging interval arithmetic to approximate the output range of a given layer. Given an input \(\mathbf{x}\), IBP propagates the upper
and lower bounds of the perturbed input space, \(\mathcal{B}_\varepsilon(\mathbf{x})\), (corresponding to \(\overline{\mathbf{x}} := \mathbf{x} + \varepsilon\) and \(\underline{\mathbf{x}} := \mathbf{x} - \varepsilon\), respectively) through the entire network layer by layer, providing a hyper-rectangle that is monotonically increasing every layer. The final layer output of this
propagation, which we denote with Box, is an over-approximation of the possible output space, i.e., Box \(\supseteq f_\theta(\mathcal{B}_\varepsilon(
\mathbf{x}))\). Hence, using the Box relaxation, we can compute the worst-case logit difference between the true class and all other classes by putting \(\mathbf{\underline{z}}_\theta^\Delta(\mathbf{x}, y) := \left(\underline{[f_{\theta}(\mathbf{x}')]}_y - \overline{[f_{\theta}(\mathbf{x}')]}_i \right)_{i \ne y}\) and obtain a robust loss given by: \[\mathcal{L}_{\mathrm{IBP}} := \mathcal{L}_{\mathrm{CE}}(-\mathbf{\underline{z}}_{\theta}^\Delta(\mathbf{x},y),y),\] where \(\mathcal{L}_{\mathrm{IBP}}\) is a sound upper bound, \(\overline{\mathcal{L}}_\mathrm{rob}(f_\theta(\mathbf{x}), y)\), from Equation 4 . Further details on IBP can be found in Appendix 7.
Interestingly, alternative certified training methods that utilise tighter relaxations, e.g., linear relaxations [19], [30], consistently yield worse performance than imprecise IBP-based training. [25]
attribute this phenomenon to tighter relaxations adding discontinuity and difficulty to the optimisation objective. Moreover, [26] explicitly show that
IBP-based training leads to better propagation tightness, while non-IBP-based methods do not. Additionally, they note that over-regularisation provides worse standard-certified accuracy trade-offs. In line with these findings, all current
state-of-the-art certified training methods [21]–[24] move away from previous over-regularised approaches [18], [28], [29] and instead shift their focus to combining AT with IBP-based training, achieving state-of-the-art standard-certified accuracy trade-offs while preserving
certification properties. [23] further elucidate this in their definition of expressive losses whereby expressivity is attained through
interpolation between lower and upper bounds of Equation 4 during training, providing an unsound but favourable solution to Equation 3 .
2.3 Knowledge Distillation↩︎
[31] introduced knowledge distillation as a method for transferring knowledge from a large teacher model, \(T_{\theta_t}\), to a smaller, more compact student model, \(S_{\theta_s}\). This allows the student to retain much of the teacher’s performance while reducing computational cost. Knowledge distillation utilises Kullback-Leibler (KL) divergence to compare the predictive distributions of the student and teacher, which is defined by: \[\mathrm{KL}\!\left(P^\tau_{\theta_t}(\mathbf{x}) \,\|\, P_{\theta_s}^\tau(\mathbf{x})\right) = \sum^C_{i=1} [ P_{\theta_t}^\tau(\mathbf{x})]_i \log \left( \frac{ [ P_{\theta_t}^\tau(\mathbf{x})]_i}{ [ P_{\theta_s}^\tau(\mathbf{x})]_i} \right), \label{eq:KL}\tag{5}\] where \(\tau\) is a temperature applied to the softmax operation and \(P_{\theta_t}^\tau(\mathbf{x}) := \mathrm{softmax}\left(\frac{T_{\theta_t}(\mathbf{x})}{\tau}\right)\), \(P_{\theta_s}^\tau(\mathbf{x}) := \mathrm{softmax}\left(\frac{S_{\theta_s}(\mathbf{x})}{\tau}\right)\) are the respective predictive distributions of the teacher and student after softmax. Knowledge distillation then optimises the following objective: \[\min_{\theta_s} \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ (1-\alpha)\mathcal{L}_{\mathrm{CE}} \left(S_{\theta_s} ( \mathbf{x}), y\right) + \alpha \tau^2 \mathrm{KL}\!\left(P_{\theta_t}^\tau(\mathbf{x}) \,\|\, P_{\theta_s}^\tau(\mathbf{x})\right) \right], \label{eq:KD}\tag{6}\] where \(\alpha\in[0,1]\) is a balancing factor between standard training and learning the teacher’s predictive distribution. Knowledge distillation has since been applied in the AT setting to boost empirical robustness of a student using an adversarially trained teacher [32]–[35]. This was first explored by [32], through adversarially robust distillation (ARD), which injects adversarial examples into the KL term from Equation 6 . More specifically: \[\mathcal{L}_{\mathrm{ARD}} := (1-\alpha)\mathcal{L}_{\mathrm{CE}} \left(S_{\theta_s} ( \mathbf{x}), y\right) + \alpha \tau^2 \mathrm{KL}\!\left(P_{\theta_t}^\tau(\mathbf{x}) \,\|\, P_{\theta_s}^\tau(\mathbf{x_{adv}})\right). \label{eq:ARD}\tag{7}\] Additionally, [24] recently introduced knowledge distillation within the field of certified training, where knowledge is distilled from an empirically robust teacher over the feature space (further details in Appendix 13). In the coming section, we build on previous certified training [18], [21]–[24], [28], [29] and adversarial knowledge distillation [32], [33] works, presenting a novel certified training objective that distils the predictive distribution of an empirically robust teacher onto a student training for certifiable robustness.
3 Methodology↩︎
In this section, we introduce Adversarial Distillation for Certification (AD-CERT), a novel certified training scheme that interpolates between adversarial distillation (AD) and IBP [18]. We leverage the key insight that replacing a hard-label adversarial lower bound with a teacher-guided surrogate could further improve certified training. In this way, the empirical teacher shapes the student on concrete adversarial examples, while certification is enforced directly through IBP. Full proofs and additional theoretical results are deferred to Appendices 8 and 9, respectively. Pseudocode for the complete training procedure is provided in Appendix 11 and further details on AD-CERT’s computational cost can be found in Appendix 12.5, where AD-CERT corresponds to performing IBP and PGD training, with the slight added cost of a single forward pass on the detached teacher.
3.1 AD-CERT Formulation↩︎
Going forward, we denote by \(f_{\theta_t}\) a fixed teacher network trained via AT, and let \(f_{\theta_s}\) be the student model that we aim to certify. Note that both the student and teacher share the same architecture in this distillation setting. Moreover, for all experiments, we use a temperature setting of \(\tau = 1\), and henceforth, omit the \(\tau\) notation from all softmax operations, writing \(P_\theta(\mathbf{x}) := \mathrm{softmax}(f_\theta(\mathbf{x}))\) for the predictive distribution of a network. Excluding additional regularisation, AD-CERT attains the following loss objective: \[\mathcal{L}_{\mathrm{AD-CERT}} := (1-\alpha)\underbrace{{\mathrm{KL}} \left(P_{\theta_t}(\mathbf{x}) \,\|\, P_{\theta_s}(\mathbf{x_{adv}}) \right)}_{\text{Adversarial Distillation}} + \alpha \underbrace{\mathcal{L}_\mathrm{CE}\left(-\mathbf{\underline{z}} _{\theta_s}^\Delta(\mathbf{x}, y), y \right)}_{\text{IBP}}, \quad \alpha \in [0, 1]. \label{eq:ad95cert95loss}\tag{8}\] The AD component of Equation 8 is exactly the KL portion of Equation 7 , where the student is evaluated on an adversarially perturbed input, and tasked to match the clean distribution of an adversarially trained teacher, aiming to preserve standard and empirically robust accuracy. On an intuitive level, AD-CERT is designed to keep the teacher signal completely outside of verified bounds, using it purely for its empirical strengths, while leveraging IBP to tighten the worst-case logit differences. This aims to make the student amenable to verification, while still retaining a large proportion of the teacher’s adversarial and standard accuracy.
3.2 Teacher-Guided Lower Bound Surrogates↩︎
In this subsection, we take an analytical view of the objective in Equation 8 and formalise the role of AD within certified training. Unlike AT and IBP, which, respectively, provide formal lower and upper bounds from Equation 4 , the AD term in AD-CERT is used as a teacher-guided surrogate for the adversarial lower bound. This is an intentional relaxation of the lower bound objective, using an empirical teacher only to improve the adversarial endpoint, while leaving the verifiable IBP upper bound objective unchanged. We begin by showing that, up to a teacher-dependent constant, the distillation component of AD-CERT is exactly AT with soft teacher labels.
Proposition 1 (Equivalence of AD and soft-label AT). Let \(\mathbf{x_{adv}}\) be fixed, and let the teacher predictive distribution \(P_{\theta_t}(\mathbf{x})\) be fixed. Then the AD objective: \[\mathcal{L}_{\mathrm{AD}} := \mathrm{KL}\!\left(P_{\theta_t}(\mathbf{x}) \,\|\, P_{\theta_s}(\mathbf{x_{adv}})\right),\] is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: \[\mathcal{L}_{\mathrm{soft-AT}} := \mathcal{L}_\mathrm{CE}\!\left(f_{\theta_s}(\mathbf{x_{adv}}),\, P_{\theta_t}(\mathbf{x})\right).\] Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index \(i\): \[\frac{\partial \mathcal{L}_{\mathrm{AD}}}{\partial z_{\theta_s}^i} = [P_{\theta_s}(\mathbf{x_{adv}})]_i - [P_{\theta_t}(\mathbf{x})]_i = \frac{\partial \mathcal{L}_{\mathrm{soft-AT}}}{\partial z_{\theta_s}^i}.\]
Proof. See Appendix 8. ◻
Given Proposition 1, we have shown that the distillation component of AD-CERT preserves AT at the gradient level, while being conditioned by an empirical teacher. We further elucidate this connection through an analytical view of our loss function from Equation 8 .
Proposition 2 (Analytical form of AD-CERT). Let \(H(P)\) and \(H(Q,P)\) denote the entropy and cross entropy respectively for probability distributions \(P\) and \(Q\). That is: \[H(P) = -\sum_{i=1}^C p_i\log(p_i), \qquad H(Q,P) = -\sum_{i=1}^C q_i\log(p_i).\] Then, \(\mathcal{L}_{\mathrm{AD-CERT}}\) from Equation 8 can be re-written as: \[\mathcal{L}_{\mathrm{AD-CERT}} = (1-\alpha)H(P_{\theta_t}(\mathbf{x}),P_{\theta_s}(\mathbf{x_{adv}})) + \alpha \mathcal{L}_{\mathrm{IBP}} - (1-\alpha)H(P_{\theta_t}(\mathbf{x})).\] Therefore, up to a constant independent of \(\theta_s\), AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (ii) a certified IBP upper bound.
Proof. See Appendix 8. ◻
Given Proposition 2, AD-CERT fits naturally within the broader class of modern certified training objectives [21]–[24] that combine empirical and certifiable endpoints. Its
distinguishing feature is that the adversarial endpoint is no longer a formal lower bound of Equation 3 , but a relaxed surrogate induced by AD from an empirically robust teacher. Additionally, this positions AD-CERT as
distinct and complementary to the recent work of CC-DIST [24], which computes both IBP bounds and adversarial examples within their robust
feature-space distillation objective. Instead, AD-CERT integrates teacher distillation directly into the lower bound endpoint, while keeping IBP bounds purely for the certified endpoint.
While Proposition 2 characterises AD-CERT at the level of its objective, the next result highlights a key geometric difference between soft-label and hard-label
adversarial supervision. In particular, the AD branch of AD-CERT induces a finite target in the logit space, whereas standard AT only approaches its optimum as the correct class margin grows indefinitely.
Lemma 1. For AD, the optimal logit margin, \(\Delta z_{ij}^*\), between any two classes \(i,j\) is: \[\Delta z_{ij}^* = \log\left(\frac{[P_{\theta_t}(\mathbf{x})]_i}{[P_{\theta_t}(\mathbf{x})]_j}\right).\] Moreover, standard AT has no finite minimiser, and its infimum \(0\) is approached only as \([f_{\theta_s}(\mathbf{x_{adv}})]_y-[f_{\theta_s}(\mathbf{x_{adv}})]_j\to+\infty, \quad \forall j\neq y\).
Proof. See Appendix 8. ◻
Lemma 1 suggests that teacher-guided soft-label supervision may be a principled approach to enhance certified training, since it arguably induces a smoother lower bound objective, not pushing for arbitrarily large class margins. We evaluate our hypotheses experimentally in the next section.
4 Experimental Evaluation↩︎
In this section, we conduct experiments to assess the effectiveness of our proposed certified training method, AD-CERT (§3.1). Across our experiments, we report standard, certified and AutoAttack (AA) [15] accuracy. Certified accuracy is the primary metric used in certified training literature [18], [21]–[24], [27]–[29], defined as the proportion of the validation set for which a verifier proves robustness within \(\mathcal{B}_\varepsilon(\mathbf{x})\). Standard accuracy is the clean accuracy on the validation set without perturbation. AA accuracy refers to the proportion of the validation set that could not be attacked by the AA strategy. As such, AA accuracy is strictly greater than or equal to certified accuracy.
4.1 Settings↩︎
We implement AD-CERT on top of CTBench [27], a unified certified training library using PyTorch [36]. We adopt three datasets, MNIST [37], CIFAR-10 [38] and TinyImageNet [39]. We train on the corresponding training sets and certify on the validation sets. To align with recent literature [20]–[24], [27], we use a 7-layer convolutional network (\(\mathrm{CNN7}\)) for both the teacher and student models, and the perturbation radii applied are \(\varepsilon = \{0.1, 0.3\}\) for MNIST, \(\varepsilon = \{\frac{2}{255}, \frac{8}{255}\}\) for CIFAR-10 and \(\varepsilon = \{\frac{1}{255}\}\) for TinyImageNet. For verification, we use the \(\alpha\),\(\beta\)-CROWN library [7], [10], [11], [40], which combines linear bound propagation with branch-and-bound [13] for complete neural network verification. Full details on training of both student and teacher models, certification and the experimental setup can be found in Appendix 12.
4.2 Comparison with State-of-the-Art↩︎
Table 1 compares the performance of AD-CERT (§3.1) to current state-of-the-art certified training methods [18], [21]–[24], [41]. For each method, we report the standard and certified accuracy on the validation set. When available, we use the benchmark results from CTBench [27], as they generally report the strongest certified accuracies of each reported certified training method compared to the original paper [18], [21]–[24], [41], while also providing a fair comparison to AD-CERT, which is built on top of the unified library.
Overall, AD-CERT achieves state-of-the-art certified accuracy for ReLU-based architectures across all evaluated settings, while retaining competitive standard accuracy. On MNIST, AD-CERT obtains modest but consistent improvements over prior methods at both
perturbation radii. While the gains are small, this is expected given the relative simplicity of MNIST and the already saturated performance of existing methods. For CIFAR-10 and TinyImageNet, AD-CERT improves the best prior ReLU-based certified accuracy
by an average of \(0.37\) percentage points, with the largest gain observed at \(\varepsilon=\frac{2}{255}\) on CIFAR-10. For CIFAR-10 at \(\varepsilon=\frac{8}{255}\), SORTNET [41] obtains higher certified accuracy, however, it relies on a specialised 1-Lipschitz
architecture known to perform particularly well on this setting. We therefore distinguish it from the ReLU-based methods, with further discussion in related work (§5). On TinyImageNet, AD-CERT achieves the highest
certified accuracy among all compared methods, while substantially improving standard accuracy compared to previous non-distillation-based certified training methods [18], [21]–[23]. This
suggests that AD-CERT scales favourably beyond the smaller MNIST and CIFAR-10 settings, where certified training is often closer in performance. An interesting comparison is that of AD-CERT and CC-DIST [24], the only distillation-based certified training methods. In general, CC-DIST retains higher standard accuracy, while AD-CERT obtains higher certified accuracy. We investigate this trade-off
further through a controlled comparison in Table 2.
| Dataset | \(\varepsilon\) | Method | Source | Std. acc. [%] | Cert. acc. [%] |
|---|---|---|---|---|---|
| MNIST | \(0.1\) | AD-CERT | Ours | 99.25 | 98.53 |
| IBP | [27] | 98.87 | 98.26 | ||
| SORTNET\(^\dagger\) | [41] | 99.01 | 98.14 | ||
| SABR | [27] | 99.08 | 98.40 | ||
| TAPS/STAPS\(^\ddagger\) | [27] | 99.16 | 98.52 | ||
| MTL-IBP | [27] | 99.18 | 98.37 | ||
| \(0.3\) | AD-CERT | Ours | 98.85 | 93.98 | |
| IBP | [27] | 98.54 | 93.80 | ||
| SORTNET\(^\dagger\) | [41] | 98.46 | 93.40 | ||
| SABR | [27] | 98.66 | 93.68 | ||
| TAPS/STAPS\(^\ddagger\) | [27] | 98.56 | 93.95 | ||
| MTL-IBP | [27] | 98.74 | 93.90 | ||
| \(\frac{2}{255}\) | AD-CERT | Ours | 79.21 | 65.08 | |
| IBP | [27] | 67.49 | 55.99 | ||
| SORTNET\(^\dagger\) | [41] | 67.72 | 56.94 | ||
| SABR | [27] | 77.86 | 63.61 | ||
| TAPS/STAPS\(^\ddagger\) | [27] | 77.05 | 64.21 | ||
| MTL-IBP | [27] | 78.82 | 64.41 | ||
| CC-DIST | [24] | 81.55 | 64.60 | ||
| 2-6 | \(\frac{8}{255}\) | AD-CERT | Ours | 53.69 | 35.84 |
| 3-6 | IBP | [27] | 48.51 | 35.28 | |
| SORTNET\(^\dagger\) | [41] | 54.84 | 40.39 | ||
| SABR | [27] | 52.71 | 35.34 | ||
| TAPS/STAPS\(^\ddagger\) | [27] | 49.96 | 35.25 | ||
| MTL-IBP | [27] | 54.28 | 35.41 | ||
| CC-DIST | [24] | 55.13 | 35.52 | ||
| TinyImageNet | \(\frac{1}{255}\) | AD-CERT | Ours | 40.06 | 28.19 |
| IBP | [27] | 26.77 | 19.82 | ||
| SORTNET\(^\dagger\) | [41] | 25.69 | 18.18 | ||
| SABR | [27] | 30.58 | 20.96 | ||
| TAPS/STAPS\(^\ddagger\) | [27] | 30.63 | 22.31 | ||
| MTL-IBP | [27] | 35.97 | 27.73 | ||
| CC-DIST | [24] | 43.78 | 27.88 | ||
| \(^\ddagger\) We report the stronger of TAPS and STAPS based on certified accuracy. |
4.2.0.1 Comparison of Distillation-based Certified Training Methods
Table 2 compares AD-CERT against alternative distillation-based certified training methods under the exact same experimental setup for a direct comparison. This controlled comparison allows us to
isolate the effect of the distillation mechanism itself by training and evaluating all methods in the CTBench library [27] with the same teacher model,
training pipeline and verifier. We compare AD-CERT with CC-DIST [24], which performs robust feature-space distillation. Additionally, we introduce
RSLAD-CERT, an alternative logit-level objective obtained by replacing the adversarial distillation term in AD-CERT with RSLAD [33], but retaining the same IBP
interpolation, i.e., \(L_{\mathrm{RSLAD-CERT}}:= (1-\alpha)L_{\mathrm{RSLAD}} + \alpha L_{\mathrm{IBP}}\). For all three methods, we use the exact same hyperparameter values as those used for AD-CERT in Table 1, which we report in detail in Appendix 12.3, Table 6. The only exception is \(w_{rob}\) for
TinyImageNet, where we used an earlier untuned value of \(0.7\), as per MTL-IBP’s [23] reported value in CTBench [27].
Overall, the results support the design choice of AD-CERT, showing that directly distilling the teacher’s predictive distribution at the adversarial endpoint provides a simple and effective empirical branch for certified training. The addition of the clean
distillation term in RSLAD-CERT reduces certifiability by \(0.6\) percentage points on average, while only improving standard accuracy by an average of \(0.38\) percentage points compared to
AD-CERT. Consistent with the pattern observed in Table 1, CC-DIST is generally strongest at transferring standard accuracy, but shows weaker certified accuracy under the controlled setup. This results in worse
standard-certified accuracy trade-offs, with CC-DIST improving standard accuracy over AD-CERT by an average of \(1.69\) percentage points, but reducing certifiability by \(3.52\) percentage
points. Both logit-level objectives, i.e., RSLAD-CERT and AD-CERT, improve certified accuracy over the feature-space distillation objective of CC-DIST, despite not matching the formal definition of expressivity [23]. One possible explanation is that CC-DIST induces a more regularised objective, since it computes adversarial examples and IBP bounds during distillation, before being
coupled with another expressive loss CC-IBP [23]. This suggests that tightly coupling distillation-based certified training objectives with the formal
definition of expressivity may not translate into stronger certification properties. In contrast, AD-CERT keeps the distillation branch as a pure adversarial endpoint, while leaving the certified component as the standard IBP loss. This relaxes the
definition of expressivity by extending it to surrogate lower bounds (§3.2), prioritising simplicity and smoothness in the loss objective. Finally, we acknowledge that further exhaustive hyperparameter
and teacher tuning on CC-DIST could potentially reduce the gap between the results reported in Table 2 and those in Table 1. However, the most important shared
hyperparameters, namely the IBP coefficient \(\alpha\) and \(L_1\) regularisation, coincide with the original CC-DIST setup [24] (Table 3). The main remaining difference is therefore the teacher model. For completeness, we include an additional ablation on the sensitivity of AD-CERT and CC-DIST to different teachers in
Appendix 10, which shows a similar trend to the above ablation.
| Dataset | \(\varepsilon\) | Method | Method source\(^\dagger\) | Dist. space | Std. [%] | AA [%] | Cert. [%] |
|---|---|---|---|---|---|---|---|
| 1-8 | \(\frac{2}{255}\) | AD-CERT | Ours | Logits | 79.21 | 68.38 | 65.20 |
| RSLAD-CERT | Ours | Logits | 79.87 | 68.27 | 64.36 | ||
| CC-DIST | [24] | Features | 81.04 | 69.97 | 59.80 | ||
| 2-8 | \(\frac{8}{255}\) | AD-CERT | Ours | Logits | 53.49 | 36.46 | 35.60 |
| RSLAD-CERT | Ours | Logits | 53.95 | 36.50 | 35.19 | ||
| CC-DIST | [24] | Features | 54.32 | 35.69 | 34.55 | ||
| TinyImageNet | \(\frac{1}{255}\) | AD-CERT | Ours | Logits | 41.38 | 30.68 | 27.68 |
| RSLAD-CERT | Ours | Logits | 41.39 | 30.90 | 27.14 | ||
| CC-DIST | [24] | Features | 43.79 | 32.66 | 23.57 |
4.3 Ablations & Further Discussions↩︎
4.3.0.1 Teacher-Student Comparison
Table 3 compares each AD-CERT student with its corresponding empirically robust teacher. Overall, AD-CERT converts models that are empirically robust but not certifiable into students with strong certified accuracy, while retaining a substantial fraction of the teacher’s standard and adversarial performance. This effect is particularly clear on MNIST, where AD-CERT stays within one percentage point of the teacher in both standard and AA accuracy across both perturbation radii, while introducing strong certifiability. On TinyImageNet, the teacher-student gap remains moderate, with losses below seven percentage points in standard accuracy and close to two percentage points in AA accuracy, though notably achieving \(28.19\%\) in certified accuracy. The largest gap occurs on CIFAR-10 at \(\varepsilon_\infty=\frac{8}{255}\), where the student sees a decrease of almost twenty percentage points in standard accuracy relative to the teacher, although the drop in AA accuracy is much smaller. Nevertheless, AD-CERT still achieves the strongest certified accuracy among ReLU-based architectures in this setting. However, this highlights that closing the remaining gap between empirically robust teachers and certifiably robust students at larger perturbation radii remains an important direction for future work. Motivated by this observation, we next ablate different pure AD techniques on CIFAR-10, aiming to better understand the difficulty of transferring empirical robustness into certifiable robustness and to identify potential directions for stronger distillation-based certified training objectives in future.
| Dataset | \(\varepsilon_\infty\) | Model | Std. acc. [%] | \(\Delta\) Std. | AA acc. [%] | \(\Delta\) AA | Cert. acc. [%] |
|---|---|---|---|---|---|---|---|
| 1-8 | \(0.1\) | Teacher | 99.47 | – | 98.97 | – | \(\approx0^\dagger\) |
| Student | 99.25 | 98.55 | 98.53 | ||||
| 2-8 | \(0.3\) | Teacher | 99.43 | – | 93.83 | – | \(\approx0^\dagger\) |
| Student | 98.85 | 94.64 | 93.98 | ||||
| CIFAR-10 | \(\frac{2}{255}\) | Teacher | 88.67 | – | 72.41 | – | \(\approx0^\dagger\) |
| Student | 79.21 | 68.39 | 65.08 | ||||
| \(\frac{8}{255}\) | Teacher | 73.55 | – | 41.99 | – | \(\approx0^\dagger\) | |
| Student | 53.69 | 36.69 | 35.84 | ||||
| TinyImageNet | \(\frac{1}{255}\) | Teacher | 46.78 | – | 33.16 | – | \(\approx0^\dagger\) |
| Student | 40.06 | 31.14 | 28.19 | ||||
| \(^\dagger\) None of the first 10 samples were verified within a 1000 second time-limit. | |||||||
4.3.0.2 Exploring Adversarial Distillation Techniques
Table 4 compares several distillation objectives on CIFAR-10 [38] after removing any certified training
component and trained against a PGD-trained teacher on the same architecture. This ablation isolates the empirical distillation endpoint, allowing us to study which teacher-student objectives best preserve the robustness of the PGD teacher before
introducing certifiability. Natural knowledge distillation [31], denoted Nat-KL, transfers the teacher’s clean predictive distribution to the student.
AD corresponds to removing the IBP term from AD-CERT (§3.1), while RSLAD [33] extends adversarially robust distillation
[32] using soft-labels. Finally, adversarial feature-space knowledge distillation (AF-KD) corresponds to setting \(\alpha=0\) in CC-DIST [24], thereby removing the IBP components from the objective. For RSLAD and AF-KD, we use \(\lambda=\frac{5}{6}\) and \(\beta=\frac{5}{k}\), respectively, following the suggestions of the original works [24], [33], where \(k\) denotes the dimensionality of the feature space.
From the results, it can be observed that Nat-KL improves standard accuracy at both perturbation radii, but dramatically fails to transfer adversarial robustness, losing more than \(41\) percentage points in AA accuracy in
both settings. This confirms that matching an adversarially trained teacher only on clean inputs is insufficient for transferring adversarial robustness. In contrast, objectives that expose the student to adversarial inputs preserve teacher-level AA
accuracy much more effectively. Across both perturbation radii, AD and RSLAD remain close to the teacher’s AA accuracy, while AF-KD is competitive at the smaller radius but degrades more noticeably at \(\varepsilon_\infty=\frac{8}{255}\). Among these methods, AD provides the most direct and consistently effective adversarial endpoint, improving teacher-level AA accuracy at the smaller radius and remaining within one percentage
point at the larger radius. These results further motivate our use of AD as the empirical endpoint of AD-CERT, since it provides a simple but effective mechanism for retaining adversarial accuracy from an empirically robust teacher, while remaining
straightforward to combine with an IBP-certified upper bound. This yields a loss objective that resembles previous state-of-the-art expressive losses [23], but in a distillation setting with lower bound surrogates, as discussed in §3.2.
| Dataset | \(\varepsilon_\infty\) | Method | Std. acc. [%] | \(\Delta\) Std. | AA acc. [%] | \(\Delta\) AA |
|---|---|---|---|---|---|---|
| CIFAR-10 | \(\frac{2}{255}\) | PGD Teacher | 88.67 | – | 72.41 | – |
| Nat-KL | 89.89 | 30.72 | ||||
| AD\(^\dagger\) | 88.64 | 73.62 | ||||
| RSLAD\(^\ddagger\) | 87.88 | 73.32 | ||||
| AF-KD\(^\dagger\) | 89.52 | 73.55 | ||||
| \(\frac{8}{255}\) | PGD Teacher | 78.95 | – | 42.48 | – | |
| Nat-KL | 81.32 | 0.52 | ||||
| AD\(^\dagger\) | 77.08 | 41.64 | ||||
| RSLAD\(^\ddagger\) | 76.47 | 42.38 | ||||
| AF-KD\(^\dagger\) | 78.95 | 37.49 |
4.3.0.3 IBP Coefficient Sensitivity Analysis
In certified training, it is well known that higher certified accuracy often comes at the cost of standard accuracy. Recent methods fall into a family of expressive losses [23], e.g., SABR [22], CC-IBP & MTL-IBP [23], and CC-DIST [24], including hyperparameters
that regulate this trade-off by interpolating between empirical and certifiable training objectives. Our goal in this study is to empirically support our methodology claim (§3.2) that the AD term in
AD-CERT provides a sensible teacher-guided surrogate for the lower bound endpoint used in prior expressive losses [22]–[24].
Figure 2 shows the trade-off between standard, AA, and certified accuracy for AD-CERT on TinyImageNet with \(\varepsilon=\frac{1}{255}\) as we decrease the IBP
coefficient, \(\alpha\), from Equation 8 . Consistent with the sensitivity analyses of [22] (Figure 7), [23] (Figure 1) and [27] (Figure 7), standard and AA accuracy increase as the strength of IBP regularisation is reduced. Up to a point, this also benefits certified accuracy, before the model starts
to dramatically lose its certification properties when \(\alpha\) is decreased beyond this. This behaviour suggests that the adversarial distillation endpoint in AD-CERT acts analogously to the hard-label adversarial
endpoints used in prior expressive losses, while also achieving state-of-the-art certified accuracy (§4.2).
![Figure 2: Sensitivity analysis of AD-CERT with respect to the IBP coefficient, \alpha, on TinyImageNet at \epsilon=1/255. We report standard, AA, and certified accuracy over eight values of \alpha \in[5\times10^{-4}, 5\times10^{-2}].](/article/2606.31653/images/adcert_alpha_sensitivity_tinyimagenet.png "fig:")
5 Related Work↩︎
5.0.0.1 Certified Training
As previously mentioned (§2.2), IBP [17], [18]
performs interval arithmetic over the bounds of a simple bounding-box relaxation that over-approximates the possible output ranges of each layer, before computing an upper bound of the worst-case loss based on the output range of the final layer. [20] later improved IBP-based training by introducing a parameter initialisation technique that induces a
constant growth rate of IBP bounds, along with specially designed regularisation to stabilise initial phases of training and improve overall performance. [22] propose SABR, which computes the IBP bounds over a small adversarially selected region rather than over the full perturbation set. It defines a smaller box of radius
\(\tau = \lambda_{\tau}\varepsilon\), with \(\lambda_{\tau} \in [0,1]\), centred around a projected adversarial point, ensuring the local region \(\mathcal{B}_{\tau}(\mathbf{x}_{\tau}) = \{\mathbf{x}' : \|\mathbf{x}'-\mathbf{x}_{\tau}\|_{\infty} \leq \tau\}\) is fully contained within \(\mathcal{B}_{\varepsilon}(\mathbf{x})\).
Final IBP bounds are then computed over \(\mathcal{B}_{\tau}(\mathbf{x}_{\tau})\) and used to calculate the final IBP loss, i.e., \(\mathcal{L}_{\mathrm{SABR}}
=\mathcal{L}(-\underline{\mathbf{z}}^{\Delta_\tau}_{\theta}(\mathbf{x},y), y)\). The intuition is that propagating a smaller adversarially selected region reduces the approximation error induced by IBP bounds, which grows with respect to network
depth [26], thereby providing an unsound but effective approximation of the robust loss over the full perturbation set. [21] introduced TAPS, which splits a network into two sub-parts before training the first part through IBP and the second through AT. The idea is that the PGD under-approximation
can compensate for some of the over-approximation error induced by IBP bounds. STAPS [21] carries out an identical procedure, only changing the perturbation
region to a smaller adversarially selected one as in SABR. [23] formalise a family of expressive losses, which interpolate between empirical
lower bounds and certified upper bounds of the worst-case robust loss. MTL-IBP is one such method that achieved relative state-of-the-art performance through leveraging a task balancing coefficient \(\alpha \in [0,1]\) for
fine-grained control over a linear interpolation of PGD and IBP based training, i.e., \(\mathcal{L}_{\mathrm{MTL-IBP}} = (1-\alpha)\mathcal{L}_\mathrm{PGD} + \alpha \mathcal{L}_\mathrm{IBP}\). [24] introduced CC-DIST, which splits the network into a feature extractor and classification head, using an empirically trained teacher for feature-space
distillation before learning final classifications through their prior CC-IBP objective [23] on the full model. CC-DIST is closest to our work, since it
distils knowledge from an empirically robust teacher. However, CC-DIST is tightly coupled with the notion of expressivity [23], combining a robust
feature-space distillation loss with the expressive CC-IBP objective. Moreover, CC-DIST calculates both adversarial examples and IBP bounds for distillation. In contrast, AD-CERT uses a simple logit-level AD branch as the empirical endpoint, while keeping
the certified branch as the baseline IBP loss.
All of the previously mentioned works train standard feed-forward ReLU networks. A related but distinct line of work instead studies robustness by construction. Sortnet [42] is a notable such model that utilises \(\ell_\infty\)-distance
functions to provide a 1-Lipschitz architecture. While Sortnet performs particularly well on the CIFAR-10 dataset with \(\varepsilon=\frac{8}{255}\), it is less competitive with standard
certified training methods on ReLU architectures, particularly against smaller \(\varepsilon\) values (see §4.2). The non-smoothness and gradient sparsity introduced by
1-Lipschitz architectures leave room for exploration in their practical applicability to certified training but remains outside the scope of this work.
6 Conclusions↩︎
In this paper, we introduced AD-CERT, a novel certified training algorithm that combines the empirical strengths of an adversarially trained teacher with the verifiability of IBP bounds through a simple linear combination of adversarial distillation over the logit-level and an IBP loss objective. AD-CERT achieves overall state-of-the-art certified accuracy, outperforming all prior methods across five standard certified training benchmarks, while remaining competitive in standard accuracy. Our ablations further suggest that adversarial logit-level distillation provides an effective empirical endpoint for certified training, acting analogously to previous hard-label lower bound endpoints while improving certified performance. Nevertheless, a substantial gap remains between empirically robust teachers and certifiably robust students in terms of standard and adversarial accuracy, particularly on harder settings, highlighting the need for future work on stronger mechanisms for transferring empirical robustness into certifiable robustness.
7 Interval Bound Propagation (IBP)↩︎
IBP has been at the core of all recent state-of-the-art certified training methods [20]–[24], [29]. Here, we formalise the bound propagation for a single layer, as proposed by [18]. For an affine layer (e.g., convolutional or linear layer) represented by \(h_k(z_{k-1}) = Wz_{k-1}+b\), we can define the bound calculation as follows: \[\begin{align} \mu_{k-1} = \frac{\bar{z}_{k-1} + \underline{z}_{k-1}}{2} &, \quad \mu_k = W\mu_{k-1} + b, \\ r_{k-1} = \frac{\bar{z}_{k-1} - \underline{z}_{k-1}}{2} &, \quad r_k = |W|r_{k-1}, \\ \underline{z}_k = \mu_k - r_k &, \quad \bar{z}_k = \mu_k + r_k, \end{align}\] where \(\mu\) denotes the centre of the interval box, \(r\) its radius, and \(\underline{z}_k,\bar{z}_k\) denote lower and upper bounds of the layer output, respectively. When \(h_k\) is an element-wise increasing activation function, e.g., ReLU, sigmoid, or tanh, we instead trivially propagate the bounds as \(\underline{z}_k = h_k(\underline{z}_{k-1})\) and \(\bar{z}_k = h_k(\bar{z}_{k-1})\). For ReLU networks, an unstable ReLU activation implies the ReLU activation may be either active or inactive given the input interval, i.e., \(\underline{z}_{k-1}< 0 <\bar{z}_{k-1}\). Hence, incomplete verifiers must over-approximate the possible activation outputs. Figure 3 compares the loose IBP box relaxation with the tighter convex hull relaxation for an unstable ReLU.
8 Deferred Proofs of Theoretical Results↩︎
In this section, we provide proofs for each of the theoretical results from our methodology section (§3) of the main text. In each case, we restate the claim for the ease of the reader.
Proposition 3 (Equivalence of AD and soft-label AT). Let \(\mathbf{x_{adv}}\) be fixed, and let the teacher predictive distribution \(P_{\theta_t}(\mathbf{x})\) be fixed. Then the AD objective: \[\mathcal{L}_{\mathrm{AD}} := \mathrm{KL}\!\left(P_{\theta_t}(\mathbf{x}) \,\|\, P_{\theta_s}(\mathbf{x_{adv}})\right),\] is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: \[\mathcal{L}_{\mathrm{soft-AT}} := \mathcal{L}_\mathrm{CE}\!\left(f_{\theta_s}(\mathbf{x_{adv}}),\, P_{\theta_t}(\mathbf{x})\right).\] Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index \(i\): \[\frac{\partial \mathcal{L}_{\mathrm{AD}}}{\partial z_{\theta_s}^i} = [P_{\theta_s}(\mathbf{x_{adv}})]_i - [P_{\theta_t}(\mathbf{x})]_i = \frac{\partial \mathcal{L}_{\mathrm{soft-AT}}}{\partial z_{\theta_s}^i}.\]
Proof. For brevity, we let \(z^i_{\theta_s}:= [f_{\theta_s}(\mathbf{x_{adv}})]_i\) and \(P^i_{\theta_s}:= [P_{\theta_s}]_i\) denote the i-th entry of the raw logit outputs and predictive distributions of the student classifier, respectively. First, recall the derivative of the softmax function: \[\frac{\partial P^j}{\partial z^i} = \begin{cases} P^i(1 - P^i), & i=j, \\ -P^i P^j, & i \neq j. \end{cases}\] For ease of notation, we can rewrite this in terms of the Kronecker delta function: \[\frac{\partial P^j}{\partial z^i} = P^j(\delta_{ij} - P^i), \quad \text{where } \delta_{ij} = \begin{cases} 1, & i=j, \\ 0, & i \neq j . \end{cases}\] Now, we compare the derivatives of \(\mathcal{L}_{\mathrm{AD}}\) and \(\mathcal{L}_{\mathrm{soft-AT}}\) with respect to \(z^i_{\theta_s}\). Given that \(\mathcal{L}_{\mathrm{AD}} = \sum^C_{j=1} P_{\theta_t}^j(\mathbf{x}) \cdot \log \left( \frac{ P_{\theta_t}^j(\mathbf{x})}{ P_{\theta_s}^j(\mathbf{x_{adv}})} \right) = \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) \cdot \log(P_{\theta_t}^j(\mathbf{x})) - \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) \cdot \log(P_{\theta_s}^j(\mathbf{x_{adv}}))\), we note that the \(\sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) \cdot \log(P_{\theta_t}^j(\mathbf{x}))\) term is independent of student parameters, so: \[\begin{align} \frac{\partial \mathcal{L}_{\mathrm{AD}}}{\partial z_{\theta_s}^i} &= \frac{\partial}{\partial z_{\theta_s}^i} \left[ - \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) \log(P_{\theta_s}^j(\mathbf{x_{adv}})) \right], \tag{9}\\ &= - \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) \frac{1}{P_{\theta_s}^j(\mathbf{x_{adv}})} \frac{\partial P_{\theta_s}^j(\mathbf{x_{adv}})}{\partial z_s^i}, \\ &= - \sum_{j=1}^{C} \frac{P_{\theta_t}^j(\mathbf{x})}{P_{\theta_s}^j(\mathbf{x_{adv}})} P_{\theta_s}^j(\mathbf{x_{adv}})(\delta_{ij} - P_{\theta_s}^i(\mathbf{x_{adv}})), \\ &= - \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}) (\delta_{ij} - P_{\theta_s}^i(\mathbf{x_{adv}})), \\ &= - P_{\theta_t}^i(\mathbf{x}) + P_{\theta_s}^i(\mathbf{x_{adv}}) \sum_{j=1}^{C} P_{\theta_t}^j(\mathbf{x}), \\ &= P_{\theta_s}^i(\mathbf{x_{adv}}) - P_{\theta_t}^i(\mathbf{x}). \tag{10} \end{align}\] Finally, recall that \(\mathcal{L}_{\mathrm{soft-AT}} = \mathcal{L}_\mathrm{CE}(f_{\theta_s}(\mathbf{x_{adv}}), P_{\theta_t}(\mathbf{x}))\), and, \[\mathcal{L}_\mathrm{CE}(f_{\theta_s}(\mathbf{x_{adv}}), P_{\theta_t}(\mathbf{x})) =-\sum^C_{j=1} P_{\theta_t}^j(\mathbf{x}) \log(P_{\theta_s}^j(\mathbf{x_{adv}})).\] Hence, the result falls immediately from Equations 9 10 , and by extension, the two objectives must only differ by an additive constant independent of \(\theta_s\). ◻
Proposition 4 (Analytical form of AD-CERT). Let \(H(P)\) and \(H(Q,P)\) denote the entropy and cross entropy respectively for probability distributions \(P\) and \(Q\). That is: \[H(P) = -\sum_{i=1}^C p_i\log(p_i), \qquad H(Q,P) = -\sum_{i=1}^C q_i\log(p_i).\] Then, \(\mathcal{L}_{\mathrm{AD-CERT}}\) from Equation 8 can be re-written as: \[\mathcal{L}_{\mathrm{AD-CERT}} = (1-\alpha)H(P_{\theta_t}(\mathbf{x}),P_{\theta_s}(\mathbf{x_{adv}})) + \alpha \mathcal{L}_{\mathrm{IBP}} - (1-\alpha)H(P_{\theta_t}(\mathbf{x})).\] Therefore, up to a constant independent of \(\theta_s\), AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (ii) a certified IBP upper bound.
Proof. The proof is immediate since we can write: \[\begin{align} \mathcal{L}_{\mathrm{KL}}\!\left(P_{\theta_t}(\mathbf{x}) \,\|\, P_{\theta_s}(\mathbf{x_{adv}})\right) &= \sum^C_{i=1} P_{\theta_t}^i(\mathbf{x}) \cdot \log \left( \frac{ P_{\theta_t}^i(\mathbf{x})}{ P_{\theta_s}^i(\mathbf{x_{adv}})} \right), \\ &= \sum^C_{i=1} P_{\theta_t}^i(\mathbf{x}) \cdot \big ( \log(P_{\theta_t}^i(\mathbf{x})) - \log(P_{\theta_s}^i(\mathbf{x_{adv}})) \big ), \\ &= - \sum^C_{i=1} P_{\theta_t}^i(\mathbf{x}) \log(P_{\theta_s}^i(\mathbf{x_{adv}})) + \sum^C_{i=1} P_{\theta_t}^i(\mathbf{x})\log(P_{\theta_t}^i(\mathbf{x})),\\ &= H(P_{\theta_t}(\mathbf{x}),P_{\theta_s}(\mathbf{x_{adv}})) - H(P_{\theta_t}(\mathbf{x})). \end{align}\] Hence, substituting this identity into Equation 8 gives: \[\mathcal{L}_{\mathrm{AD-CERT}} = (1-\alpha)H(P_{\theta_t}(\mathbf{x}),P_{\theta_s}(\mathbf{x_{adv}})) +\alpha \mathcal{L}_{\mathrm{IBP}} -(1-\alpha)H(P_{\theta_t}(\mathbf{x})),\] as required. ◻
Lemma 2. For AD, the optimal logit margin, \(\Delta z_{ij}^*\), between any two classes \(i,j\) is: \[\Delta z_{ij}^* = \log\left(\frac{[P_{\theta_t}(\mathbf{x})]_i}{[P_{\theta_t}(\mathbf{x})]_j}\right).\] Moreover, standard AT has no finite minimiser, and its infimum \(0\) is approached only as \([f_{\theta_s}(\mathbf{x_{adv}})]_y-[f_{\theta_s}(\mathbf{x_{adv}})]_j\to+\infty, \quad \forall j\neq y\).
Proof. First note that, from Equation 10 , \(\mathcal{L}_{\mathrm{AD}}\) is minimised iff \(P_{\theta_s}=P_{\theta_t}\), or equivalently, \(P_{\theta_s}^i(\mathbf{x_{adv}})=P_{\theta_t}^i(\mathbf{x})\) \(\forall\) \(i\in\{1,\dots,C\}.\) Rewriting this using the definition of softmax, we have: \[\frac{e^{z_s^i}}{\sum_{j=1}^C e^{z_s^j}} = P_{\theta_t}^i(\mathbf{x}) \quad \forall \;i\in\{1,\dots,C\},\] where \(z^i_{\theta_s} = [f_{\theta_s}(\mathbf{x_{adv}})]_i\), which implies: \[z_s^i = \log \left(P_{\theta_t}^i(\mathbf{x}) \cdot \sum_{j=1}^C e^{z_s^j} \right) = \log \left(P_{\theta_t}^i(\mathbf{x}) \right) + \log \left(\sum_{j=1}^C e^{z_s^j} \right)\] Then, we can write the optimal logit for a given
class \(i\) as \(z_s^{i*}:=\log(P_{\theta_t}^i(\mathbf{x}))+c\) for some shared constant, \(c\in\mathbb{R}\). Since softmax is invariant to adding a constant
to each logit, the optimal logit margin between any two classes \(i\) and \(j\) is: \[\Delta z_{ij}^* := z_s^{i*}-z_s^{j*} =
\log(P_{\theta_t}^i(\mathbf{x}))-\log(P_{\theta_t}^j(\mathbf{x})) = \log\left(\frac{P_{\theta_t}^i(\mathbf{x})}{P_{\theta_t}^j(\mathbf{x})}\right),\] which is finite since \(P_{\theta_t}^i(\mathbf{x})>0\) \(\forall\) \(i\).
In contrast, standard AT loss is defined by \(\mathcal{L}_{AT} = -\log(P_{\theta_s}^y(\mathbf{x_{adv}}))\), with \(y\) denoting the index of the true class label here. Hence, we have: \[\mathcal{L}_{AT} = -\log\left(\frac{e^{z_s^y}}{\sum_{j=1}^C e^{z_s^j}}\right) = \log\left(\sum_{j=1}^C e^{z_s^j-z_s^y}\right) = \log\left(1+\sum_{j\neq y} e^{z_s^j-z_s^y}\right).\] For any finite logits, we have \(e^{z_s^j-z_s^y}>0\), and hence \[1+\sum_{j\neq y} e^{z_s^j-z_s^y} > 1 \implies \mathcal{L}_{AT} > 0.\] Therefore, \(\mathcal{L}_{AT}\) has no finite
minimiser. Since we have established that \(\mathcal{L}_{AT} > 0\) for all finite logits, its global minimum is never attained. However, we can show that its infimum is exactly 0. Let \(m =
\displaystyle \min_{j \neq y} (z_s^y - z_s^j)\) denote the minimum logit margin of the correct class over all incorrect classes. We can bound the loss as follows: \[0 < \mathcal{L}_{AT} = \log\left(1+\sum_{j\neq y}
e^{-(z_s^y-z_s^j)}\right) \leq \log\left(1 + (C-1)e^{-m}\right),\] where \(C\) is the total number of classes. Taking the limit as the minimum margin grows to infinity (\(m \to
+\infty\)), we have: \[\lim_{m \to +\infty} \log\left(1 + (C-1)e^{-m}\right) = \log(1 + 0) = 0.\] Thus, \(\inf \mathcal{L}_{AT} = 0\). This infimum is not reachable for any finite
network weights, but is approached asymptotically as the correct-class logit margins tend to positive infinity. ◻
9 Additional Results↩︎
In this section, we present additional theoretical results complementary to those in the main text.
Proposition 5 (Teacher-margin transfer in logit space). Assume the teacher \(f_{\theta_t}\) predicts the true class \(y\) on the natural input \(\mathbf{x}\). Then, if: \[\left\|f_{\theta_s}(\mathbf{x_{adv}}) - f_{\theta_t}(\mathbf{x})\right\|_\infty < \frac{1}{2} \min_{j\neq y} [\mathbf{z}_{\theta_t}(\mathbf{x},y)]_j,\] it follows that \(\displaystyle \mathop{\mathrm{argmax}}_i [f_{\theta_s}(\mathbf{x_{adv}})]_i = y\).
Proof. First, define \(\delta :=\left\|f_{\theta_s}(\mathbf{x}_{\mathrm{adv}}) - f_{\theta_t}(\mathbf{x})\right\|_\infty\). Then, \(-\delta \le [f_{\theta_s}(\mathbf{x_{adv}})]_i -
[f_{\theta_t}(\mathbf{x})]_i \le \delta,\) for any class index \(i\). Which means, \([f_{\theta_s}(\mathbf{x_{adv}})]_y \ge [f_{\theta_t}(\mathbf{x})]_y - \delta\) and \([f_{\theta_s}(\mathbf{x_{adv}})]_i \le [f_{\theta_t}(\mathbf{x})]_i + \delta\) for any \(i \ne y\). Moreover: \[\begin{align} \implies
[f_{\theta_s}(\mathbf{x_{adv}})]_y - [f_{\theta_s}(\mathbf{x_{adv}})]_i &\ge [f_{\theta_t}(\mathbf{x})]_y - [f_{\theta_t}(\mathbf{x})]_i - 2\delta, \\ \implies [\mathbf{z}_{\theta_s}(\mathbf{x_{adv}},y)]_i &\ge
[\mathbf{z}_{\theta_t}(\mathbf{x},y)]_i - 2\delta.
\end{align}\] Hence, if \(\delta < \frac{1}{2} \min_{i\neq y} [\mathbf{z}_{\theta_t}(\mathbf{x},y)]_i\), then clearly \([\mathbf{z}_{\theta_t}(\mathbf{x},y)]_i - 2\delta > 0\),
which implies
\([\mathbf{z}_{\theta_s}(\mathbf{x_{adv}},y)]_i > 0\) and \(\displaystyle \mathop{\mathrm{argmax}}_i [f_{\theta_s}(\mathbf{x}_{\mathrm{adv}})]_i = y\). ◻
Lemma 3. Let \(\mathcal{L}_{AT}\) be bounded by some \(U \in \mathbb{R}\) and \(U > 0\), i.e., \[\mathcal{L}_{AT} = \mathcal{L}_\mathrm{CE}(f_{\theta_s}(\mathbf{x_{adv}}), y) = -\log(P_{\theta_s}^y) \le U,\] where \(y \in \{1, \dots, C\}\) is the true label index and \(P_{\theta_s}^y = \left[\mathrm{softmax}(f_{\theta_s}(\mathbf{x_{adv}}))\right]_y\). Then, for every \(j \neq y\), the corresponding true-vs-false logit margin satisfies \[z_s^y - z_s^j \ge -\log(e^U - 1).\]
Proof. Notice that AT loss can be written as: \[\mathcal{L}_{AT} = -\log\left(\frac{e^{z_s^y}}{\sum_{k=1}^C e^{z_s^k}}\right) = \log\left(1+\sum_{j\neq y} e^{z_s^j-z_s^y}\right).\] Now assume that \(\mathcal{L}_{AT} \le U\). Then, \[\begin{align} \log\left(1+\sum_{j\neq y} e^{z_s^j-z_s^y}\right) &\le U, \\ \implies 1+\sum_{j\neq y} e^{z_s^j-z_s^y} &\le e^U, \\ \implies \sum_{j\neq y} e^{z_s^j-z_s^y} &\le e^U-1. \end{align}\] Since each term in the sum is non-negative, it follows that for every \(j\neq y\), we have: \[\begin{align} e^{z_s^j-z_s^y} &\le e^U-1, \\ \implies z_s^j-z_s^y &\le \log(e^U-1), \\ \implies z_s^y-z_s^j &\ge -\log(e^U-1), \qquad \forall j\neq y, \end{align}\] as required. ◻
10 Additional Experiments↩︎
In this section, we evaluate the sensitivity of AD-CERT and CC-DIST [24] to teachers trained under different adversarial training methods and
schedules.
The sensitivity results from Table 5 suggest that the choice of teacher can affect the standard and adversarial accuracy of the resulting students, but the certified accuracy is comparatively
stable across teacher variants, particularly for AD-CERT. Here, short-cycle refers to the training procedure described in [24], where we use the SGD
optimiser with momentum set to 0.9, a 30-epoch training cycle with a cyclic learning rate, which linearly increases from 0 to 0.2 during the first half of the training, and then decreases back to 0. Long-cycle corresponds to the standard AT settings used
in the CTBench library [27], where PGD [14] and EDAC [43] models are trained for 240 epochs using the Adam
optimiser. Short-cycle teachers tend to slightly improve the certified results of AD-CERT and CC-DIST, which coincides with the observations of [24]
in their work. In general, CC-DIST appears to benefit more noticeably from short-cycle teachers, with AD-CERT being slightly less sensitive. Nevertheless, AD-CERT remains consistently stronger in certified and AA accuracy across all teacher protocols,
while CC-DIST tends to achieve slightly higher standard accuracy in most settings, consistent with previous observations in §4.2.
| Teacher | Teach. Std. [%] | Teach. AA [%] | Method | Std. [%] | AA [%] | Cert. [%] |
|---|---|---|---|---|---|---|
| 1-7 | 78.71 | 35.93 | AD-CERT | 53.17 | 35.95 | 35.21 |
| CC-DIST | 52.35 | 34.80 | 33.75 | |||
| 1-7 | 76.14 | 42.55 | AD-CERT | 53.90 | 36.52 | 35.61 |
| CC-DIST | 54.44 | 36.05 | 34.81 | |||
| 1-7 | 78.95 | 42.48 | AD-CERT | 53.49 | 36.46 | 35.60 |
| CC-DIST | 54.32 | 35.69 | 34.55 | |||
| 1-7 | 73.55 | 41.99 | AD-CERT | 53.69 | 36.69 | 35.84 |
| CC-DIST | 54.33 | 35.76 | 34.68 |
11 Pseudo-code↩︎
In this section, we outline the pseudocode for the training procedure of AD-CERT (§3.1) in Algorithm 4.
12 Experimental Setup↩︎
12.1 Dataset↩︎
We conduct experiments on MNIST [37], CIFAR-10 [38], and TinyImageNet [39]. These datasets are open-source and freely available, with unspecified licenses. We follow the data preprocessing directly available in the CTBench library [27]. No preprocessing is applied to MNIST. CIFAR-10 and TinyImageNet are normalised using the dataset mean and standard deviation, and random horizontal flips are applied. For CIFAR-10, additional random cropping to \(32 \times 32\) is applied after zero-padding each image by 2 pixels on all sides. For TinyImageNet, random cropping to \(64 \times 64\) is applied after zero-padding each image by 4 pixels on all sides. We train on the corresponding training sets and certify on the validation sets, following common practice in the certified training literature [20]–[24], [27].
12.2 Model Architectures↩︎
We use the standard CNN7 architecture, a convolutional network consisting of seven convolutional and linear layers. Each layer, except for the final linear layer, is followed by Batch Normalisation and a ReLU activation. This architecture has been shown to perform consistently well across settings [20], [27], and is therefore widely adopted in the certified training literature [20]–[24], [27]. For TinyImageNet, we double the stride of the final convolutional layer to reduce computational cost.
12.3 Training Details↩︎
12.3.0.1 Implementation
We implement AD-CERT in CTBench using PyTorch. The training loss follows the CTBench decomposition into a natural loss, a robust loss, and regularisation terms. For AD-CERT, the robust loss is given by Equation 8 , where the AD term uses a fixed adversarially trained teacher, and the certified term is computed with IBP bounds. Unless stated otherwise, the teacher and student use the same CNN7 architecture.
12.3.0.2 Initialisation
Adversarial teacher models are initialised by Kaiming uniform [44], while certified student models are initialised using the IBP initialisation from [20].
12.3.0.3 Training Schedule
We follow the standard CTBench training schedule. Certified models are first trained for one epoch with \(\varepsilon = 0\), and then use a warm-up phase where \(\varepsilon\) is smoothly increased from \(0\) to the target value. The warmup phase is 20 epochs for MNIST with \(\varepsilon = 0.1\) and \(\varepsilon = 0.3\), 80 epochs for CIFAR-10 with \(\varepsilon = \frac{2}{255}\), 120 epochs for CIFAR-10 with \(\varepsilon = \frac{8}{255}\), and 80 epochs for TinyImageNet with \(\varepsilon = \frac{1}{255}\). We use the IBP regularisation proposed by [20], with weight \(0.5\) on MNIST and CIFAR-10, and \(0.2\) on TinyImageNet, during warmup. In total, we train for 70 epochs on MNIST, 160 epochs on CIFAR-10 with \(\varepsilon = \frac{2}{255}\), 240 epochs on CIFAR-10 with \(\varepsilon = \frac{8}{255}\), and 160 epochs on TinyImageNet.
12.3.0.4 Optimisation
In general, we follow the exact optimisation setup from CTBench. We use Adam [45] with a learning rate of \(5 \times 10^{-4}\). The learning rate is decayed by a factor of \(0.2\) at epochs 50 and 60 for MNIST, epochs 120 and 140 for CIFAR-10 with \(\varepsilon = \frac{2}{255}\), epochs 200 and 220 for CIFAR-10 with \(\varepsilon = \frac{8}{255}\), and epochs 120 and 140 for TinyImageNet. We use a batch size of 256 for MNIST and 128 for CIFAR-10 and TinyImageNet. Gradients are clipped to \(10\) in \(\ell_2\) norm before every optimiser step. No weight decay is applied, and \(L_1\) regularisation is applied only to the weights of linear and convolutional layers. Batch normalisation follows the CTBench population-statistics setting. We apply Stochastic Weight Averaging (SWA) using the settings in accordance with MTL-IBP’s training setup in CTBench since both methods use an unsound robust objective, making direct model selection on the robust loss impractical.
12.3.0.5 Teacher Models
We use the pre-trained PGD checkpoints provided by CTBench as teacher models, except for CIFAR-10 with \(\varepsilon = \frac{8}{255}\), where our ablation teacher sensitivity analysis from Table 5 showed that a 30-epoch short-cycle PGD model trained with a cyclic learning-rate schedule exactly as described by [24] alongside an EDAC [43] step size of \(0.3\), gives marginally better performance. All teachers are kept fixed during AD-CERT training.
12.3.0.6 Tuning of Hyperparameters
In general, we perform minimal hyperparameter tuning due to the computational overhead of running complete verification. We ran initial experiments following the exact hyperparameters and settings used in CTBench [27] for the MTL-IBP [23] method. For the tuning of \(\alpha\), we use a manually selected search range guided by some reported quantities in the CTBench library, namely, unstable ReLU ratio, IBP certification rate, and standard, adversarial, and certified accuracy. These metrics help indicate whether the objective places too much or too little weight on the IBP branch, allowing us to adjust \(\alpha\) accordingly. We find that slightly reducing \(\alpha\) is beneficial on MNIST at \(\varepsilon=0.1\) and on TinyImageNet at \(\varepsilon=\frac{1}{255}\). For TinyImageNet, we also observe improvements from increasing \(w_{\mathrm{rob}}\) and enlarging the PGD attack region used in the empirical branch. The final hyperparameters used for AD-CERT across all settings are reported in Table 6.
| MNIST | CIFAR-10 | TinyImageNet | |||
|---|---|---|---|---|---|
| 2-3 (lr)4-5 (lr)6-6 Hyperparameter | \(0.1\) | \(0.3\) | \(\frac{2}{255}\) | \(\frac{8}{255}\) | \(\frac{1}{255}\) |
| \(L_1\) regularisation | \(1\times10^{-6}\) | \(1\times10^{-6}\) | \(3\times10^{-6}\) | 0.0 | \(5\times10^{-5}\) |
| \(w_{\mathrm{rob}}\) | 0.7 | 1.0 | 1.0 | 1.0 | 0.9 |
| IBP coefficient (\(\alpha\)) | \(7.5\times10^{-3}\) | 0.5 | 0.01 | 0.5 | \(5\times10^{-3}\) |
| Train \(\varepsilon\) | 0.2 | 0.3 | \(\frac{1}{255}\) | \(\frac{8}{255}\) | \(\frac{1}{255}\) |
| PGD steps | 10 | 1 | 8 | 1 | 1 |
| Attack range scale | 1.0 | 1.0 | 2.0 | 1.0 | 2.0 |
12.4 Certification Details↩︎
Certification is performed within CTBench [27] using its wrapper for the \(\alpha,\beta\)-CROWN verification library. For each example, we first check whether the model predicts the correct clean label. Incorrectly classified examples are marked as uncertified. For correctly classified examples, we run AutoAttack [15] and if an adversarial counterexample is not found, we attempt certification using a sequence of progressively stronger verifiers. Namely, IBP [18], CROWN [7], \(\alpha\)-CROWN [10], and finally, the complete \(\alpha,\beta\)-CROWN branch-and-bound verifier [11]. Each stage is only invoked when the previous stage fails to certify the example. We use a timeout of 400 seconds for the \(\alpha,\beta\)-CROWN branch-and-bound stage and an overall timeout of 1000 seconds per example.
12.5 Computational Setup & Cost↩︎
Table 7 reports the training and certification runtime of AD-CERT under the training and certification setups described in Sections 12.3 and 12.4, respectively. All experiments were run on an internal cluster with access to NVIDIA V100, A100, A100 MIG slices, and H100 GPUs. Training jobs were run on either 2g.20GB or 3g.40GB A100 MIG slices, using 8 CPU
cores. For certification, we used MIG slices in most cases, but occasionally used full A100 or H100 GPUs for harder settings, such as CIFAR-10 with \(\varepsilon = \frac{2}{255}\) and TinyImageNet. For clarity, Table 7 reports runtimes for experiments run on a 3g.40GB A100 MIG slice. Training experiments used 64GB of RAM, while certification experiments used 128GB. The training times range from \(\sim\) 1 hour on the simplest MNIST setting, to \(\sim\) 1 day on TinyImageNet. Certification times range from \(\sim\) 12 minutes to over 3 days.
Additionally, we report the training cost complexity associated with an AD-CERT student, assuming a pre-trained teacher, compared to previous adversarial and certified training methods [14], [18], [21]–[24], [43] in Table 8.
On a high level, AD-CERT corresponds to performing IBP and PGD, with the small added cost of a single forward pass of a teacher model that is detached from training. Note that Table 8 is a direct extrapolation of the
table provided in the original CTBench paper [27] with added entries for AD-CERT and CC-DIST.
| Dataset | \(\varepsilon\) | Train Time (seconds) | Certification Time (seconds) |
|---|---|---|---|
| MNIST | \(0.1\) | \(1.20 \times 10^{4}\) | \(7.67 \times 10^2\) |
| \(0.3\) | \(4.08 \times 10^{3}\) | \(5.32\times 10^4\) | |
| 2-4 | \(\frac{2}{255}\) | \(2.66 \times 10^{4}\) | \(2.61 \times 10^{5}\) |
| \(\frac{8}{255}\) | \(1.59 \times 10^{4}\) | \(3.58 \times 10^{4}\) | |
| 2-4 TinyImageNet | \(\frac{1}{255}\) | \(8.97 \times 10^{4}\) | \(1.54 \times 10^5\) |
| Method | Training cost per batch | Details |
|---|---|---|
| Standard | \(T\) | Forward + backward |
| PGD / EDAC | \((M+1)T\) | \(M\) attack steps + standard loss computation |
| IBP | \(2T\) | Lower and upper bounds propagation |
| SABR | \((M+2)T\) | IBP + PGD |
| MTL-IBP | \((M+2)T\) | IBP + PGD |
| AD-CERT\(^\dagger\) | \((M+2)T + F\) | IBP + PGD + detached teacher forward |
| CC-DIST\(^\dagger\) | \((M+2)T + F_h\) | CC-IBP + PGD + detached teacher forward on feature split |
| TAPS | \(2t + K(M+1)(T-t)\) | IBP for first split and PGD for second split for each class |
| STAPS | \(2t + K(M+1)(T-t) + (M+1)T\) | TAPS + PGD |
| Legend | ||
| \(T\) | Time cost for standard training, including forward + backward pass | |
| \(F\) | Time cost for a forward pass through the teacher network | |
| \(F_h\) | Time cost for a forward pass through the teacher feature extractor network | |
| \(M\) | Number of adversarial attack steps, including repeats | |
| \(K\) | Number of classes | |
| \(t\) | Time cost for standard training in the first network split in TAPS | |
| \(^\dagger\) Added entries from original table in CTBench [27]. | ||
13 CC-IBP Distillation (CC-DIST)↩︎
[24] formulate the CC-DIST loss function as a linear combination of an expressive robust distillation loss in the feature space, \(\mathcal{R}_{f_\theta}(\alpha; \mathbf{x}, y)\), and their prior expressive loss formulation, CC-IBP [23]. Let \(f_\theta: \mathbb{R}^{d} \mapsto{\mathbb{R}^C}\) be a classification model as expressed earlier in §2. Then, we can rewrite \(f_\theta\) as a composition of
a feature extractor \(h_\theta: \mathbb{R}^{d} \mapsto{\mathbb{R}^k}\) and a classification head \(g_\theta: \mathbb{R}^{k} \mapsto{\mathbb{R}^C}\), yielding \(f_\theta = g_\theta \circ h_\theta\) (note that we abuse notation of \(\theta\) for brevity). Then, the expressive robust distillation loss over the feature space is defined by:
\[\mathcal{R}_{f_\theta}(\alpha; \mathbf{x}, y) := \sum^k_{i=1} \max \left\{\left([\overline{h}_\theta(\alpha; \mathbf{x})]_i - [h_{\theta_t}(\mathbf{x})]_i\right)^2,
\left([\underline{h}_\theta(\alpha; \mathbf{x})]_i - [h_{\theta_t}(\mathbf{x})]_i\right)^2\right\}, \label{eq:rob-dist-loss-ccdist}\tag{11}\] where \(\overline{h}_\theta(\alpha; \mathbf{x}) :=
(1-\alpha)h_\theta(\mathbf{x_{adv}}) + \alpha \overline{h}_\theta(\mathbf{x})\) and \(\underline{h}_\theta(\alpha; \mathbf{x}) := (1-\alpha)h_\theta(\mathbf{x_{adv}}) + \alpha \underline{h}_\theta(\mathbf{x})\), with
\(\overline{h}_\theta(\mathbf{x})\) and \(\underline{h}_\theta(\mathbf{x})\) denoting the IBP upper and lower bounds of \(h_\theta(\mathbf{x})\),
respectively. Note that \(h_{\theta_t}\) denotes the feature extractor of a pre-trained PGD teacher model.
Now, let \(\beta\) be the distillation coefficient, determining the relative weight of \(\mathcal{R}_{f_\theta}(\alpha; \mathbf{x}, y)\) from Equation 11 . Omitting any regularisation, the training loss for CC-DIST takes the following form: \[\mathcal{L}^{\text{CC-DIST}}_{f_{\theta}}(\alpha, \beta; \mathbf{x}, y) :=
\;\mathcal{L}^{\text{CC-IBP}}_{f_{\theta}}(\alpha; \mathbf{x}, y) + \beta \;\mathcal{R}_{f_{\theta}}(\alpha; \mathbf{x}, y),\] where \(\mathcal{L}^{\text{CC-IBP}}_{f_{\theta}}(\alpha; \mathbf{x}, y)\) is the CC-IBP
loss from [23]. Denoting \(k\) as the dimensionality of the feature space, \(\beta =
5/k\) was used for the majority of experiments.
References↩︎
Corresponding author.↩︎