1 Introduction↩︎

Deep neural networks have increasingly been used in many applications such as autonomous driving. Unlike the controlled environments in which these models are trained, test-time inference presents new challenges including noisy real-world data and latency and memory constraints. These challenges have led to recent efforts in network robustness and compression, but through largely separate lines of work.

Robustness to input variation unseen during training is especially important for deployed deep learning models in safety-critical applications such as autonomous driving, where, for instance, artifacts such as dirt or snow may obscure the camera image. More generally, in computer vision applications, this input variation falls into categories including distribution shifts, image corruptions, adversarial attacks, and label noise [1]–[6]. In attempts to address these challenges, the vision community has collected and released datasets to assess model robustness [1], [7] and investigated and improved performance on a variety of types of robustness [8]–[11]. Despite significant progress in this area, most existing robustness work focuses on dense networks [9], [12].

Running large models imposes substantial computational burdens that can be mitigated via model compression methods. Given the observation that neural networks often include redundant computation at inference, this line of work aims to reduce network inference costs through techniques such as knowledge distillation  [13]–[15], quantization [16]–[19], pruning [20]–[25], and network adaptation [14], [26], [27]. Here we focus on pruning due to its simple procedure for network speedup. The pruning process retains neurons deemed important over the training distribution, so it is unsurprising that pruned networks can have reduced out-of-distribution (OOD) generalization, as observed in [28], [29]. The effort to jointly address robustness and compression remains an open problem.

The goal of our work is to produce compact and robust neural networks. In particular, we emphasize robustness to input variations that are unseen at training time, in contrast with methods such as robust pruning [30]–[32] which assume access to the input variations at training time. It might seem as if the goals of sparsity and robustness are at odds since one aims to exploit the current dataset and task for extreme compactness whereas the other strives for maximal generalization. However, our method leverages a new flatness-based optimization procedure that both primes the network for pruning and improves network robustness. Specifically, flatness can mitigate the loss in performance during the pruning procedure and regularize the network towards improved robustness.

We introduce Adaptive Sharpness-Aware Pruning (AdaSAP), a three-step algorithm that prunes a network so that it is robust to OOD inputs. (1) We introduce the novel adaptive weight perturbations which are incorporated into the optimization procedure to adaptively penalize sharpness in order to prepare the network for pruning, (2) apply any pruning method to the model, and (3) continue training the model while uniformly penalizing sharpness across the network to encourage robustness.

AdaSAP significantly improves the relative robustness over prior pruning art, seen in Figure 1. As models are pruned (with SOTA methods) more, their performance on corrupted images suffers disproportionately as compared to their validation performance. This highlights the lack of robustness preservation in recent SOTA pruning methods and AdaSAP’s success at reducing this degradation².

Our contributions can be summarized as follows:

• We introduce AdaSAP, a sharpness-aware pruning and fine-tuning process to jointly optimize for sparsity and robustness in deep neural networks.

• We propose novel adaptive weight perturbations that prepare a network for both pruning and robustness to OOD inputs. This strategy actively manipulates the flatness of the loss surface so as to minimize the impact of removing neurons with lower importance scores.

• We demonstrate state-of-the-art performance compared to prior art by noticeable margins across a wide range of setups, covering (i) two tasks (classification and detection), (ii) two OOD types (image corruption and distribution shift), (iii) four networks (ResNet50 and MobileNet V1/V2 for classification, SSD512 for detection), (iv) two pruning paradigms (parameter-based and latency-based), and (v) sparsity ratios spanning from around 20% to 80%.

• We present a detailed evaluation of robustness and compactness and show, as a first encouraging attempt, that both goals can be unified through the lens of sharpness, with analysis to encourage the community to continue pursuing this direction.

2 Related Works↩︎

Robustness. Researchers and practitioners care about many types of robustness, including adversarial robustness, robustness to distribution shift, robustness to image corruptions such as weather effects, and robustness to label noise or data poisoning [1]–[6], [10], [11]. Several prior works examine the adversarial robustness of sparse networks [30], [33]. Simultaneously, work such as [34] suggests that adversarially robust networks have flatter minima.

In this work we focus on robustness to corrupted images [1], [7] due to its real-world applications such as autonomous driving. [35] shows that sparse subnetworks can outperform the original dense model on clean and robust accuracy. However, almost all current pruning methods still lead to worse performance on OOD data than dense networks, a result we replicate in Figure 1 [28], [29].

Efficiency. Efforts to improve network efficiency include pruning, distillation, quantization, and adaptation  [13], [14], [16], [17], [20], [22]–[27], [36]. Two broad categories of pruning are structured and unstructured pruning. While unstructured pruning removes individual weights and can retain strong performance at high sparsities, structured pruning removes larger elements in the network, such as convolutional channels, and often allows for direct size or latency reduction of the pruned model [24], [25], [37]. Unstructured pruning dates back to Optimal Brain Damage, but the Lottery Ticket Hypothesis has incited recent interest in the topic and spurred developments such as SNIP and GraSP  [38]–[44]. In the current work, we focus on structured channel-wise pruning.

Flat minima. The relationship between flat minima and generalization has been studied for many years [45]–[54]. Flat minima are often correlated with improved generalization, but the investigations into the extent of this correlation and the underlying causal relationship are still ongoing. Also, flatness is metric-dependent and sharp minima can generalize well [55]. Sharpness-Aware Minimization (SAM) [56] is a way to optimize for finding flatter minima and leads to better generalization. Work which builds upon SAM improves its efficiency [57]–[59], modifies the optimization for improved performance [60], applies it for compression in NLP applications [61] and as a pre-pruning optimizer [62], and modifies the objective for unstructured pruning [63]. Stochastic weight averaging (SWA) is another method that has been used to optimize for flatter minima [52]–[54]. The present work is inspired by SAM and its use in model compression [56], [61] but differs in its use of adaptive weight perturbations and its emphasis on robustness. In this work we use the formulation introduced in ASAM [60] due to its improved performance over that of SAM.

3 The AdaSAP Method↩︎

Flat minima are minima within relatively large regions of the parameter space that have low loss. It has previously been observed that optimizing for flatter minima improves generalization and adversarial robustness in deep networks and generalization in sparse networks [34], [56], [61]. Based on these observations, we hypothesize that optimizing for flatter minima during pruning can also enhance robustness. The main objective of our design is to find flat minima in order to produce models that are simultaneously prunable and robust. We introduce a new method, Adaptive Sharpness-Aware Pruning (AdaSAP). Figure 2 and Algorithm 4 detail the following three step procedure:

(1) Adaptive Weight Perturbations. During warmup, we ensure that the neurons that will be pruned, are regularized to lie in flat regions so that they will not cause the model’s performance to suffer too much after pruning.

(2) Neuron Removal. We conduct structured channel pruning and remove neurons according to any pruning criteria. Here we use magnitude pruning [20]. During pruning, the model will preserve much of its performance since unimportant neurons have already been situated in flatter loss regions.

(3) Robustness Encouragement. While training the pruned model to convergence, we enforce flat minima, this time uniformly (i.e., non-adaptively) across the entire network to promote robustness. This step is compatible with any flatness-based optimizer. Here we use ASAM [60].

3.1 Adaptive Weight Perturbation↩︎

Flatness-informed pruning. Our procedure performs gradient updates which are informed by the local flatness in order to best prepare the network for pruning. We adapt the regularization per neuron based on the importance, or likelihood of that neuron to be pruned. Pruning can be viewed as a special case of directional robustness, in which the weight and activation of a pruned neuron are set to zero. When the loss landscape is flat, the pruning process will incur only a small impact on the performance of the model. This formulation indicates that pruning and robustness share a common characterization through the perspective of sharpness, as demonstrated in Figure 3.

Taking inspiration from the uniform \({\epsilon}\)-ball perturbation based method in [56] and its scale-invariant version in [60], among several other approaches for achieving flatness in dense models [53], [64], we perturb neurons according to their importance to better prepare the network for pruning. We follow the intuition that important weights are worth adding some sharpness to our model, while unimportant weights can have stronger regularization. This idea is supported by [24], which shows that pruning neurons in flat regions hurts the loss less than pruning neurons in sharp regions. We adapt the size of allowable perturbations based on the likelihood of each weight to be pruned: weights with low importance scores have large perturbation balls which enforce that they lie within flatter regions and vice versa.

Setup. Following this intuition, we derive the gradient update for training. Consider a network whose parameters are grouped into \(I\) partitions, where each partition \(i\) includes the set of weights \(\mathbf{w}_i\). In our setting, since we consider channel-wise pruning, each channel constitutes a partition. For simplicity, throughout this paper we refer to a channel-wise group of weights as a neuron. We use \(\mathbf{w}_i\) to refer to a single neuron and \(\boldsymbol{w}\) to refer to the collection of all neurons \(\{\mathbf{w}_i\}_{i=1}^I\).

Perturbation regions. Consider training loss \(L_S(\mathbf{w})\) of the network over the training dataset \(S\). Consider a set of weight perturbation ball radius sizes \(\boldsymbol{\rho}\), with each \(\rho_i\) corresponding to the size of allowable perturbations for neuron \(\mathbf{w}_i\). Our goal is to optimize for a network in which unimportant neurons lie in flatter minima prior to pruning via the following objective \[\min_{\mathbf{w}} \max_{\{{\epsilon}: \|T_{\mathbf{w}}^{-1}{\epsilon}_i\| \leq \mathbf{\rho}_i\}_{i=1}^I} L_S(\mathbf{w} + \mathbf{{\epsilon}}). \label{objective}\tag{1}\] where \(T_{\mathbf{w}}\) and its inverse \(T_{\mathbf{w}}^{-1}\) are transformations that can be applied during optimization so as to reshape the perturbation region (i.e., not necessarily a ball). [60] showed that this strategy allows for greater perturbation exploration and leads to improved performance. \(\rho_i\) values are based on computing an importance score for each neuron, similar to a pruning criterion. Consider a neuron importance score function \(\psi(\cdot)\), such as the \(\ell_2\) norm of the neuron weight or other scores that are some function of weights, gradients, and higher order terms [24], [25], [38], [43], [44]. Given each neuron score \(s_i = \psi(\mathbf{w}_i)\), we can now compute the perturbation ball size \(\rho_i\), which projects \(s_i\) to lie within \((\rho_{min}, \rho_{max})\) \[\rho_i = \rho_{\max} - \frac{s_i - s_{\min}}{s_{\max} - s_{\min}} (\rho_{\max} - \rho_{\min}). \label{alpha95mapping}\tag{2}\] Scores \(s_{\min}\) and \(s_{\max}\) are obtained empirically at each gradient step, but \(\rho_{\max}\) and \(\rho_{\min}\) are hyperparameters that are set ahead of time and we find them to scale across various experiment settings (further details in Section 4 and Appendix 6). Neuron importance estimates \(\psi(\cdot)\) can be evaluated during training with little additional computational overhead.

Optimal gradient update. Our enhanced gradient update optimizes for finding a minimum that also lies in a region of low loss. Beginning with our objective in Eq. 1 , we perform a series of approximations in order to find the most adversarial perturbations \(\boldsymbol{\epsilon}= \{{\epsilon}_i\}\) (i.e., highest loss near the minimum) which we can apply to the neurons \(\boldsymbol{w} = \{\mathbf{w}_i\}\) in our network in order to obtain a flatness-informed gradient update.

We use a first order Taylor expansion around each neuron \(i\) with respect to \(\boldsymbol{\epsilon}\) around \(0\) (in line with [56]) to approximate the inner maximization in Eq. 1 . This gives us \(\boldsymbol{\epsilon}^*\), the optimal \(\boldsymbol{\epsilon}\) we can use to inform our gradient update. Define \(\tilde{\epsilon}_i = T_{\mathbf{w}}^{-1}\epsilon_i\). For simplicity we denote the conditional loss due to weight perturbations as the function \(L_{S, \mathbf{w}}(\cdot)\). Then we derive our optimal update as follows for each neuron: \[\begin{align} \tilde{{\epsilon}}^*_i &= \mathop{\mathrm{arg\,max}}_{\|\tilde{{\epsilon}}_i\|_2 \leq \rho_i} L_{S, \mathbf{w}} (\mathbf{w}_i + T_{\mathbf{w}}\tilde{{\epsilon}}_i), \nonumber \\ \nonumber &\approx \mathop{\mathrm{arg\,max}}_{\|\tilde{{\epsilon}}_i\|_2 \leq \rho_i} L_{S, \mathbf{w}} (\mathbf{w}_i) + \tilde{{\epsilon}}_i^\top T_{\mathbf{w}} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i), \\ &= \mathop{\mathrm{arg\,max}}_{\|\tilde{{\epsilon}}_i\|_2 \leq \rho_i} \tilde{{\epsilon}}_i^\top T_{\mathbf{w}} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i), \end{align}\] \(\|\cdot\|_2\) being the \(\ell_2\) norm. We can then approximate \(\tilde{\boldsymbol{\epsilon}}^*\) by rescaling the gradient associated with each neuron so that its norm is \(\rho_i\) through \[\tilde{{\epsilon}}^*_i = \rho_i \frac{ \;\mathrm{sign} (\nabla_{w_i} L_{S, \mathbf{w}}(\mathbf{w}_i)) |T_{\mathbf{w}} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)| }{ (\|T_{\mathbf{w}} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)\|_2^2)^{1/2} }.\] Since we were originally approximating \(\tilde{\epsilon}_i = T_{\mathbf{w}}^{-1}\epsilon_i\), we recover our approximation of \(\mathbf{{\epsilon}}^*\) as \[\hat{{\epsilon}}_i= \rho_i \frac{T_{\mathbf{w}}^2 \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)}{\|\nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)\|_2}. \label{eps95hat}\tag{3}\] Approximating the optimal gradient update. We then find an approximation to our desired gradient. Referring to Eq. 1 , we can find the gradient of the inner maximization in order to derive a sharpness-aware gradient update. We use the inner maximization and plug in our approximation \(\hat{{\epsilon}}\) for each individual neuron during adjacent pruning steps, defined as \[\begin{align} \nabla_{\mathbf{w}_i} \max_{\|T_{\mathbf{w}}^{-1}{\epsilon}_i\|_2 \leq \rho_i} L_{S, \mathbf{w}}(\mathbf{w}_i + {\epsilon}_i) &\approx \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i + \hat{{\epsilon}}_i) \nonumber \\ &= \frac{d(\mathbf{w}_i + \hat{{\epsilon}}_i)}{d\mathbf{w}_i} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i) |_{\mathbf{w}_i + \hat{{\epsilon}}_i} \nonumber \\ &= \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)|_{\mathbf{w}_i + \hat{{\epsilon}}_i} \nonumber + \frac{d\hat{{\epsilon}}_i}{d\mathbf{w}_i} \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)|_{\mathbf{w}_i + \hat{{\epsilon}}_i} \nonumber \\ &\approx \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)|_{\mathbf{w}_i + \hat{{\epsilon}}_i}, \end{align}\] where we drop the second order terms for efficiency. This leads us to our final gradient update approximation between adjacent pruning steps as \(\nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i) \approx \nabla_{\mathbf{w}_i} L_{S, \mathbf{w}}(\mathbf{w}_i)|_{\mathbf{w}_i + \hat{{\epsilon}}_i}\).

We use this gradient update in place of the standard gradient update during the warmup phase of our pruning procedure before neuron removal. This sets up our network so that when pruning begins, we have a model that is closer to a flat minima. Optimization details are in Algorithm [adasam95alg95one95epoch].

3.2 Neuron Removal↩︎

We focus on structured (channel-wise) pruning since it leads to models that can take advantage of direct computational resource savings on GPUs, leading to inference speedup [14], [21], [25].

In this stage of the procedure, we remove unimportant neurons according to any scoring function \(\phi(\cdot)\) that measures neuron saliency. This may be the same or different from the scoring function \(\psi(\cdot)\) used in determining adaptive weight perturbation ball sizes during the first step. AdaSAP works with a range of pruning methods as we show later in Section 4.

3.3 Robustness Encouragement↩︎

In the third and final phase, we focus on optimizing our model for robust performance. We regularize our weights uniformly since we have completed pruning and now want to enforce robustness for the entire network. This is illustrated in Figure 3 in the plot on the right: the network loss relative to a flatter neuron will stay more stable in the presence of corruptions, whereas the loss relative to a sharper neuron could fluctuate significantly. This portion of the procedure could be instantiated with any sharpness-based method that optimizes for an overall flatter minima.

3.4 Final Comments↩︎

We emphasize that AdaSAP is not a pruning method but rather an optimization paradigm that performs robustness-aware pruning. Our method can be used in conjunction with any existing pruning criteria. In Section 4 we show that AdaSAP (in conjunction with magnitude pruning) generally outperforms existing SOTA pruning methods. As new pruning techniques arise, they could be enhanced via AdaSAP in order to obtain increased performance, particularly robust performance.

Additionally, we note that our method differs from robust pruning methods such as [30]–[32] since we consider a setting in which the model may be exposed to novel types of corruptions at test time. Therefore, we do not want to include, i.e. ImageNet C or ImageNet V2 images in our training set. We demonstrate results on these OOD datasets in an attempt to demonstrate the versatility of our method: despite only training on ImageNet data, we can still expect the models to be robust to unseen input variation.

4 Experiments & Results↩︎

4.1 Experiment Details↩︎

We describe the basics of our experimental setup here. Additional experiment details and confidence intervals can be found in Appendix 6.

Datasets. For image classification, we train on ImageNet-1K  [65] and additionally evaluate on ImageNet-C [1] and ImageNet-V2 [7]. For object detection, we use the Pascal VOC dataset [66]. To assess robustness, we create a Pascal VOC-C dataset by applying ImageNet-C-style corruptions to the test set.

Pruning criteria. Our proposal is agnostic to the particular importance criteria \(\phi\) used to prune the network. We evaluate our method on two types of pruning: parameter-based and latency-based. We choose to group our methods into these two categories due to the different goals of pruning methods. Some methods, such as magnitude pruning, seek to reduce the number of parameters to produce a small model. Other methods, such as HALP [67], seek to produce a model with the fastest latency speedup, regardless of the number of parameters.

In both cases, we perform structured pruning, that is, pruning channels rather than individual parameters. We refer to our two methods as \(\text{AdaSAP}_P\) and \(\text{AdaSAP}_L\), to denote the parameter-specific method and the latency-specific method, respectively. We use \(\ell_2\) norm magnitude pruning for \(\text{AdaSAP}_P\) and HALP for \(\text{AdaSAP}_L\). Details of the pruning schedule are in Appendix 6.

Metrics. For image classification, we report the Top1 accuracy on each dataset and two robustness ratios, defined as the ratio in robust accuracy to validation accuracy: \(R_\text{C} = \text{acc}_{\text{C}} / \text{acc}_{val}\) and \(R_{\text{V2}} = \text{acc}_{\text{V2}} / \text{acc}_{val}\). Improvement in these ratios indicates that we are closing the gap between robust and validation performance. For object detection, we evaluate the model on Pascal VOC and VOC-C. We report mean average precision (mAP) and a robustness ratio \(R_\text{C} = \text{mAP}_{\text{C}} / \text{mAP}_{val}\).

Baselines. We primarily focus on comparing to two SOTA baselines which we re-run in our experiments, Taylor pruning for parameter-based pruning [24], and HALP for latency-based pruning [25]. Taylor pruning assigns a score to each neuron based on a Taylor series approximation of how much the loss would change if the neuron were removed. HALP is a pruning method that optimizes for improving latency. We additionally run results on magnitude pruning [20], to demonstrate the improvements of AdaSAP\(_P\) over its closest baseline. We also compare to the results cited in several other pruning methods such as GR-eg [68], EagleEye [69], ABCPruner [70], and SMCP [71].

4.2 ImageNet Classification↩︎

Parameter reduction. Table 1 shows how AdaSAP compares favorably to other pruning methods at a variety of pruning ratios for ResNet50. MobileNet V1 and V2 results show a similar trend and are included in Appendix 7. First, AdaSAP has the best overall \(R_\text{C}\) and \(R_{\text{V2}}\) ratios among all pruning methods. This means that our method helps to close the gap between robust performance and standard validation performance. Additionally, our method consistently outperforms comparisons on both ImageNet validation performance and on robust performance (via ImageNet C and ImageNet V2) while at the same or even smaller compressed model size.

Latency Reduction. In a second set of experiments, we focus on latency constraints. To this end, we consider the most recent hardware-aware pruning method HALP (NeurIPS’22 [67]), which incorporates latency into the metric to measure the saliency of each neuron. Table 2 shows that using AdaSAP\(_L\) outperforms prior works both in terms of the \(R_\text{C}\) and \(R_\text{V2}\) robustness ratios, and also in terms of standard validation performance and performance on OOD datasets.

4.3 Object Detection↩︎

We show object detection results in Table 5. Across two pruning levels, AdaSAP consistently outperforms the HALP baseline by noticeable margins. More details can be found in Appendix 6.

4.4 Ablations↩︎

Sensitivity to importance metrics. In our experiments, we considered \(\ell_2\) norm, or magnitude, as the saliency metric for pruning since this led to the best performance. However, our method is flexible enough to work with other metrics such as Taylor importance as seen in Table [tab:adasap95taylor].

Adaptive perturbations. In order to determine the importance of using adaptive weight perturbations, we compare our method to the use of uniform perturbations throughout all three steps, which is equivalent to using SAM. In Table [tab:ablation], we compare three different setups: Taylor pruning with SGD as a baseline, Taylor pruning with SAM, and AdaSAP\(_P\). All three settings are fine-tuned for \(90\) epochs. Our method outperforms both SGD and SAM at the same or smaller compressed size. This demonstrates that our method’s adaptive perturbations during the warmup phase are helping the network achieve a more prunable state. See Appendix 8 for more extensive results and further analysis of the choice of uniform perturbations during robustness encouragement.

4.5 Sharpness Analysis↩︎

Consider the sharpness defined in  [56] as \(\max_{\|{\epsilon}\|_2 \leq \rho} L_S(\mathbf{w} + {\epsilon}) - L_S(\mathbf{w}).\) This measures the maximum amount that the loss could change if the weights are perturbed within a ball of radius \(\rho\). We measure sharpness directly before and after pruning, to evaluate how sharpness prepares the network to be optimally pruned, as well as how pruning affects the model sharpness. Table [tab:sharpness] shows that using AdaSAP leads to flatter models both before and after pruning. In line with previous results that found an association between flatness and generalization, this result can help explain why our models have better generalization and robust generalization performance.

4.6 Limitations↩︎

One limitation is that AdaSAP requires twice the training time due to the two backward passes necessary for optimization. A 90 epoch pruning and finetuning run on 8 V100 GPUs to reduce a ResNet50 network to \(0.2 \times\) its size takes takes 32 hours and 12 minutes for AdaSAP\(_P\) while SGD with magnitude pruning takes 16 hours and 53 minutes. Recent strategies  [57]–[59] could be used to reduce this additional overhead. Another limitation is that our analysis is based on existing robustness datasets that don’t fully capture the real world corruptions encountered in autonomous driving. Despite a focus in the present work on applying pruning (and fine-tuning) to an existing high performing dense model, AdaSAP shows promise in also being used to prune models from scratch. Finally, although we focus here on the more challenging structured pruning, we believe AdaSAP could also benefit unstructured pruning.

5 Conclusion↩︎

We introduce the Adaptive Sharpness-Aware Pruning method (AdaSAP) which optimizes for both accuracy and robust generalization during the pruning procedure. The method consists of three steps: application of our novel adaptive weight perturbations, pruning, and flatness-based robustness encouragement. AdaSAP outperforms a variety of SOTA pruning techniques on clean and robust performance and relative robustness in both image classification and object detection.

6 Experiment Details↩︎

6.1 Datasets↩︎

We conduct image classification experiments on the ImageNet dataset [65]. To evaluate the generalization ability of pruned models, we also evaluate models on ImageNet-C [1] and ImageNet-V2 [7] datasets. The former consists of the validation images from the ImageNet dataset, but with nineteen types of corruptions applied with five different levels of severity. The latter is a dataset created in the same manner as the original dataset, but consisting of different images. Therefore, it is intended to measure out of distribution performance of models. This ImageNet-V2 dataset includes three different sets, each one with a slightly different sampling strategy. We focus our experiments on the MatchedFrequencies dataset, but include evaluation on all three datasets in Table 4 and see that our method consistently outperforms the baseline. These results do not use the ASAM formulation.

We conduct object detection experiments on the Pascal VOC dataset [66]. We additionally create a Pascal VOC-C dataset, in which we apply the ImageNet-C corruptions to the Pascal VOC test set. We only use one severity level.

6.2 Additional Robustness Datasets↩︎

As described above, we choose the ImageNet V2 Matched Frequencies dataset to perform our main set of experiments, but there are two other ImageNet V2 datasets: Threshold 0.7 and Top Images. These three datasets vary in terms of selection criteria for included images. Matched Frequencies attempts to match the image selection frequency of MTurk workers in the original ImageNet validation set. Threshold 0.7 samples from among images with a selection frequency of at least 0.7. Top Images includes the images with the highest selection frequency within each class. We provide the pruned model robustness comparison on these the additional ImageNet V2 datasets in Table 4.

6.3 Network Architectures↩︎

In our experiments we prune three different networks for the classification task: ResNet50, MobileNet V1 and MobileNet V2 [72], [73]. All ablation studies use ResNet50 with ImageNet and ImageNet C datasets unless otherwise specified. “Val” column labels refer to standard ImageNet validation accuracy or loss. We use a pretrained model trained for \(90\) epochs with a cosine learning rate as in HALP [67] and EagleEye [69]. For object detection, we follow the experimental setup in HALP [67] to prune an SSD512 model with ResNet50 as the backbone. We perform Distributed Data Parallel training across 8 V100 GPUs with batch size 128 for all experiments.

6.4 Pruning Schedule↩︎

Given a pre-trained model, for any architecture, we run the warm up for \(10\) epochs, and then we follow the same pruning schedule as in [74]: we prune every \(30\) iterations and, in each iteration, we prune away a \(p_r\) fraction of neurons so that the final network is pruned by a fraction \(p\) (resulting in a network of size \(1-p\)). To determine the \(p_r\) fraction, we follow an exponential decay schedule. Let \(k = 1-p\) and let \(k_r\) be the number of neurons remaining after \(r\) pruning iterations, where the total number of pruning iterations is \(R\). Let \(m\) be the number of neurons in the dense network, and define \(\alpha = \frac{r}{R}\). Then, \(k_r = \exp \{ \alpha \log k + (1 - \alpha \log m) \}\). We fine-tune the pruned model for another \(79\) epochs (to reach \(90\) epochs total).

6.5 Optimization Hyperparameters↩︎

The base optimizer is SGD with cosine annealing learning rate with a linear warmup over 8 epochs, a largest learning rate of \(1.024\), momentum of \(0.875\), and weight decay \(3.05e-05\). Unless otherwise stated we use \(\rho_{min}=0.01\) and \(\rho_{max}=2.0\) for all experiments – we observe these values scale well across networks and tasks. For robustness encouragement we use \(\rho=2.0\) in line with prior work [60]. For some ablation experiments, the original SAM [56] optimizer is sufficient and in these cases we reduce \(\rho_{max}=0.1\) and use constant \(\rho=0.05\) for finetuning. We include some simple hyperparameter tuning results for \(\rho\) values in Table 3.

7 Additional Results↩︎

7.1 MobileNet Results↩︎

We include results on MobileNet V1 and MobileNet V2 in Tables 5 and 6. These parallel our main results in the main paper, in which we evaluated AdaSAP\(_P\) and AdaSAP\(_L\) on ResNet50 and compared them against other pruning methods. Here, we perform a similar comparison, where Table 5 includes our results on AdaSAP\(_P\) and Table 6 includes our results on AdaSAP\(_L\). We compare against Taylor importance [24], EagleEye  [69], and PolarReg  [75] for AdaSAP\(_P\) and against HALP [67], SMCP [71], MetaPruning [76], AutoSlim [77], AMC [78], and EagleEye  [69] for AdaSAP\(_L\). Results here do not use ASAM. We can observe that AdaSAP performs strongly compared to baselines, particularly in the parameter-based setting.

7.2 Margin of Improvement in Classification↩︎

Figure 6 shows the margin of performance on various corruption types for the classification task. AdaSAP outperforms a Taylor pruned model on all corruptions, across three different sparsities. We can see that it tends to particularly improve performance on several corruptions that may be important for the autonomous driving application, such as pixelated images, fog, and snow.

7.3 Margin of Improvement in Object Detection↩︎

Similarly to our result on classification, we include margins of performance improvement on various corruption types for the object detection task in Figure 7.

7.4 Relative Robustness↩︎

In Figure 8 we show that AdaSAP outperforms baselines on each of the constituent elements of the relative robustness metric. Recall that relative robustness is robust accuracy divided by standard validation accuracy. In addition to outperforming baselines on relative robustness, AdaSAP also outperforms on ImageNet validation accuracy and ImageNet C robust accuracy.

7.5 Additional Sharpness Metrics↩︎

We additionally run limited measurements of an alternative sharpness metric: the top eigenvalue of the Hessian. We compare the sharpness of a model trained with AdaSAP vs. SGD in Table 7.

8 Additional Ablations↩︎

8.1 Performance change after pruning↩︎

In Table 10 we show how the loss and accuracy change over the course of pruning. Our hypothesis is that our method sets up the network for better pruning, so that the performance drop over the course of pruning is minimized. In most cases, our method has the best validation loss and accuracy both before and after pruning. This indicates that our method sets up the model to be pruned well, and also preserves performance well throughout the pruning process.

8.2 Varying the length of the adaptive weight perturbation step.↩︎

Throughout the main set of experiments use 10 epochs for the adaptive weight perturbation step. This delineates how long we use the AdaSAP optimizer for, as well as how soon into the procedure we begin pruning. In this experiment, we analyze the sensitivity of the approach to this parameter. We report results for this experiment in Table 11. We can observe that as we make this period longer, validation accuracy and ImageNet C accuracy both drop slightly, while ImageNet V2 seems to have no discernible pattern. Ratios \(R_C\) and \(R_{V2}\) also stay consistent across the experiment.

8.3 Robustness Encouragement↩︎

As mentioned in the main text, we consider various ablations to determine the necessity of various steps of the AdaSAP procedure. In the third step of our procedure, robustness encouragement, we choose to apply uniform perturbations across all weights in the network. This differs from the first step, in which we apply adaptive weight perturbations. In Table 12, we examine the effects of different weight perturbation strategies during the robustness encouragement phase. We can see that while all three strategies lead to relatively close final performance across the three datasets, uniform weight perturbations perform slightly better, suggesting that our choice of applying them in our procedure may be slightly benefitting the performance.

8.4 Importance of Adaptive Weight Perturbations↩︎

In Table 13 we extend an ablation from the main paper in which we compare AdaSAP to SAM, effectively evaluating the importance of warmup with adaptive weight perturbations. Here, we perform the comparison on a wider range of sparsities and observe that a similar pattern emerges.

References↩︎