1 Introduction↩︎

As the complexity and scale of tasks for machine learning and computer vision continue to grow, so does the size of neural networks designed to address these tasks [1]–[6]. Larger models typically achieve superior performance. However, deploying these large models can be computationally expensive and resource-intensive, making them impractical for environments with limited computational power, such as mobile devices or embedded systems [7]–[9]. Pruning is a critical technique that reduces network size without significantly compromising performance [10]–[17]. By removing redundant parts of a network, pruning can help achieve results similar to those of fully trained networks, especially in inference scenarios where efficiency is essential.

Pruning techniques are broadly divided into two categories: one-shot pruning and iterative pruning (Figure 1). One-shot pruning consists of a single cycle of training, pruning, and retraining, as detailed in Section 3. In contrast, iterative pruning involves multiple cycles of these steps, leading to potentially more refined and efficient network structures. Despite their prevalence, a systematic comparison of these two approaches under different regimes is limited in the literature. Prior research has largely focused on either the criteria for pruning [17], [18] or the specific pruning methods [15], [19], [20], leaving a gap in understanding the relative performance of different pruning strategies under varying conditions.

In this work, we conduct a thorough comparison of one-shot versus iterative pruning across multiple settings. Such a comparison requires isolating the pruning regime as a variable, which fits into the broader view of pruning as a multi-objective optimization problem trading off accuracy, model size, and computational cost. Our controlled experimental setup holds the pruning criterion and target sparsity fixed to achieve this isolation. Furthermore, to properly evaluate the retraining cost, we propose patience-based fine-tuning as a more natural way to gauge fine-tuning duration. Our analysis focuses primarily on vision tasks, where the pruning literature is most extensive. To broaden our investigation, we also present an exploratory case study on a text generation task to explore how these regimes translate to the NLP domain.

For iterative pruning, we introduce a geometric pruning ratio scheduler in addition to the constant pruning ratio scheduler. Unlike the constant scheduler, which prunes a fixed percentage of weights across the entire network, the geometric scheduler prunes a fixed percentage of the remaining weights at each step, progressively removing fewer weights as pruning progresses. Our experiments reveal that the geometric pruning scheduler generally outperforms the constant scheduler.

Additionally, inspired by the respective strengths of one-shot and iterative pruning, we propose a hybrid few-shot pruning regime, combining aspects of both methods. This hybrid approach shows advantages over the individual techniques in certain scenarios. Notably, this study does not seek to establish one method as universally superior; instead, it aims to provide guidelines to help practitioners choose the most suitable pruning approach based on specific requirements and constraints.

To summarize, the contributions of this work include:

• Defining the problem of pruning regime selection, and the first broad comparison of one-shot and iterative pruning methods.

• The introduction of a geometric pruning ratio scheduler that prunes a fixed percentage of remaining weights in each pruning step, demonstrating benefits for iterative pruning.

• Proposing a hybrid approach that combines aspects of one-shot and iterative pruning to provide flexibility in specific cases.

• Broad empirical results that describe preferred settings for each pruning regime, in particular the impact of pruning regime on pruning criteria.

2 Pruning regimes in literature↩︎

Pruning is one of the earliest and most established techniques for compressing neural networks. This section attempts to show the extent of the bias in the pruning literature towards proposing new pruning criteria. This bias underscores the importance of the proposed problem, which is broadly orthogonal and shows an alternative direction of pruning research.

Early research demonstrated that training large networks and then removing weights can significantly improve computational efficiency and reduce parameter size [21], [22]. Much of the pruning literature focuses on developing new methods to define pruning criteria, with the most common approaches based on the magnitude of weights [7], [23], sensitivity to derivatives [22], [24], or empirical filter sensitivity [25], [26]. Other methods have been introduced that prune based on neuron similarity [27], activation functions [28], or even more unconventional criteria, such as game-theoretic [29], reinforcement learning [12], or genetic algorithms [14], [30].

Despite this diversity of pruning methods, there is no unified procedure for how networks are pruned. Instead, two main pruning strategies are prevalent in the literature: one-shot pruning and iterative pruning. These approaches differ in whether the network is pruned all at once or in successive stages. However, the impact of this choice on pruning outcomes is not well understood. Each approach also comes with a unique set of parameters that can significantly influence final network performance, such as fine-tuning duration and the number of pruning iterations. Details like the exact percentage of weights removed at each step are often not explicitly provided, leaving gaps in the understanding of how specific pruning configurations affect results. In this work, we argue that these choices are as crucial as pruning criteria and can greatly influence the overall performance of the pruned network. Our study aims to address this gap in the literature, conducting a detailed analysis to clarify how different pruning strategies affect network performance.

In the following sections, we discuss studies that employ both one-shot and iterative pruning, and in Table 1, we summarize common pruning methods and their associated training protocols.

2.0.0.1 One-shot pruning.

One-shot pruning involves a single cycle of pruning followed by retraining, making it a computationally efficient approach widely appreciated in the literature. In this method, node ranking is computed only once, resulting in the final network structure after a single pruning step.

One-shot pruning is often chosen for methods where pruning costs are high. For instance, [23] removes filters near the geometric median, arguing that these filters are effectively represented by the remaining ones. Calculating the geometric median, however, is computationally intensive. Similarly, Dirichlet pruning [31] requires a training phase to compute parameters of the Dirichlet distribution, which serve as importance weights. CURL [32], on the other hand, removes all unimportant filters across layers in one step using a KL-divergence-based criterion. [33] conducted one-shot pruning according to the gradients of the latent vectors in a hypernetwork, which is introduced to guide network pruning.

After the pruning step, retraining is typically performed. However, the optimal length for retraining has not been thoroughly investigated. Instead, many methods fix the retraining duration arbitrarily; for example, CURL retrains the model for 100 epochs.

2.0.0.2 Iterative pruning.

Iterative pruning refines neural network architectures by progressively removing less important parameters or structures while fine-tuning the network to maintain performance. Unlike one-shot pruning, which removes a significant portion of weights or neurons at once, iterative pruning involves multiple cycles of pruning and retraining.

Iterative pruning requires computing parameter importance at every step. Therefore, for large networks, iterative pruning can be computationally prohibitive in cases of unstructured pruning, due to the vast number of individual parameters. However, in smaller networks, iterative pruning dates back to early methods such as Optimal Brain Damage (OBD) and Optimal Brain Surgeon (OBS) [21], which remove unimportant connections based on the second-order derivatives of the loss function with respect to the weights. In second-derivative methods, [21] requires calculating the inverse Hessian matrix, which involves significant computational effort. Given the computational cost of calculating such pruning criteria, only one [34] or a few batches [26] are often used to evaluate parameter sensitivity.

In iterative pruning, a common approach is to use a constant pruning rate—a fixed percentage of weights pruned in each cycle. However, this rate varies across methods. For example, [16] uses Taylor expansions to approximate filter contributions, removing 10 neurons every 30 mini-batches until the target number of pruned neurons is reached, followed by fine-tuning for 25 epochs. In [35], 20% of the lowest-magnitude weights are pruned globally, after which the network is retrained using learning rate rewinding and the original training time. [18] explores sub-architecture optimization by randomly removing 10% of weights, followed by fine-tuning.

The number of elements pruned in each cycle may be determined by various criteria, often treated as hyperparameters. [36] sets a custom threshold, removing elements whose magnitude is below that threshold. [37] prunes channels at each iteration based on Fisher Information scores, allowing the number of pruned channels to vary. [26] uses a hyperharmonic sequence for pruning ratios, where the i-th pruning ratio follows \(1 - \frac{1}{(i+1)^\alpha}\). [28] prunes a specified number of channels layer-wise, fine-tuning after each layer’s pruning. Additionally, [38] uses statistics from subsequent layers to prune each layer iteratively, fine-tuning for one or two epochs after each layer is pruned.  [39] applied group sparsity to the sparsity-inducing matrix and conducted a proximal gradient descent algorithm to progressively prune the network during the optimization of the pruning procedure.

2.0.0.3 Pruning at initialization and during training.

While this work primarily focuses on post-training pruning using a train-prune-retrain cycle, we acknowledge other types of pruning. Pruning at initialization [40]–[42] avoids pruning a fully trained model, instead identifying a smaller subnetwork to train from scratch for comparable performance to the larger model.

Another category involves pruning during training through regularization, which encourages certain parameters to approach zero [43], [44]. For example, [44] introduces scaling factors that selectively scale the outputs of certain CNN structures, applying sparse regularization to these scaling factors to progressively reduce their influence during training.

2.0.0.4 Unstructured and structured pruning.

Pruning networks can be done for individual weights or for structures within the network. Each of them is important in its own regard and is broadly researched. Hence, this benchmark includes tests where both unstructured and structured pruning are considered. We build our benchmark around the Torch-Pruning [46] for structured pruning, and extend it with unstructured pruning using the built-in pruning utilities in PyTorch. The details for how unstructured and structured pruning are done can be found in the Appendix.

3 Pruning regimes↩︎

In the context of neural network optimization, pruning regimes can be split into categories based on the following question: How should we divide the pruning process? Should we prune all the redundant weights at once? Or should we divide the pruning between different iterations that are separated by modifying the structure of the network and weight updates? In this section, we present rigorous definitions of pruning criteria.

For the purpose of the comparison of the pruning regimes, assume that \(W\) is the total number of weights in the neural network and \(p\) is the desired pruning percentage, e.g. \(p = 0.8\) means a pruning of 80% of weights.

• One-shot Pruning: One-shot pruning is a regime where a substantial portion of the network’s weights are removed in a single pruning step, following the process:

  • Initial Training: The neural network is first fully trained to ensure it learns the patterns in the data.

  • Pruning Step: A certain percentage of the least important weights or neurons (based on metrics like magnitude or importance) are pruned in one go.

  • Fine-tuning: After pruning, the network may undergo additional fine-tuning to regain any performance lost from pruning.

  The least significant weights are eliminated based on a predetermined importance criterion. The number of weights removed after one-shot pruning is given by: \[p \times W.\]

• Iterative Pruning: Iterative pruning is a process where weights are pruned over multiple iterations, allowing the network to gradually retrain and recover some performance loss. In each iteration the ranking, pruning, and fine-tuning occurs. We define two common approaches to iterative pruning and name them differently to avoid confusion.

  • Iterative Constant: A constant number of parameters is pruned at each step. Let \(\textit{steps}\) be the number of iterations, then \[\frac{p \times W}{\text{steps}}\] weights are pruned at each stage. The pruned percentage per step is fixed in relation to the initial number of weights.

  • Iterative Geometric: A fixed percentage \(p\) of the remaining weights is pruned at each step, meaning that as the pruning process progresses, fewer weights are pruned at each iteration.

    The number of weights at step \(n\) is given by the following formula: \[W \times (1 - \frac{p}{\text{steps}})^{n}\]

• Hybrid pruning: We propose a new pruning regime, hybrid (few-shot) pruning, which is a combination of the idea of one-shot and geometric regimes. The majority of the weights are removed at the first step and the model is retrained for a longer time. Then for the remaining weights, we perform a more fine-grained geometric-like pruning over several iterations. The hybrid pruning approach can be summarized as follows:

  • Apply large pruning ratio \(p_k\) to the original network and perform a longer fine-tuning phase.

  • Perform iterative geometric pruning with smaller pruning ratio, \(p_i \ll p_k\) starting from the state with \(p_k\) parameters removed. Proceed until the desired final pruning percentage \(p\).

We elaborate on the hybrid regime and empirical estimates for \(p_k\) and \(p_i\) in Section 5.5.

4 Pruning regime factors↩︎

Pruning regimes, whether one-shot or iterative, involve parameters that significantly impact their effectiveness. To properly evaluate these regimes, it is essential to consider these parameters.

4.1 Retraining↩︎

Retraining, or fine-tuning, is a critical step in the pruning process, helping the neural network recover performance after a portion of its weights or channels has been removed. Retraining is necessary because the initial pruning can degrade the model’s accuracy by eliminating parameters that the network previously relied upon for making predictions.

Both one-shot and iterative pruning require retraining, though the duration may vary. Generally, the larger the drop in accuracy, the longer the retraining phase should be. The accuracy drop typically correlates with the number and importance of pruned parameters, leading to longer fine-tuning for one-shot pruning and shorter, repeated fine-tuning phases for each step in iterative pruning.

However, the optimal length of retraining has not been rigorously studied, with methods often using arbitrary values. For instance, [35] fine-tunes the model for the full original training time, which may be computationally excessive since the network is not being trained from scratch. Conversely, in [38], [47], fine-tuning is limited to one epoch after each pruning step in iterative pruning.

4.1.0.1 Patience (early stopping).

In this work, we advocate for using patience or early stopping to determine the number of fine-tuning epochs in both one-shot and iterative pruning. Patience allows a dynamic approach to fine-tuning, where the model is trained until a specified criterion (e.g., validation accuracy) no longer improves over a set number of epochs. The best-performing checkpoint is retained. Patience is beneficial because it adapts the fine-tuning duration based on actual performance, whereas fixed epochs may either be insufficient or wasteful. In this study, we use patience as an alternative to a fixed epoch count, defining it as the number of epochs to continue fine-tuning before stopping if no improvement occurs. The details of our patience-based algorithm are outlined in Appendix Algorithm 1. In Appendix Section 8 we present a comparative study to emphasize the importance of proper selection of both patience-based retraining length and step size.

4.2 Iteration Pruning Rate↩︎

In iterative pruning, a key parameter is the iteration pruning rate, which determines the percentage of parameters removed at each step. Assuming a fixed target pruning rate, the iteration pruning rate is defined either by the number of steps or, conversely, the number of steps is determined based on the selected pruning rate.

As discussed in Section 1, the pruning rate varies widely across methods, from as low as 1% to as high as 20–30%. Reflecting on these values, we examine both single-digit and double-digit iterative pruning rates for constant and geometric pruning. In constant pruning, the iterative rate should evenly divide the final pruning rate. In geometric pruning, however, the amount pruned at each step decreases, and the rate is set so that the geometric sum of pruning rates across steps achieves a predefined final pruning level. Further details can be found in the Appendix.

5 Pruning regime empirical evaluation↩︎

This section consists of five parts. We first present some main takeaways and then present the detailed experiments. In the second part, we present broad results in Computer Vision and Natural Language Processing settings for the most common magnitude pruning. In the third part, we extend the results to other pruning criteria and highlight that the choice of pruning regime impacts the pruning outcomes differently when different pruning criteria are applied, a problem broadly overlooked in the literature. In the fourth section, we consider comparing the regimes when the pruning computational budget is fixed. In the final part, based on our analysis of one-shot and iterative regimes, we propose a novel hybrid approach that combines both existing pruning regimes, retaining its strength and producing a more informed and better-performing pruning regime.

5.0.0.1 Experimental set-up.

We perform experiments on several datasets and model architectures. The datasets include vision datasets, CIFAR-10 [48], CIFAR-100, and Imagenet1K [49] and the language dataset TinyStories [50]. The experiments are performed both on convolutional neural networks and transformers, in particular ResNet [3], EfficientNet [51], Visual Transformer [1] and TinyStories-33M  [50]. The open-sourced codebase allows for other custom choices. As recommended in [36], we use \(1/10^{\text{th}}\) of the original learning rate for the fine-tuning phase.

5.1 Key observations↩︎

• One-shot pruning can perform better than iterative pruning for CNNs and lower pruning rates, and iterative pruning is better for transformers and higher rates.

• One-shot pruning typically reduces retraining time compared to iterative pruning by avoiding repeated cycles of pruning and retraining, which is especially helpful when computational resources are limited.

• Iterative geometric pruning is superior to constant iterative pruning in most cases.

• Early stopping ensures optimal fine-tuning time.

• Number of retraining iterations matters significantly.

• Iterative pruning is preferable for second-derivative methods.

5.2 Comparison of one-shot and iterative regimes.↩︎

5.2.0.1 One-shot pruning with patience-based retraining.

We demonstrate that one-shot pruning, when paired with an adaptive retraining duration, can be highly effective, surpassing both forms of iterative pruning, as shown in Fig. 2. Our approach to one-shot pruning uses patience-based retraining, allowing the model to stop fine-tuning once there is no improvement over a specified number of epochs. This method is more adaptive than using a fixed number of epochs, which may result in either insufficient or excessive retraining. Notably, one-shot pruning consistently outperforms iterative pruning, particularly at pruning rates below 80%.

5.2.0.2 Iterative pruning with fixed retraining.

In contrast, iterative pruning in the literature is often paired with a fixed fine-tuning phase, sometimes limited to as little as one epoch [47]. In our experiments, we test a range of fixed retraining durations for both iterative geometric and iterative constant pruning, selecting the best-performing configuration, which is plotted in Figure 2. Results indicate that short, fixed retraining phases in iterative pruning lead to suboptimal performance. Geometric iterative pruning performs better at higher pruning ratios and in transformer models and structured pruning contexts.

5.2.0.3 Iterative pruning with patience-based retraining.

To enable a fair comparison with one-shot pruning, we propose implementing patience-based fine-tuning for iterative pruning, allowing both methods to benefit from early stopping. Since iterative pruning removes smaller fractions of weights in each step, we set a shorter patience period than in one-shot pruning to maintain efficiency. An ablation study on patience values is included in Section 8. As shown in Figure 2, patience-based iterative pruning improves fine-tuning effectiveness over standard iterative pruning, achieving higher accuracies, especially at high pruning ratios where heavily pruned networks are more sensitive to overtraining or undertraining. Iterative pruning is also preferable for transformers.

5.2.1 Natural language processing↩︎

In addition to computer vision tasks, we also conduct experiments on a natural language processing (NLP) task, specifically text generation. For these experiments, we prune the pre-trained TinyStories-33M language model [52], which is based on GPT-Neo [53]. We use the perplexity metric to evaluate pruning and fine-tuning on the TinyStories dataset. Perplexity measures how well a probabilistic model predicts a sequence of words, with lower perplexity indicating better performance. As in our previous experiments, we explore various pruning schedules and apply patience-based fine-tuning. The results are shown in Fig. 3. Generally, we observe a similar relative performance pattern between pruning regimes: one-shot pruning performs better at lower pruning ratios, while iterative pruning excels at higher compression rates, with iterative pruning showing a notably larger advantage in this context. However, unlike vision tasks, in case NLP models are more sensitive to one-shot pruning, showing performance degradation even when only 10–20% of the parameters are removed. On the other hand, interestingly, we find that in the case of iterative pruning, perplexity decreases as pruning progresses, suggesting that the LLM contains a substantial number of redundant parameters and benefits from pruning.

5.3 Pruning methods comparison↩︎

We then examine how the choice of pruning criteria influences the selection of a training regime. In this study, we compare three key criteria: magnitude-based, Taylor Expansion, and Hessian-based pruning. The results are presented in Fig. 5.

Generally, for lower pruning ratios, the one-shot regime performs better across all criteria except the constant regime. Notably, second-order approaches outperform Taylor Expansion at 70% pruning and Hessian-based pruning at 88%. However, at higher pruning rates (over 90%), an interesting conclusion emerges: the pruning regime becomes less significant, as all criteria yield similar performance. In these cases, the iterative geometric approach is preferred across the board.

From a computational perspective, these findings are encouraging. The cost of computing pruning rankings varies: it is lowest for magnitude-based pruning and highest for second-order approaches. Since second-order pruning performs better at lower pruning ratios, it is computationally efficient in one-shot scenarios, as the rankings need to be computed only once. Conversely, for higher pruning ratios where iterative pruning is preferable, the choice of criterion becomes less critical. In such cases, magnitude-based pruning is advantageous due to its faster ranking computation.

5.4 Retraining Budget↩︎

In this section, we consider the retraining budget alongside pruning rate and accuracy. We pose the question: For a given pruning rate and computational budget, which method yields the best performance? In Figure 4 we present three plots representing different pruning rates, comparing the budgets used by one-shot and iterative pruning to achieve a given accuracy. The results shown here are based on the ResNet architecture trained on CIFAR-10; additional examples can be found in the Appendix.

The retraining budget is measured in terms of the total number of retraining epochs. For one-shot pruning, this budget corresponds to a single sequence of epochs. For iterative pruning, it represents the sum of epochs over all iterations. As illustrated in Figure 4, one-shot pruning proves to be the most efficient approach for pruning rates up to 80% across all computational budgets, achieving higher accuracy across the range of total epochs. However, at higher pruning rates, iterative pruning shows improved performance, making it the preferred method in these cases.

5.5 Hybrid Regime↩︎

The findings from this work indicate that for lower pruning ratios, one-shot pruning is generally more effective than iterative pruning. Building on this insight, we propose a hybrid few-shot approach that combines elements of one-shot and iterative pruning. This hybrid method prunes a large portion of the network in a one-shot-like step, followed by a more refined, geometric pruning strategy. The results, shown in Figure 6, demonstrate that the hybrid approach performs best across nearly all pruning rates, particularly enhancing performance at lower pruning rates. Hybrid pruning leverages the strengths of both one-shot and iterative approaches: it removes the majority of weights in the initial iteration, reducing redundant cycles early in the pruning process, while retaining the precision of geometric iterations at higher pruning rates, where remaining weights carry greater importance and require finer adjustment.

Benchmarking the hybrid approach provides valuable insights into optimal parameter settings. As a general guideline, in the initial step, 60–80% of the target pruning rate \(p\) can be pruned (denoted as \(p_k\)), followed by retraining with extended patience (approximately 200 epochs). The remaining weights are then pruned iteratively with a rate \(p_i \ll p_k\), using diminishing amounts defined by a geometric sequence. For final pruning rates \(p <80\%\), the iterative pruning rate \(p_k\) can be around 10%, while for higher pruning rates \(p_k\) decreases to about 2%. Fine-tuning then continues with patience set to approximately \(\frac{1}{20}\) of the patience used in one-shot pruning.

6 Conclusion↩︎

In summary, this study provides a broad evaluation of one-shot and iterative pruning strategies, addressing a critical gap in neural network optimization research. While one-shot pruning is effective at lower pruning ratios, iterative pruning proves superior for higher pruning rates, and arguably transformer architectures and second-derivative pruning criteria. Additionally, our proposed hybrid pruning integrates the strengths of both one-shot and iterative approaches.

This study offers an empirical basis for practitioners to select a pruning regime, including key hyperparameters such as pruning length, incorporating a proposed patience-based approach and step size. Choosing an optimal pruning strategy should be tailored to the specific performance objectives and computational constraints. Future research should further investigate the impact of pruning strategies under different pruning criteria, addressing limitations identified in this work and refining techniques for more effective pruning regimes.

7 Acknowledgments↩︎

We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2024/017173. The work of Tomasz Wojnar was supported by the National Centre of Science (Poland) Grant No. 2023/50/E/ST6/00068. The work of Mikołaj Janusz was funded by the "Interpretable and Interactive Multimodal Retrieval in Drug Discovery" project. The „Interpretable and Interactive Multimodal Retrieval in Drug Discovery” project (FENG.02.02-IP.05-0040/23) is carried out within the First Team programme of the Foundation for Polish Science co-financed by the European Union under the European Funds for Smart Economy 2021-2027 (FENG).

APPENDIX↩︎

8 Ablating pruning retraining length and step size.↩︎

As discussed in this work, the retraining duration is a crucial parameter for network pruning, influenced by two main factors: patience and step size. This ablation study highlights the substantial impact of these parameters on pruning performance. Patience-based training allows for better adaptation during retraining cycles. However, as shown in Fig. 7 (a-c), choosing an arbitrary patience value can negatively affect performance. Extended retraining may not only waste computational resources but also lead to performance degradation. Similarly, step size is essential in iterative pruning, as illustrated in Fig. 7 (d-e). Step size determines the frequency of pruning and fine-tuning cycles, and our findings reveal a non-monotonic trend: both overly small and excessively large steps can reduce performance.

9 Pruning criteria further insights↩︎

In this work, we experiment with several pruning criteria, including magnitude pruning (Fig. 2), Hessian-based pruning [22] (Fig. 3b-c; see Appendix for structured pruning adaptation), and Taylor expansion-based contribution approximation pruning [16] (see Appendix).

Our primary focus is on magnitude pruning due to its simplicity, effectiveness, reliability, and low computational cost, allowing for extensive benchmarking and experimentation [54]. Magnitude pruning is one of the most widely used pruning criteria across a variety of methods [7], [35], [36].

In this section, however, we want to address the potential similarities and differences in evaluating pruning regimes when alternative criteria are applied. Overall, we find that the relative performance of pruning regimes is largely consistent across different criteria. With the appropriate retraining duration, one-shot pruning performs best up to 80% of the original parameter count, while iterative pruning is preferable at higher compression ratios. Notably, for Hessian-based criteria, one-shot pruning at high pruning rates results in a significant accuracy drop, suggesting iterative pruning may be a more stable solution for second-derivative-based methods.

For second-derivative pruning, the Hessian matrix, which captures the curvature of the loss function, identifies weights in low-curvature regions (small eigenvalues) as good pruning candidates. The experimental results may be explained by the fact that single-step pruning can dramatically alter the loss landscape, rendering the pre-pruning Hessian less accurate in assessing remaining weights. In contrast, iterative pruning enables recalculating the Hessian at each step, ensuring a more precise sensitivity evaluation of the weights retained.

We then expand on the main text, which compares different pruning criteria under various training regimes. We present additional results on structured pruning using two criteria: Hessian-based pruning [22] and Taylor expansion-based contribution approximation pruning [16]. The results, shown in Fig. 8, are largely consistent with the conclusions drawn in the main paper. Specifically, one-shot pruning performs better or comparably up to about a 90% pruning rate, whereas iterative pruning yields better performance at higher compression ratios.

9.1 Further magnitude-pruning results.↩︎

Figure 2 presents further comparison of one-shot and iterative pruning across various network architectures and vision datasets. The iterative pruning comes in two types: constant and geometric.

10 Early stopping.↩︎

In this section, we provide the detailed algorithm for early stopping performed in this paper. We use this algorithm for both one-shot and iterative geometric and iterative constant pruning.

11 Structured and unstructured pruning↩︎

11.0.0.1 Unstructured pruning.

Unstructured pruning involves selectively removing individual weights from the neural network based on criteria such as weight magnitude or their impact on the loss function [7], [40]. This method creates sparse weight matrices with many zero elements, which can significantly reduce the parameter count. However, practical computational gains often require specialized hardware or software optimizations because the remaining weights are irregularly distributed across the network.

11.0.0.2 Structured pruning.

In contrast, structured pruning removes entire components within the neural network, such as filters, channels, neurons, or even layers [12], [15], [17], [39], [55]. This method produces a more compact and regular network structure that retains a dense matrix format, making it easier to optimize on standard hardware. Structured pruning can substantially reduce both model size and computational requirements while maintaining a more organized network. However, achieving high sparsity ratios with structured pruning is more challenging, as it requires removing entire rows or columns rather than individual elements within a weight matrix.

The pruning regimes discussed in the following section apply to both unstructured and structured pruning. However, implementation details may vary due to constraints imposed by the structure of the pruned components.

11.1 Structured pruning pruning ratios↩︎

We then provide details on the pruning percentages for each layer in structured pruning. In unstructured pruning, we perform global pruning, allowing pruning to occur freely in any layer. However, applying this approach to structured pruning can lead to pruning collapse, where a layer ends up without any channels. To prevent this, we define a separate pruning ratio for each layer. These ratios are chosen so that the total number of pruned channels across the entire network matches the desired overall pruning percentage. The details of each layer’s pruning ratio and the final pruning percentages are given in Table 2.

12 Hybrid regime experimental details↩︎

In the hybrid regime experiments, we used the same configuration as in the ResNet-18 experiments on CIFAR-100 and CIFAR-10 datasets. The hybrid regime consisted of an initial one-shot pruning step to a value of \(p\%\), followed by iterative geometric steps with a ratio of \(p_i\) until reaching the desired total pruning percentage. The first iterative step begins at \(p_k\%\). For the hybrid regime, we used different patience values for the one-shot part and the iterative geometric part. We tested all the configurations provided in Tables 3 and 4. For the sake of preciseness we provide the exact pruning percentages \(p_k\) which were used in the iterative phase of the hybrid regime; however in this phase, we aimed to test a set of iterative percentages from the set, \(p_k=0.02, 0.05, 0.10\). The adjustments were necessary to obtain the exact final pruning ration \(p\) and the fair comparison with other pruning methods.

13 Experiments set-up↩︎

13.1 Dependencies↩︎

The technical setup for the experiments included the following dependencies:

• Python 3.11

• CUDA 12.1

• PyTorch 2.2

• Torchvision 0.17.0

• timm 0.9.16

• torch-pruning 1.4.2

Experiments were conducted mainly on NVIDIA A100 and RTX 2080ti GPU’s.

13.2 Dataset transformations↩︎

Cifar-10 and Cifar-100 transformations include normalization (values are located in the code repository), random crop of size 32×32 with padding 4 and random horizontal flip. The images were also resized to higher resolution for some models. ImageNet1K was normalized, resized and cropped.

13.3 Checkpoints↩︎

The checkpoints for the models used in the experiments are shared here: https://www.dropbox.com/scl/fo/u0d8a087o3c2ynzpb6chd/AJz5w2ozXzcrBzxUwXVMiYM?rlkey=gag0w2r89kmt1huek6zsy9re2&st=4guxofag&dl=0.

13.4 Parameters↩︎

All experiments were conducted using the SGD optimizer with the following parameters:

• Learning rate: 0.01

• Momentum: 0.9

• Weight decay: 0.0005

The batch size was set to 512, consistent across all pruning experiments. The training data was shuffled for every run.

13.4.1 CIFAR-100 / ResNet18↩︎

We used the ResNet-18 model from the following GitHub repository: https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py

Before pruning, the model was trained for 328 epochs with the following configuration:

• SGD optimizer with learning rate: 0.1, momentum: 0.9, and weight decay: 0.0005

• Linear scheduler for 100 epochs with start factor: 0.01

• After 100 epochs: CosineAnnealingWarmRestarts with \(T_0 = 50\), \(T_{\text{mult}} = 2\), and \(\eta_{\text{min}} = 1 \times 10^{-5}\)

• Early stopping with patience of 100 epochs

Before pruning, the top-1 accuracy was 74.64%.

13.4.2 CIFAR-10 / ResNet18↩︎

The same ResNet-18 model as for CIFAR-100 was used. Before pruning, the model was trained for 226 epochs with the same configuration as the CIFAR-100 experiment. Before pruning, the top-1 accuracy was 94.14%.

13.4.3 ImageNet / ResNet18↩︎

We used the ResNet-18 model from the PyTorch torchvision library with pretrained weights. During fine-tuning in the pruning phase, images were resized to \(256 \times 256\) and cropped to \(224 \times 224\). Before pruning, the top-1 accuracy was 68.91%.

13.4.4 CIFAR-100 / EfficientNet↩︎

We used the EfficientNet V2-S model from the PyTorch torchvision library with pretrained weights (IMAGENET1K_V1). Before pruning, the model was trained for 132 epochs with the following configuration:

• SGD optimizer with learning rate: 0.1, momentum: 0.9, and weight decay: 0.0005

• Linear scheduler for 10 epochs with start factor: 0.01

• After 10 epochs: CosineAnnealingWarmRestarts with \(T_0 = 10\), \(T_{\text{mult}} = 2\), and \(\eta_{\text{min}} = 1 \times 10^{-5}\)

• Early stopping with patience of 80 epochs

• Images resized to \(128 \times 128\)

Images were resized during fine-tuning in the pruning phase. Before pruning, the top-1 accuracy was 87.53%.

13.4.5 CIFAR-10 / EfficientNet↩︎

The same EfficientNet V2-S model as for CIFAR-100 was used. Before pruning, the model was trained for 152 epochs with the same configuration as the CIFAR-100 experiment. Before pruning, the top-1 accuracy was 97.88%.

13.4.6 CIFAR-100 / ViT↩︎

We used the ViT small patch16 224 model from the timm library with pretrained weights (vit small patch16 224 augreg in1k). Before pruning, the model was trained for 18 epochs with the following configuration:

• SGD optimizer with learning rate: 0.1, momentum: 0.9, and weight decay: 0.0005

• Linear scheduler for 10 epochs with start factor: 0.01

• After 10 epochs: CosineAnnealingWarmRestarts with \(T_0 = 10\), \(T_{\text{mult}} = 2\), and \(\eta_{\text{min}} = 1 \times 10^{-5}\)

• Early stopping with patience of 50 epochs

• Images resized to \(224 \times 224\)

Images were resized during fine-tuning in the pruning phase. Before pruning, the top-1 accuracy was 88.16%.

13.4.7 CIFAR-10 / ViT↩︎

The same ViT small patch16 224 model as for CIFAR-100 was used. Before pruning, the model was trained for 19 epochs with the same configuration as the CIFAR-100 experiment. Before pruning, the top-1 accuracy was 98.11%.

References↩︎

---

1. Corresponding Author. Email: mikolaj1.janusz@student.uj.edu.pl↩︎