1 Introduction↩︎

The advancements in modern deep learning have revolutionized various domains, from computer vision [1]–[5] to natural language processing [6]–[8]. However, this success is often obtained by a paradigm that prioritizes performance metrics over computational efficiency. State-of-the-art models increasingly rely on massive and potentially redundant datasets, as well as over-parameterized architectures that lead to extremely high memory and computational costs during both training and deployment phases. As the environmental and economic impacts of these computational demands become more pronounced [9]–[11], developing sustainable and resource-efficient practices has emerged as an important requirement for the deep learning community [11].

To address the severe inefficiencies associated with model training, two equivalent effective solutions that researchers have explored are coreset selection and data pruning [12]–[16]. These methodologies seek to transform a massive dataset into a minimal, highly representative subset that preserves the statistical and structural distribution of the original data. Usually, coreset selection methods [13]–[15], [17], [18] leverage statistical properties to identify and discard uninformative or redundant examples, while retaining only the most critical samples. Consequently, coreset selection allows models to be trained on highly compact datasets, reducing computational overhead and facilitating effective few-shot model training with reduced performance impact.

Complementary to the training optimizations offered by coreset selection, knowledge distillation (KD) [17], [19]–[21] targets resource efficiency at the inference stage. This technique works by compressing the predictive capabilities and learned feature representations of a large teacher model into a much smaller and more resource-efficient student network. The main goal is to ensure low latency and to reduce memory usage during real-world deployment. KD has been extensively explored, as deploying heavy models on resource-constrained edge devices remains a critical industry bottleneck [22], [23]. Although coreset selection and knowledge distillation share the objective of resource optimization, they have largely been treated as isolated domains. The former is applied exclusively for training optimization, while the latter is reserved for inference optimization. Consequently, there is a notable absence of literature investigating the intersection of these two methodologies. Specifically, to the best of our knowledge, there are only a few recent works [17], [24] that investigated the application of coreset selection to knowledge distillation, showcasing that the random selection baseline often outperforms several other specialized methods. Overall, utilizing a coreset for distillation is underexplored, despite the benefits that are brought by this setup. Specifically, it could simultaneously reduce the computational burden of training the student model, while ensuring the student still learns effectively from the teacher’s representations.

In this work, we address the identified research gap and expand the exploration of coreset selection in conjunction with KD. Moreover, we introduce a novel, simple, yet powerful baseline called few-medoids, designed to optimally identify representative samples for distillation by leveraging the latent space of the teacher. Specifically, few-medoids analyzes the per-class feature representations extracted from the teacher. Rather than relying on complex heuristics, it determines the most informative samples within a given class by calculating their centrality. Specifically, we select the data points that exhibit the least average distance to all other samples in the same class. Through comprehensive empirical validation of various convolutional and transformer architectures on four image datasets, we demonstrate that by capturing these geometric cluster centers, few-medoids can consistently outperform previously established baselines across a diverse range of experimental setups.

In summary, our contribution is three-fold:

• We explore the effectiveness of adapting several existing baseline coreset methods to the distillation pipeline.

• We introduce few-medoids, a simple and powerful coreset selection method that leverages the latent space of the teacher model.

• We evaluate our proposed method against established baselines across four datasets and multiple model architectures.

2 Related Work↩︎

Recent advances in deep learning have been largely driven by scale, with improved performance often achieved by training continuously growing models on increasingly larger datasets [2]–[8], [25]–[27]. However, this trend comes at a substantial computational cost, as state-of-the-art performance typically requires significant training resources [11]. Moreover, supervised learning relies on large amounts of annotated data, and acquiring such labels is often expensive, time-consuming, and, in many domains, requires expert knowledge. These limitations have motivated research directions that seek to reduce either the amount of training data or the labeling burden, while maintaining competitive performance [12]–[16], [22].

One important line of research is active learning [28]–[32], which aims to reduce annotation cost by selecting the most informative samples from an unlabeled pool for labeling. Coreset selection [12], [14]–[16], [33]–[35] and data pruning [13], [17], [18], [36]–[39] are closely related, as they also seek to identify a compact yet informative subset of examples. Coreset selection has its origins in computational geometry, where early work [12] studied small subsets that approximate the original set with respect to a given objective. The three paradigms are connected by their shared objective of selecting samples that are especially useful for learning, often according to criteria such as informativeness, representativeness, or diversity [13]–[15], [17], [18], [32], [34], [37]. The main distinction lies in the learning setting: active learning operates on unlabeled data and focuses on which samples should be annotated, whereas coreset selection and data pruning typically assumes that labels are already available, and instead, aims to construct a smaller subset that can effectively approximate the full labeled training set.

Following the taxonomy proposed by Moser et al. [16], coreset selection methods can be broadly grouped into three categories: training-based, training-free, and label-free approaches. Training-based methods rely on model-dependent signals observed during optimization, such as gradients [40], [41], forgetting events [42], decision-boundary information [28], gradient matching [43], and bilevel optimization [44]. While often effective, these methods usually require repeated optimization, retraining, or additional scoring procedures, which introduces substantial computational overhead. In contrast, training-free methods rely on criteria that are independent of training dynamics, making them especially attractive in large-scale settings where efficiency and scalability are critical. This category includes simple strategies such as random sampling, as well as geometry-based methods, which remain competitive and serve as strong reference points not only in the coreset selection literature [15], [16], [34], [45], but also in dataset pruning [18], [32], [37], [46], [47], as both directions pursue the reduction of the training set, while preserving model performance. A third category, label-free methods, aims to estimate sample importance without using ground-truth labels, often through clustering or pre-trained models [33], [35], [48].

Our work belongs to the training-free category, and more specifically to geometry-based coreset selection [16], [34], [49]–[51]. Accordingly, the most relevant baselines for comparison are random selection, herding, and k-center Greedy. Random selection provides a simple baseline with negligible computational overhead, but it does not explicitly enforce representativeness or diversity, capturing dataset structure only through uniform sampling. In contrast, the two geometric baselines rely on the assumption that samples that are close in the latent space tend to share similar properties, so the geometry of that space carries meaningful information about the data distribution. Herding is designed to promote representativeness by greedily selecting samples such that the mean of the selected subset approximates the mean of the full dataset [49]–[51]. K-center Greedy instead emphasizes diversity by selecting samples that maximize coverage of the data space under a geometric distance criterion [32].

In the context of data pruning for knowledge distillation (KD) [19]–[22], recent work [17] has shown that the random sampling method is a surprisingly strong baseline. Moreover, a recent pruning study shows that the examples preferred in standard supervised subset selection are not always the most suitable for distillation. In particular, very hard examples may be suboptimal because the student cannot reliably match the teacher on them, while medium-difficulty ones provide a better trade-off between informativeness and learnability, leading to smoother decision boundaries and more stable knowledge transfer [24]. Specifically, Chen et al. [24] rank data points by prediction difficulty using the teacher’s cross-entropy loss on the ground-truth label, and select the medium-difficulty samples. Differently, few-medoids ranks samples within each class by their average Euclidean distance to all members of the same class, operating in the latent space of the teacher model, thereby favoring geometrically central points. As a result, the two methods retain different types of data.

3 Method↩︎

Problem statement. Coreset selection seeks to reduce a labeled dataset \({D}=\{(x_i, y_i)\}_{i=1}^n\) into a substantially smaller subset \({C}=\{(x_j, y_j)\}_{j=1}^p\), called coreset, such that \(C \subset D\) and \(p \ll n\). The goal is to preserve the most representative samples from each class, such that the empirical risk of a model \(h\) trained on \(C\) is as close as possible to the empirical risk of \(h\) trained on \(D\).

Few-medoids. In our KD-based approach, the coreset selection process is executed independently for each class, using the representations extracted by a teacher model. We denote this latent feature mapping of the teacher as \(f_\phi:\mathcal{X}\rightarrow \mathbb{R}^d\), where \(\mathcal{X}\) is the input space, \(\phi\) denotes the parameters of the teacher, and \(d\) represents the feature space dimension.

Let \(X_c = \{x_i \mid y_i=c\}_{i=1}^m\) denote the set of training samples belonging to a given class \(c\). Few-medoids assigns a score \(s_i\) to each sample, which is equal to its average Euclidean (\(L_2\)) distance to all other elements in the class. These values are then ranked in ascending order, so that lower scores indicate greater representativeness. The top-ranked image is therefore the class medoid, while the remaining ones can be viewed as progressively weaker medoid-like representatives under the same centrality criterion. In Algorithm 1, we summarize the proposed ranking method. In line 2, we initialize the list of importance scores \(S_{\!c}\). Subsequently, in lines 3-5, we determine these scores by calculating the distances between the latent feature vector \(f_\phi(x_i)\) of each sample and the latent feature vectors of all samples in class \(c\). Lastly, we sort the samples in \(X_c\) according to \(S_{\!c}\) and store the top \(k\) elements in \(C_c\). Then, the final corset \(C\) is given by: \[C = \bigcup_{j=1}^l C_j,\] where \(l\) is the number of classes.

Knowledge distillation. For our knowledge distillation framework, we adopt the soft-label objective introduced by Hinton et al. [20]. For a given input batch \(X\), let \(z_\theta(X)\) and \(z_\phi(X)\) denote the output logits produced by the student and teacher models, respectively. We define the distillation loss, as follows: \[\label{eq95L95KD} \mathcal{L}_{\text{KD}}(X,\phi, \theta) = \tau^2 \cdot D_{\text{KL}}\left(p_\phi(X) \,\|\, p_\theta(X)\right),\tag{1}\] where \(\tau\) is a temperature parameter, \(D_\mathrm{KL}\) is the Kullback–Leibler (KL) divergence, and \(p_\phi\) and \(p_\theta\) are soft probability distributions given by: \[p_\phi(X) = \text{softmax}\left(\frac{z_\phi(X)}{\tau}\right), \qquad p_\theta(X) = \text{softmax}\left(\frac{z_\theta(X)}{\tau}\right).\]

The soft-target loss in Eq. 1 is combined with the standard cross-entropy loss, denoted by \(\mathcal{L}_{\text{CE}}\), computed using the ground-truth hard labels \(Y\). The total loss optimized by the student network is thus defined as: \[\mathcal{L}_{\text{total}} = \lambda_{1}\cdot\mathcal{L}_{\text{KD}}(X,\phi, \theta) + \lambda_2 \cdot \mathcal{L}_{\text{CE}}(X, Y, \theta),\] where \(\lambda_{1}\) and \(\lambda_{2}\) are hyperparameters weighting the two objectives.

4 Experiments↩︎

Datasets. We conduct experiments on four benchmark datasets: CIFAR-10 [52], CIFAR-100 [52], Oxford Flowers 102 [53], and Food-101 [54]. For CIFAR-10, CIFAR-100, and Food-101, the validation set was created from the official training split using a stratified 80/20 split to preserve class balance, whereas for Oxford Flowers 102, we used the official split. CIFAR-10 and CIFAR-100 have a fixed image resolution of \(32 \times 32\) pixels, while Oxford Flowers 102 and Food-101 contain images with variable resolutions. The classes in CIFAR-10 represent general object categories, primarily animals and vehicles. CIFAR-100 spans a broader range of categories, including animals, plants, household objects, people, and vehicles. In Oxford Flowers 102, the classes correspond to flower species, while in Food-101, they represent types of food dishes. Detailed statistics for all datasets are presented in Table 1.

Underlying neural models. We evaluate several teacher-student combinations, including pairs based on the same model family, as well as a cross-architecture setting. The former pairs include ResNet-34\(\rightarrow\)ResNet-18 [3] and ViT-B/16\(\rightarrow\)ViT-Small [2], while the latter is represented by ViT-B/16\(\rightarrow\)ResNet-50. In all cases, the teacher is initialized with ImageNet-1K pre-trained weights and then fine-tuned on the target dataset. The teacher is used both to provide the distillation signal and to extract features during the sample selection stage.

Each student is also trained independently on the full training set using standard supervised learning, providing a reference for the upper performance bound, outside the few-shot distillation framework. In this setting, ResNet-18 and ResNet-50 are trained from scratch, whereas ViT-Small is initialized from pre-trained weights. These models serve as supervised baselines. In the proposed framework, KD is performed only on the subsets selected by coreset selection methods, enabling a direct comparison between supervised training on the full training set and distillation on the selected subsets. The ViT-B/16\(\rightarrow\)ViT-Small pair also allows evaluation in a pre-trained teacher-student setting.

Hyperparameters. All architectures are trained for 100 epochs using a Cosine Annealing learning rate scheduler. ResNet-based models are optimized via SGD with momentum \(0.9\), while vision transformers are optimized via AdamW. A batch size of \(128\) is used for teacher training, while student models are trained with a batch size of \(32\). The learning rate varies between \(5\cdot 10^{-2}\) and \(10^{-4}\). Students trained via KD on the selected subsets use the same optimization settings, training schedule, and data augmentation pipeline as the corresponding architectures trained on the full training set. The distillation temperature is set to \(\tau = 4\). The loss weights are set to \(\lambda_1=0.6\) and \(\lambda_2=0.4\).

Baselines. To benchmark the performance of few-medoids, we compare it against four baseline selection strategies. For a fair comparison, we applied these baselines in the latent space of the teacher. The first is random selection, where we uniformly sample \(k\) instances per class without replacement. The second is herding [51], which ensures that the running mean of the selected subset progressively approximates the true class centroid. The third is k-center Greedy [32], an algorithm that iteratively selects samples to maximize diversity in the feature space. Lastly, we introduce a custom baseline, PCA-guided matching, which leverages Principal Component Analysis (PCA) applied to the transposed feature matrix of the class samples. This transposition yields principal components that represent linear combinations of the data samples rather than the features. We select the original samples closest to these principal components to obtain distinct data samples that explain independent variations and promote subset diversity.

Evaluation protocol. We perform few-shot KD experiments for \(k \in \{1,2,4,8,16,32,64,128\}\) samples per class. For Oxford Flowers 102, the evaluation is limited to \(k \in \{1,2,4,8\}\), since each class only contains 10 training samples. For each selection strategy and each value of \(k\), five independent selections are generated using different seeds. Each resulting subset is used for a separate training run. For a given run, the checkpoint retained for final evaluation is the one that achieves the highest validation accuracy during training. When multiple checkpoints reach the same validation accuracy, the one with the lowest validation loss is selected. This checkpoint is then evaluated on the official held-out test set to obtain the final test accuracy. The reported accuracy rates represent average values over five runs.

Results. In Table 3, few-medoids demonstrates strong CIFAR-10 performance across two KD model pairs. For ResNet-34\(\to\)ResNet-18, it achieves top accuracy for five out of eight few-shot scenarios, outperforming herding and random selection by over \(5\%\) in some cases. Herding marginally leads at \(k=2\) and \(k=4\), while random selection leads at \(k=128\). For ViT-B/16\(\to\)ResNet-50, herding attains the best score for \(k \leq 8\), but few-medoids dominates for \(k \geq 16\), outperforming the other baselines by margins of up to \(11\%\). In Table 4, we show results on CIFAR-100, where we observe a clear \(k\)-dependent trend across both the ResNet-34\(\to\)ResNet-18 and ViT-B/16\(\to\)ResNet-50 settings. Few-medoids dominates the low- and mid-\(k\) regimes (\(k \le 16\) and \(k \le 32\)), outperforming herding and random selection by up to \(4\%\). However, at larger coreset dimensions (\(k \ge 64\)), both herding and random selection overtake few-medoids. As per Table 5, the rankings seem unstable on Oxford Flowers 102, since the top performance alternates among few-medoids, herding, random selection, and PCA-guided matching. This sensitivity is likely due to the limited number of training images in the dataset (10 per class) and the highly imbalanced test set. In contrast, the results reported in Table 6 demonstrate that few-medoids dominates on Food-101, where it achieves the highest accuracy across all coreset dimensions for the first two KD model pairs. In the ResNet-34\(\to\)ResNet-18 case, it outperforms herding by up to \(8\%\), and random selection by up to \(10\%\), at \(k=16\). This advantage extends to ViT-B/16\(\to\)ResNet-50, where few-medoids surpasses herding and random selection by up to \(10\%\) and \(11\%\) when \(k=16\), respectively.

In Figure 2, we illustrate the performance of ResNet-18 students for each of the studied coreset selection methods. We include these graphs as they allow a better visualization of the performance trends as a function of the coreset size, \(k\). Overall, we observe that few-medoids frequently achieves the best performance, and it shows better stability across datasets compared with the other baselines.

In summary, few-medoids consistently outperforms coreset selection baselines on the first two KD model pairs (ResNet-34\(\to\)ResNet-18 and ViT-B/16\(\to\)ResNet-50) across most datasets. The main exception is the ViT-B/16\(\to\)ViT-Small transfer, where herding generally prevails as \(k\) increases. This indicates that few-medoids excels when the student model is trained from scratch, and its initial latent space is not semantically organized a priori. For pre-trained student setups, herding seems to be more effective for \(k \ge 4\), while few-medoids remains competitive for \(k \le 2\).

Latent space analysis. In Figure 3, we compare the latent space representations of ResNet-18 student models trained with few-medoids versus the random selection baseline. We can observe that few-medoids yields more well-defined class clusters than random coreset selection. Moreover, even classes that are distinguishable in both cases form tighter clusters under few-medoids, for example the dark blue cluster. This indicates that few-medoids learns more discriminative features, and is expected to perform better in terms of generalization capacity.

5 Conclusion↩︎

In this paper, we addressed the computational inefficiencies of model training by exploring the intersection of coreset selection and knowledge distillation. Moreover, we introduced few-medoids, a simple yet highly effective training-free sample selection strategy designed for the few-shot distillation framework. Rather than relying on complex heuristics or prediction difficulty, few-medoids directly leverages the latent feature space of the pre-trained teacher model. By calculating the average Euclidean distance between samples of the same class, the method selects the most geometrically central and representative samples (the class medoids) to form the coreset for student training. Our results underscored that retaining examples that are semantically relevant for the teacher, which are located near the center of the latent distribution of each class, yields a superior supervision signal for the student model, in most of the cases. Therefore, we consider that few-medoids can be employed as a strong yet simple sample selection baseline in future research on coreset selection and data pruning for knowledge distillation.

Acknowledgments↩︎

This research is supported by the project “Romanian Hub for Artificial Intelligence - HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021-2027, MySMIS no. 351416.

References↩︎