1 Introduction↩︎

Dataset distillation (DD) [@wang2018dataset] aims to condense the knowledge from a massive training dataset into a remarkably small synthetic set, preserving comparable generalization performance. By substituting the original data with this compact proxy, dataset distillation drastically reduces training overhead and storage costs. Consequently, it has emerged as an enabling technique for diverse applications, including continual learning [@zhao2020dataset; @rosasco2021distilled], neural architecture search [@wang2021rethinking; @such2020generative], and privacy-preserving learning [@xiong2023feddm; @dong2022privacy; @shang2022federated; @chen2022private].

To scale dataset distillation to large-scale datasets like ImageNet-1K [@deng2009imagenet] and ImageNet-21K [@ridnik2021imagenet], a recent line of research focuses on information extraction pipelines [@sun2024diversity; @yin2023squeeze; @shao2023generalized; @shen2024delt; @tran2025enhancing]. Most notably, the pioneering SRe\(^2\)L [@yin2023squeeze] introduces a decoupled three-stage paradigm: (i) Squeeze the dataset information into a pre-trained model, (ii) Recover this information back into the image space to form synthetic data, and (iii) Relabel the distilled images using the pre-trained model to inject label-space knowledge. By avoiding costly unrolled optimization, this paradigm yields strong empirical results.

Despite their success, existing extraction-based pipelines suffer from two critical bottlenecks: poor cross-architecture generalization and prohibitive computational overhead, particularly during the Recover stage. We identify the root cause as the severe information loss induced by the implicit dual-compression process: first from data to model parameters (Squeeze), and then from model back to synthetic images (Recover). Furthermore, while the Relabel stage improves performance, our theoretical and empirical analyses reveal a hidden vulnerability: its efficacy is strictly conditional on distribution alignment. When the dual-compression process causes the synthetic samples to drift away from the original data distribution, the pre-trained model acts as an unreliable labeler, yielding sub-optimal labels. Therefore, ensuring distributional proximity is crucial to fully unlock the benefits of the Relabel strategy.

Motivated by these insights, we abandon the flawed dual-compression paradigm and propose CIM, a novel framework that explicitly condenses information by minimizing the information loss between real and distilled datasets (CIM). Unlike prior extraction-based methods, which rely on complex inversion to recover informative synthetic images from a pre-trained model, CIM is explicitly metric-driven. Specifically, we formulate a principled metric to comprehensively quantify the information gap, and consequently the distribution shift, between the synthetic samples and their real counterparts. By directly optimizing the distilled set to minimize this gap within a unified objective, CIM ensures high-fidelity information retention and strict distribution alignment. This design not only eliminates the computationally expensive recovery process but also natively satisfies the conditions required for effective relabeling. Our contributions are summarized as follows:

1. We conduct a rigorous revisiting of information extraction-based dataset distillation, revealing that the inherent dual-compression process leads to severe information loss. This loss not only degrades cross-architecture generalization and computational efficiency but also causes a distribution shift that fundamentally compromises the widely adopted Relabel strategy.

2. We introduce CIM, a novel, metric-driven framework that explicitly quantifies and minimizes the information gap between the original and distilled datasets, achieving more faithful, efficient, and aligned information condensation.

3. Extensive experiments verify that CIM establishes a new state-of-the-art across various scale datasets. Notably, our framework distills ImageNet-1K at IPC\(= 10\) in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18, outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively.

2 Related Work↩︎

The primary aim of dataset distillation is to condense a large dataset into a smaller, yet highly representative subset, preserving its core semantic and statistical characteristics. Wang et al. [@wang2018dataset] first introduce the dataset distillation as a bi-level meta-learning optimization problem. The outer loop aims at optimizing the meta-dataset, while the inner loop focuses on training models using the distilled dataset. Existing methods can be roughly divided into three paradigms for solving this complex bi-level optimization problem.

Tackling this bi-level problem is complex, especially when optimizing proxy models via gradient descent, which involves unraveling an intricate computational graph. studies [@zhou2022dataset; @loo2022efficient] have proposed approximating model training using kernel ridge regression, which provides a closed-form solution for optimal weights, thereby reducing training costs and improving performance. Despite these advancements, such methods still struggle with extensive computational demands or limitations due to the approximations in convex relaxation.

Another strategy involves emulating the behaviors of the original dataset in the distilled one. They focus on minimizing disparities between surrogate models trained on both synthetic and original datasets. The key metrics for this are matching gradients [@zhao2020dataset; @kim2022dataset; @zhang2023accelerating; @liu2023dream], features [@wang2022cafe], distribution [@zhao2023dataset; @zhao2023improved], and training trajectories [@cazenavette2022dataset; @cui2022dc; @du2023minimizing; @cui2023scaling; @yu2023dataset; @guo2023towards]. Trajectory and gradient matching, in particular, has shown impressive results with low IPC. However, these methods often tailor the distilled dataset to specific network architectures, limiting their generalizability. Cazenavette et al. [@cazenavette2023generalizing] address this by proposing the GLaD that synthesizes more realistic images to enhance generalization. These methods incur substantial computational overhead due to the frequent calculation of discrepancies between the distilled and original datasets, requiring numerous iterations for optimization until convergence. Therefore, computational and memory challenges remain, particularly when scaling to large datasets.

The SRe\(^2\)L framework [@yin2023squeeze], as the first work efficiently scalable to ImageNet-1K, introduces a novel decoupled bi-level learning paradigm. This involves three stages: 1) Squeeze relevant information from the original dataset into a pre-trained model, 2) Recover this information into the image space, 3) Relabel the distilled images by using the pre-trained model to further distill knowledge into the label space. Its efficiency and effectiveness have garnered community attention, spurring a series of research efforts. Shao et al. [@shao2023generalized] note that SRe\(^2\)L is limited to specific backbones and layers, impacting the generalization of the distilled dataset. They advocate for using diverse backbones for more precise and effective distillation. Yin et al. [@yin2023dataset] further enhance SRe\(^2\)L with curriculum data augmentation. [@liu2023dataset] use Wasserstein distance to create more representative images. Sun et al. [@sun2024diversity] introduce an optimization-free approach RDED that achieves notable diversity and realism in distilled datasets. Recently, DELT [@shen2024delt] mitigates the issue of low within-class diversity of SRe\(^2\)L by varying the number of iterations for different IPCs during the data synthesis phase. Moreover, NRR-DD [@tran2025enhancing] addresses the shortcoming of SRe\(^2\)L in often emphasizing instance-specific features by effectively capturing both class-general and instance-specific features. EDC [@shao2024elucidating] introduces a unified framework grounded in both empirical and theoretical foundations. Building on the distillation process introduced by SRe\(^2\)L [@yin2023squeeze], it incorporates two key enhancements: soft category-aware matching and dynamic adjustments to the learning rate schedule. These additions can further optimize the distillation process.

3 On the Pitfalls of Information Extraction Paradigm↩︎

In this section, we revisit the information extraction paradigm for dataset distillation in depth. We highlight two critical aspects: 1) The primary challenge is significant information loss, which hurts the quality and diversity in distilled images. 2) We empirically and theoretically demonstrate the critical attributes necessary for retaining the efficacy of Relabel and seek an approach to effectively distill information from original images while adhering to these attributes.

3.1 Preliminary↩︎

Given a large-scale dataset \(\mathcal{T}= \{ \mathbf{x}_{i}, y_{i} \}_{i=1}^{\left\lvert\mathcal{T}\right\rvert}\) which consists of \({\left\lvert\mathcal{T}\right\rvert}\) samples, dataset distillation aims to synthesize a smaller set \(\mathcal{S}=(\mathcal{S}_X, \mathcal{S}_Y) = \{ \widetilde{\mathbf{x}}_{j}, \widetilde{y}_{j} \}_{j=1}^{\left\lvert\mathcal{S}\right\rvert}\) with \({\left\lvert\mathcal{S}\right\rvert}\) synthetic samples such that models trained on \(\mathcal{T}\) will have similar performance as models trained on \(\mathcal{S}\): \[\textstyle \mathbb{E}_{\mathbf{x}\sim P_\mathcal{D}}[ \ell\left(\phi_{\boldsymbol{\theta}_{\mathcal{T}}}(\mathbf{x}), y\right)]\simeq \mathbb{E}_{\mathbf{x}\sim P_\mathcal{D}}[ \ell\left(\phi_{\boldsymbol{\theta}_{\mathcal{S}}}(\mathbf{x}), y\right)] \,, \label{eq:ddobj}\tag{1}\] where \(P_\mathcal{D}\) is the test real distribution, \(\mathbf{x}\) is a data sample, \(\ell\) is the loss function, i.e., cross-entropy loss. Here, \(\boldsymbol{\theta}_{\mathcal{T}}\) and \(\boldsymbol{\theta}_{\mathcal{S}}\) denotes the parameters of the neural network \(\phi\) trained on \(\mathcal{T}\) and \(\mathcal{S}\), respectively.

As the first effective yet efficient solution to allow the dataset distillation on diverse-scale datasets such as ImageNet-1K [@deng2009imagenet], information extraction-based methods have attracted attention and inspired many subsequent works. These methods typically employ a three-stage distillation process, indirectly transferring information from the original to the distilled images. The first stage involves condensing information from the complete dataset into pre-trained neural network models via Squeeze, followed by the extraction of this information into distilled images using Recover[@yin2023squeeze; @liu2023dataset]. Upon the distilled data samples, Relabel applies a pre-trained model to further distill knowledge into the label space. However, this paradigm encounters several key challenges:

1. The Recover stage necessitates batch normalization in pre-trained models [@ioffe2015batch] to align the statistical features between distilled and original images [@yin2023squeeze; @liu2023dataset].

2. Significant information loss during both Squeeze and Recover stages leads to distilled images with minimal content, adversely affecting performance, particularly in low IPC settings [@sun2024diversity].

3. Distilled images often exhibit unrealistic textures or semantics, tailored to specific networks, thereby limiting their generalization ability [@shao2023generalized].

4. Despite outperforming other paradigms (e.g., matching-based methods) in terms of efficiency [@yin2023squeeze], the Recover stage is still computationally demanding, requiring numerous optimization iterations [@yin2023dataset].

In the meanwhile, the influence of Relabel is under-explored [@yin2023squeeze; @shao2023generalized]. These challenges motivate us to explore a method to simultaneously address four key problems by directly condensing information from the original dataset for image distillation, abandoning traditional decoupled Squeeze and Recover stages, and to further explore the role of Relabel.

3.2 Does Relabel Always Help Distilled Images?↩︎

Relabel has become a widely adopted and effective technique in dataset distillation, as demonstrated in recent works [@yin2023squeeze; @sun2024diversity; @shang2024gift]. The core idea is to use a model pre-trained on the full real dataset to re-assign labels to the distilled images, and then train a student model on these relabeled distilled samples.

To evaluate the effect of Relabel, we compare state-of-the-art DD methods with and without this strategy. As reported in Tab. 7 (App. E), removing Relabel leads to a substantial performance degradation. For example, on ImageNet-1k, the accuracy of SRe\(^2\)L and G-VBSM drops to 1.1% and 0.8%, respectively, when Relabel is disabled.

Given its effectiveness, a natural question is whether Relabel can be used as a plug-and-play component for DD methods that do not originally rely on it. To answer this, we apply Relabel to distilled images produced by representative non-Relabel methods, such as ADD [@zhang2023accelerating] and DataDAM [@sajedi2023datadam]. The results are shown in Fig. 1. We observe that performance gains are mainly limited to the very early distillation stage (i.e., when the “distilled” images remain close to the initial real images used for initialization). In contrast, as distillation proceeds, relabeling the increasingly optimized distilled images provides little benefit and can even hurt performance.

This behavior suggests a mismatch between the relabeling model and the evolved distilled samples. Specifically, the model used for Relabel is trained exclusively on the original real dataset, whereas distilled images gradually deviate from the real-image distribution during optimization. Such a distribution shift alters semantic and/or textural cues, making the pre-trained relabeler less reliable on distilled samples (see App. K for qualitative evidence). Consequently, inaccurate relabels accumulate, and the downstream student model suffers.

Beyond the above empirical observation, we theoretically reveal that a model well-trained on the original, undisturbed samples (i.e., the model used for Relabel) may only provide suboptimal labels for shifted distilled samples. Concretely, standard DD typically starts from a subset of real samples \(\{\mathbf{x}_j\}_{j=1}^{|\mathcal{S}|}\) drawn from the full dataset \(\mathcal{T}\), and iteratively optimizes them into distilled samples \(\{\widetilde{\mathbf{x}}_j\}_{j=1}^{|\mathcal{S}|}\) with \(\widetilde{\mathbf{x}}_j=\mathbf{x}_j+\epsilon_j\), where \(\epsilon_j\) denotes the learned perturbation. A shift occurs because the optimization updates \(\epsilon_j\) may change both semantics and textures of \(\mathbf{x}_j\), causing the distilled distribution to deviate from the original one.

That is, a relabeler trained on the original dataset may provide unreliable labels when applied to distribution-shifted distilled samples. Therefore, to maximize the efficacy of Relabel, it is necessary to ensure the distilled data distribution remains close to that of the original real dataset. Only under such alignment can distilled samples be recognized and labeled correctly by the pre-trained relabeler.

To achieve these goals, a straightforward approach involves directly selecting images from the original dataset to construct the distilled dataset. Numerous works [@sun2024diversity; @forgy1965cluster; @welling2009herding; @tran2025enhancing] have explored methods for selecting diverse and representative key samples from the original dataset to create a distilled dataset. These methods can ensure that the distilled data can be accurately identified by the pre-trained model and maximize the effectiveness of Relabel. A notable contribution is RDED [@sun2024diversity], which extracts key patches from each image based on high realism scores to construct a distilled dataset.

To further validate whether the sample selected by RDED can achieve the distribution as the original dataset, we visualize the feature distributions ¹ of the distilled dataset alongside those of the original dataset in Fig. 2. Specifically, we compare the state-of-the-art methods, including SRe\(^2\)L [@yin2023squeeze], G-VBSM [@shao2023generalized], and RDED [@sun2024diversity]. Among these, RDED demonstrates the closest alignment with the feature distribution of the original dataset. Additionally, as shown in Tab. ¿tbl:tb:main95cifar? and Tab. ¿tbl:tab:main95imagenet95resnet18?, RDED achieves superior performance compared to the other two SOTA methods. Therefore, by default, CIM adopts the selection mechanism of RDED². Note that the subset selection strategy itself is not the focus of this work. Our proposed CIM is selection-method-agnostic and can incorporate various selection strategies, as demonstrated in Tab. ¿tbl:tab:selection95ipc10?.

4 Methodology↩︎

The following section begins by introducing the formal definitions of effective information for a given sample and the resulting information gap between any two given samples. Furthermore, we propose a method based on minimizing the information gap between the selected sample subsets and the distilled samples to ensure the preservation of information integrity in the distilled set. The distillation process of our CIM is depicted in Fig. 3, with the detailed algorithm outlined in Alg. 1 in Appendix.

For the selected \(\texttt{IPC}\) subsets of key images, we aim to compress each of them into a more compact pixel space, i.e., distilled image \(\widetilde{\mathbf{x}}\), thus forming \(\mathcal{S}_X\). However, given the constraints of limited pixel space storage, using a naive solution—e.g., directly resizing and concatenating multiple images from a subset into one—results in a significant reduction in the fineness and detail of the original images. To effectively capture and condense the salient information from the original samples into the distilled ones, we aim to enable each distilled sample \(\widetilde{\mathbf{x}}\) to encapsulate the information of a selected subset from \(\mathcal{T}_X\). We begin by formally defining the effective information of a data sample as follows.

The Def. [1](#def:ei){reference-type=“ref” reference=“def:ei”} posits that the effective information of a sample encompasses the set of its features as perceived or extracted by a diverse set of observers. Samples are considered to have similar effective information if they result in comparable feature sets across the observers in \(\mathcal{R}\). We consequently define the effective information gap between two given samples \(\mathbf{x}_i\) and \(\mathbf{x}_j\).

However, directly minimizing the information gap between distilled set \(\mathcal{S}_X\) and original set \(\mathcal{D}_X\) through Def. [2](#def:ei95gap){reference-type=“ref” reference=“def:ei95gap”} is intractable, due to Eq. eq. 1 is a sample-level metric that cannot be directly applied to the set. Thanks to the widely adapted data augmentation techniques \(\mathcal{A}\) to enhance the diversity of each sample \(\widetilde{\mathbf{x}}\) in practice, we relax the estimation of Eq. eq. 1 through calculating the effective information gap between a set of \(N\) original samples \(\{ \mathbf{x}_i \}_{i=1}^N\) and a distilled sample \(\widetilde{\mathbf{x}}_j\) using \(N\) augmented views of the distilled samples, i.e., \[\label{eq:dis95goal} \begin{align} {\mathbb{E}}_{(\mathbf{x}_i,\widetilde{\mathbf{x}}_j^{(i)}) \sim (\{ \mathbf{x}_i \}_{i=1}^N, \mathcal{A}(\widetilde{\mathbf{x}}_j))} I_G(\mathbf{x}_i, \widetilde{\mathbf{x}}_j^{(i)};\mathcal{R}) \mathrm{s.t.} \quad \mathcal{A}(\widetilde{\mathbf{x}}_j) = \{\widetilde{\mathbf{x}}_j^{(1)}, \widetilde{\mathbf{x}}_j^{(2)}, \ldots, \widetilde{\mathbf{x}}_j^{(N)}\}. \end{align}\tag{2}\] However, we still cannot directly distill sample \(\widetilde{\mathbf{x}}_j\) through Eq. eq. ¿eq:eq:dis95goal? due to the intractable KL divergence estimation in Eq. eq. 1, and thus we derive Thm. [1](#thm:bound95ei95gap){reference-type=“ref” reference=“thm:bound95ei95gap”} that bounds Eq. eq. 1 to enable it computationally tractable.

Therefore, we can capture a distilled sample \(\widetilde{\mathbf{x}}_j\) through combining Eq. eq. ¿eq:eq:dis95goal? and Thm. [1](#thm:bound95ei95gap){reference-type=“ref” reference=“thm:bound95ei95gap”}, namely, \[\small \label{eq:sobj} {!}{ \mathop{\mathrm{arg\,min\,}}_{\widetilde{\mathbf{x}}_j} {\mathbb{E}}_{(\mathbf{x}_i,\widetilde{\mathbf{x}}_j^{(i)}) \sim (\{ \mathbf{x}_i \}_{i=1}^N, \mathcal{A}(\widetilde{\mathbf{x}}_j))}{\mathbb{E}}_{\xi_k \sim \mathcal{R}}{\| \xi_k(\mathbf{x}_i) - \xi_k(\widetilde{\mathbf{x}}_j^{(i)}) \|^2} }\tag{3}\]

To reduce computational complexity, we utilize RandomCrop to generate the augmentations \(\mathcal{A}(\widetilde{\mathbf{x}}_j)\) and concatenate resized real images to initialize distilled sample \(\widetilde{\mathbf{x}}_j\). We further consider a specific scenario where the observer group consists of only one pre-trained model across various transformations³. This strategy allows us to employ multiple observers without requiring an excessive number of pre-trained models. Our loss function is defined as follows to achieve Eq. eq. ¿eq:eq:sobj?: \[\label{eq:loss} \begin{align} \mathcal{L}_{\Delta \widetilde{\mathbf{x}}_j} = {\mathbb{E}}_{(\mathbf{x}_i,\widetilde{\mathbf{x}}_j^{(i)}) \sim (\{ \mathbf{x}_i \}_{i=1}^N, \mathcal{A}(\widetilde{\mathbf{x}}_j+ \Delta \widetilde{\mathbf{x}}_j))} {\mathbb{E}}_{\zeta_k \sim \mathcal{G}} \left\| \zeta_k \circ \phi_{\boldsymbol{\theta}_\mathcal{T}}(\mathbf{x}_i) - \zeta_k \circ \phi_{\boldsymbol{\theta}_\mathcal{T}}(\widetilde{\mathbf{x}}_j^{(i)}) \right\|^2 \end{align}\tag{4}\] where \(\mathcal{R}= \{ \xi_k \mid \xi_k = \zeta_k \circ \phi_{\boldsymbol{\theta}_\mathcal{T}} , \forall \zeta_k \sim \mathcal{G}\}\) and \(\mathcal{G}\) denotes the transformation group, \(\mathcal{A}(\widetilde{\mathbf{x}}_j + \Delta \widetilde{\mathbf{x}}_j)=\{\widetilde{\mathbf{x}}_j^{(1)}, \widetilde{\mathbf{x}}_j^{(2)}, \ldots, \widetilde{\mathbf{x}}_j^{(N)}\}\). By minimizing Eq. eq. ¿eq:eq:loss?, we find the optimal \(\Delta \widetilde{\mathbf{x}}_j^\star\) and capture the image \(\widetilde{\mathbf{x}}_j \gets \widetilde{\mathbf{x}}_j + \Delta \widetilde{\mathbf{x}}_j^\star\) as the distilled image.

Directly generating the distilled image \(\widetilde{\mathbf{x}}_j\) through aligning its effective information with the original image set \(\{ \mathbf{x}_i \}_{i=1}^N\) in the last layer of model \(\phi_{\boldsymbol{\theta}_\mathcal{T}}\) may lead to a substantial loss of texture information. The reason is that the model \(\phi_{\boldsymbol{\theta}_\mathcal{T}}\) tends to extract semantic information from input images, which often results in a notable drop in the texture details. To balance between semantic richness and texture preservation in the distilled image \(\widetilde{\mathbf{x}}_j\), we leverage intermediate model features instead of the last-layer logits, which helps in retaining more textural details (see Section [sec:featalign] for the validation of its robustness).

We propose an improved re-labeling strategy for our informative distilled images \(\widetilde{\mathbf{x}}\), which provides more diverse and informative knowledge in the label space compared to the basic one-shot labeling approach. The standard Relabel technique [@yin2023squeeze], inspired by [@yun2021re], suggests that a random image crop may contain a different object than the one originally labeled, leading to inaccurate or misleading training data. This highlights the limitations of the one-shot labeling strategy in expressing sufficient knowledge for cropped images.

Our extended Relabel approach considers a wider range of image transformations beyond random cropping. Similar to the soft labeling approach in [@shen2022fast], we generate transformed-view-level soft label \(\widetilde{y}_k = \phi_{\boldsymbol{\theta}_\mathcal{T}} ( \zeta_k (\widetilde{\mathbf{x}}) )\), where \(\zeta_k (\widetilde{\mathbf{x}})\) represents the \(k\)-th transformation applied to the distilled image \(\widetilde{\mathbf{x}}\).

Therefore, we can train the model \(\phi_{\boldsymbol{\theta}_\mathcal{S}}\) on the distilled data by minimizing: \[\textstyle \mathcal{L}= - \sum_{j}{\sum_{k}{\| {\phi_{\boldsymbol{\theta}_\mathcal{S}}( \zeta_k(\widetilde{\mathbf{x}}_{j}) )}}} - \widetilde{y}_{(j,k)} \|^2 \,.\]

The whole distillation process for an entire dataset by using our CIM is illustrated in Alg. 1 in Appendix.

5 Experiment↩︎

In this section, we evaluate the performance of our proposed CIM over various datasets and neural architectures. First, we demonstrate the superior results of CIM on real-world datasets, cross-architecture generalization and efficiency. We next perform comprehensive ablation studies to evaluate the impact of each component in our proposed method, as well as to analyze the influence of hyperparameter choices and subset selection strategies. Finally, we demonstrate the superior performance of our approach, CIM, in continual learning applications.

5.1 Experimental Setting↩︎

We conduct experiments on varying scales and resolutions of images.

• Small-scale: we evaluate on two datasets, including CIFAR-10 (\(32 \times 32\)) [@krizhevsky2009cifar] and CIFAR-100 (\(32 \times 32\)) [@krizhevsky2009learning].

• Large-scale: we also use two large-scale high-resolution datasets including Tiny-ImageNet (\(64 \times 64\)) [@le2015tiny] and ImageNet-1K (\(224 \times 224\)) [@deng2009imagenet].

Following prior dataset distillation works [@yin2023squeeze; @zhao2023improved; @guo2023towards], we employ ConvNet [@guo2023towards], ResNet-18 [@he2016deep], and MobileNet-V2 [@sandler2018mobilenetv2], ViT-T/16 [@dosovitskiy2021image], ShuffleNet-v2-x2.0 [@ma2018shufflenet], DenseNet-121 [@huang2018densely], as our backbone networks. Specifically, for ConvNet, we use Conv-3 on CIFAR-10/100 and use Conv-4 on Tiny-ImageNet and ImageNet-1K. More details about the used datasets and architectures can be found in App. G.

We compare our method with several SOTA distillation methods that can scale to large high-resolution datasets, including G-VBSM [@shao2023generalized], SRe\(^2\)L [@yin2023squeeze], RDED [@sun2024diversity], CDA [@yin2023dataset], WMDD [@liu2024dataset], Teddy [@yu2024teddy], CUDD [@du2024diversity], EDC [@shao2024elucidating], DWA [@du2024diversity], CV-DD [@cui2025dataset], INFER [@zhang2024breaking], GIFT [@shang2024gift], NRR-DD [@tran2025enhancing], DELT [@shen2024delt] . The results of additional baseline methods, including DataDAM [@sajedi2023datadam], ADD [@zhang2023accelerating], IDM [@zhao2023improved], CDA [@yin2023dataset], WMDD [@liu2024dataset], DATM [@guo2023towards], DREAM [@liu2023dream], and FreD [@shin2023frequency], are presented in App. G. Comprehensive details regarding these approaches are also provided in the same appendix.

By default, CIM employs the subset selection mechanism of RDED. Notably, the proposed CIM is selection-method-agnostic and can seamlessly integrate alternative selection strategies. All distilled datasets synthesized from these baselines are evaluated using the same post-training process. All the hyper-parameters used in our Alg. 1 are general, insensitive, and easy-implemented for all datasets and network architectures (c.f. Sec. 5.3 and App. I for validation). We employ a generalized configuration for \(\mathcal{T}^\prime\) (c.f. for definition), where the size of subset \(|\mathcal{T}^\prime|\) is set as \(300\). We set the number \(N=4\) of images squeezed in a distilled image and number \(M=200\) of compression iteration (c.f. Alg. 1 for definition). More implementation details are provided in App. G.

5.2 Comparison with the SOTA Methods↩︎

Following previous research [@cazenavette2022dataset; @cui2023scaling; @zhao2023improved], we set IPC to 1, 10, and 50 to compare with baselines on varying datasets and networks. As the results reported in Tab. ¿tbl:tb:main95cifar?, our method CIM outperforms other methods on varying datasets and neural networks with different IPC. It is noteworthy that prior information extraction-based solutions like SRe\(^2\)L struggle in scenarios involving small distilled datasets such as CIFAR-100 with \(\texttt{IPC}= 1\) or CIFAR-10 with all IPC, further verifying our claims proposed in Sec. 3.1. Detailed comparison among more datasets and baselines can be found in App. H. Furthermore, we visualize the distilled images of various methods in App. K.

To evaluate the practicality of CIM in real-world settings, we further conduct experiments on the large-scale and high-resolution benchmarks Tiny-ImageNet and ImageNet-1K. The corresponding results are reported in Tab. ¿tbl:tab:main95imagenet95resnet18?. In addition, we compare CIM with several state-of-the-art methods on ResNet-50, as shown in Tab. ¿tbl:tb:main95imagenet95resnet50?. Note that we exclude some methods listed in Tab. ¿tbl:tab:main95imagenet95resnet18? from the ResNet-50 comparison, since their distilled datasets are not available on larger networks, making a fair reproduction infeasible. Furthermore, beyond the commonly used setting of IPC=50, we also report results under larger budgets (e.g., IPC=100) to assess scalability. It is obvious that our CIM achieves the best performance on most scenarios, demonstrating the effectiveness.

An important property of the distilled datasets is their good generalization capability across unseen architectural models. Here we evaluate the generalizability of our distilled datasets when IPC=10. As reported in Tab. ¿tbl:tb:crossarch?, our distilled dataset achieves the best performance on unseen networks, which reflects the good generalizability of the data and labels distilled by our method. Our success stems from our CIM effectively keeps both textural and semantic information in distilled images.

Efficiency is also a key factor during the distillation process. Here, we use a single RTX-4090 GPU for two methods to conduct experiments on Tiny-ImageNet. The reason why we mainly compare with SRe\(^2\)L and G-VBSM is based on their outstanding as efficient optimization-based methods currently. We evaluate the distillation efficiency by recording the run-time cost and peak GPU memory usage of distilling the image. As evidenced in Tab. ¿tbl:tb:efficiency?, our CIM achieves superior efficiency in comparison to SOTA methods, with the exception of RDED, which benefits from an optimization-free paradigm [@sun2024diversity], demonstrating a notable advantage of efficacy and efficiency. Significantly, our algorithm can offer a versatile peak memory capacity, enabling adjustments to batch size dynamically without sacrificing performance. This efficiency is attributed to the fact that our Alg. 1 can independently⁴ optimize images, allowing us to distill them one by one. More comparisons are in App. H.

5.3 Ablation Study↩︎

In this section, we set the default \(\texttt{IPC}= 10\) and employ ConvNet as the network backbone to examine how the components used in our CIM influence the quality of distilled dataset (see App. I for more investigation).

The number of distillation iterations, denoted as \(M\), impacts two aspects: 1) A higher \(M\) enhances CIM’s ability to generate higher-quality images; 2) A lower \(M\) ensures a faster execution of our Alg. 1. Consequently, choosing an optimal iteration number \(M\) represents a balance between quality and speed. As illustrated in Fig. 4 (a), an iteration count of \(M=200\) offers a well-rounded compromise for various datasets. Additionally, it is noteworthy that our CIM exhibits robustness to variations in \(M\). Specifically, setting \(M\) beyond 200 yields negligible differences in performance.

Though we can compress more original images \(\{\mathbf{x}_i\}_{i=1}^N\) into each distilled image \(\widetilde{\mathbf{x}}\) by increasing \(N\) to benefit the feature diversity (c.f. Sec. 4), it also results in less information preserved from each original image \(\mathbf{x}_i\). Fig. 4 (b) showcase that the validation performance rises to the highest on selected four datasets when \(N=4\).

The experimental results in Fig. 4 (c) illustrate the impact of the feature alignment layer, alongside the discussion in Sec. 4. As depicted in Fig. 4 (c), high performance is often achieved through alignment at the middle layer. This outcome likely stems from the varied information encoded at different network depths: shallow layers capture more textural details, whereas logit and deep layers are more adept at encoding semantic information. Thus, aligning at the middle layer enables the distilled dataset to achieve an optimal balance of these information types.

Fig. 4 (d) illustrates the effect of iteration count \(K\) on the information gap. An increased iteration count \(K\) effectively reduces the information gap, enabling the model to capture and retain more features from the original images in the distilled dataset. However, as \(K\) nears 200, further iterations offer minimal gains. This trend suggests that an appropriate choice of \(K\) balances efficient information retention with computational cost across various datasets.

Importantly, the selection mechanism is not intrinsic to our framework. To assess alternative strategies, we replaced RDED’s selection mechanism with approaches such as Random Selection, K-means Clustering [@forgy1965cluster], and Herding [@welling2009herding]. A ResNet-18 model was trained on datasets distilled using these various selection strategies, with results summarized in Tab. ¿tbl:tab:selection95ipc10?. Notably, the results indicate that even with Random Selection, our framework achieves competitive performance. Additional results for the case where IPC= 1 are provided in Tab. 15 in Appendix.

Following prior studies [@yin2023squeeze; @zhao2023dataset] that leverage synthetic datasets in continual learning to assess the quality of synthetic data, we employ the GDumb framework [@prabhu2020gdumb] for continual learning setup. In comparison to SRe\(^2\)L, our study implements a 5-step class-incremental learning approach using ResNet-18 across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, with IPC= 10. The results of these experiments are depicted in Fig. 5. It is evident that our results substantially improve upon the baseline methods.

6 Conclusion↩︎

In this paper, we demonstrate that the primary limitation of current state-of-the-art dataset distillation methods at various scales is the issue of huge information loss. To address this, we introduce the CIM technique, which effectively compresses critical data from images into distilled forms with minimal information loss. Our extensive experiments reveal that CIM markedly surpasses existing SOTA dataset distillation techniques across a range of dataset sizes and network architectures. Additionally, we highlight the efficiency of CIM by showcasing its ability to distill the ImageNet-1k dataset in just 80 minutes.

Acknowledgment↩︎

This work was supported in part by the National Science and Technology Major Project (No. 2022ZD0115101), Research Center for Industries of the Future (RCIF) at Westlake University, Westlake Education Foundation, and Westlake University Center for High-performance Computing.

7 Limitations↩︎

Although our CIM significantly outperforms existing SOTA methods, its primary limitation, as discussed in Section 5.2, is that it cannot surpass the optimization-free paradigm (e.g., RDED [@sun2024diversity]) in terms of efficiency. This constraint limits its applicability in diverse real-world scenarios.

8 Proof of Proposition [prop:subop95label]↩︎

9 Selecting Key Subsets from Original Dataset↩︎

Our goal is to devise an effective method for identifying the crucial samples that are instrumental in benefiting the Relabel process. Motivated by the Summary in Section 3.1 and Proposition [prop:subop95label]—which suggests that each chosen sample pair \((\mathbf{x}, y)\) is supposed to receive an informative and accurate label \(\phi_{\boldsymbol{\theta}_\mathcal{T}}(\mathbf{x})\) from the pre-trained model during the Relabel phase—the goal then relaxes to find the subsets of \(\mathcal{T}\) that include samples that are relabeled most accurately by the pre-trained model \(\phi_{\boldsymbol{\theta}_\mathcal{T}}\). Thus, we introduce a loss-based importance score \(s\) for each sample pair \((\mathbf{x}, y)\), defined as \(s = -\ell(\phi_{\boldsymbol{\theta}_\mathcal{T}}(\mathbf{x}), y)\). The key sample selection procedure is elaborated below.

Let \(\mathcal{T}_c := \{ (\mathbf{x}, y) \mid (\mathbf{x}, y) \in \mathcal{T}, y = c \}\) represents the subset of the dataset \(\mathcal{T}\), containing only those samples \((\mathbf{x}, y)\) that are labeled with the class \(c\). For each class data \(\mathcal{T}_c\), we identify the key samples \(\mathbf{x}\) based on their importance scores \(s\), forming \(\texttt{IPC}\) subsets of key samples as \[\begin{align} \mathcal{S}_c = \left\{ \{ \mathbf{x}_{(j,i)}, y_{(j,i)} \}_{j=1}^N \right\}_{i=1}^{\texttt{IPC}} \\ \mathrm{s.t.} \;\; s_{(j,i)} = -\mathrm{CE}(\phi_{\boldsymbol{\theta}_\mathcal{T}}(\mathbf{x}_{(j,i)}), y_{(j,i)}) \geq \bar{s} \,, \end{align}\] where \(\bar{s}\) denotes a predetermined threshold⁵, \(N\) indicates the number of samples within each selected subset, and CE denotes the CrossEntropyLoss. We further simplify the whole procedure due to the computational overhead and diversity issue, as explained below

1. computing the scores \(s\) for all samples \(\mathbf{x}\) in data \(\mathcal{T}_c\) presents a significant computational challenge;

2. focusing solely on samples that closely align with the true label can lead to a lack of diversity.

Therefore, we utilize a pre-selection strategy inspired by [@sun2024diversity], which involves selecting a subset⁶ \(\mathcal{T}_c^\prime \subset \mathcal{T}_c\) uniformly at random to serve as a proxy for the entire \(\mathcal{T}_c\). Such a pre-selection strategy not only promotes diversity in the data but also lessens the computational load [@sun2024diversity], thereby laying the groundwork for our subsequent score-based sample selection process.

10 Proof of Theorem 1↩︎

We initiate our derivation by examining the Kullback-Leibler divergence between the effective information distributions of \(x _i\) and \(\hat{x} _i\) as observed by the group \(\mathcal{R}\):

\[\begin{align} &\text{D} _\text{KL}(\mathcal{P}_{x _i | \mathcal{R}} || \mathcal{P}_{\hat{x} _i | \mathcal{R}}) = \mathbb{E} _z \left[ \log \frac{\mathcal{P}_{x _i | \mathcal{R}}(z)}{\mathcal{P}_{\hat{x} _i | \mathcal{R}}(z)}\right] &= \mathbb{E} _z \left[\log \mathcal{P}_{x _i | \mathcal{R}}(z) - \log \mathcal{P}_{\hat{x} _i | \mathcal{R}}(z)\right] \\ \nonumber &\leq \mathbb{E} _z \left[| \log \mathcal{P}_{x _i | \mathcal{R}}(z) - \log \mathcal{P}_{\hat{x} _i | \mathcal{R}}(z) |\right] \, . \end{align}\]

By applying kernel density estimation to \(p(\cdot)\), we obtain:

\[\begin{align} &\mathbb{E} _z \left[| \log \mathcal{P}_{x _i | \mathcal{R}}(z) - \log \mathcal{P}_{\hat{x} _i | \mathcal{R}}(z) | \right]\\ &=\mathbb{E} _{\xi _j} \left[| \log \sum _{\xi _k}\left[\frac{1}{(2\pi)^{\frac{1}{d}}} \exp(\frac{- \| \xi _j(x _i) - \xi _k(x _i) \|^2}{2})\right] \right. \\ & \qquad \qquad \left. - \log \sum _{\xi _k}{\left[\frac{1}{(2\pi)^{\frac{1}{d}}} \exp(\frac{- \| \xi _j(\hat{x} _i) - \xi _k(\hat{x} _i) \|^2}{2})\right]} |\right]\, , \end{align}\]

where \(d\) denotes the dimension of \(x\) and a primary term exists within \(\log(\cdot)\). Therefore, we approximate:

\[\begin{align} &\approx \mathbb{E}_{\xi_j} \left[ \left| \sum_{\xi_k} \left[ \log\left(\frac{1}{(2\pi)^{\frac{1}{d}}} \exp\left(\frac{- \| \xi_j(x_i) - \xi_k(x_i) \|^2}{2}\right)\right)\right] \right. \right. \\ & \quad - \left. \left. \sum_{\xi_k} \left[ \log\left(\frac{1}{(2\pi)^{\frac{1}{d}}} \exp\left(\frac{- \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2}{2}\right)\right) \right] \right| \right] \\ &= \mathbb{E}_{\xi_j} \left[ \left| \sum_{\xi_k} \left[ \frac{- \| \xi_j(x_i) - \xi_k(x_i) \|^2}{2} \right] - \sum_{\xi_k} \left[ \frac{- \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2}{2} \right] \right| \right] \\ &= \mathbb{E}_{\xi_j} \left[ \left| \sum_{\xi_k} \left[ \frac{- \| \xi_j(x_i) - \xi_k(x_i) \|^2}{2} \right] - \sum_{\xi_k} \left[ \frac{- \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2}{2} \right] \right| \right] \\ &\leq \mathbb{E}_{\xi_j} \left[ \sum_{\xi_k} \left[ \left| \frac{- \| \xi_j(x_i) - \xi_k(x_i) \|^2}{2} - \frac{- \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2}{2} \right| \right] \right] \\ &\leq \mathbb{E}_{\xi_j} \left[ \sum_{\xi_k} \left[ \left| \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2 - \| \xi_j(x_i) - \xi_k(x_i) \|^2 \right| \right] \right] \\ &= \mathbb{E}_{\xi_j} \left[ \sum_{\xi_k \neq \xi_j} \left[ \left| \| \xi_j(\hat{x}_i) - \xi_k(\hat{x}_i) \|^2 - \| \xi_j(x_i) - \xi_k(x_i) \|^2 \right| \right] \right] \\ &= \mathbb{E}_{\xi_j} \Bigg[ \sum_{\xi_k \neq \xi_j} \Bigg( \left| \| \xi_j(\hat{x}_i) \|^2 - 2\langle \xi_j(\hat{x}_i), \xi_k(\hat{x}_i) \rangle \right. + \| \xi_k(\hat{x}_i) \|^2 - \| \xi_j(x_i) \|^2 \\ & \quad + 2\langle \xi_j(x_i), \xi_k(x_i) \rangle - \| \xi_k(x_i) \|^2 \Bigg| \Bigg) \Bigg] \\ &\leq (|\mathcal{R}|-1) \cdot \mathbb{E}_{\xi_j} \left[ \left| \| \xi_j(\hat{x}_i) \|^2 - \| \xi_j(x_i) \|^2 \right| \right] + \sum_{\xi_k \neq \xi_j} \left[ \left| \| \xi_k(\hat{x}_i) \|^2 - \| \xi_k(x_i) \|^2 \right| \right] \\ & \quad + \mathbb{E}_{\xi_j} \left[ \sum_{\xi_k \neq \xi_j} \left[ \left| 2\langle \xi_j(x_i), \xi_k(x_i) \rangle - 2\langle \xi_j(\hat{x}_i), \xi_k(\hat{x}_i) \rangle \right| \right] \right] \\ &\approx (|\mathcal{R}|-1) \cdot \mathbb{E}_{\xi_j} \left[ \left| \| \xi_j(\hat{x}_i) \|^2 - \| \xi_j(x_i) \|^2 \right| \right] + \sum_{\xi_k \neq \xi_j} \left[ \left| \| \xi_k(\hat{x}_i) \|^2 - \| \xi_k(x_i) \|^2 \right| \right] + o(1) \\ &\approx (|\mathcal{R}|-1) \cdot \mathbb{E}_{\xi_j} \left[ \left| \| \xi_j(\hat{x}_i) \|^2 - \| \xi_j(x_i) \|^2 \right| \right] & \quad + \lambda \end{align}\]

To minimize this term, we can use an alternative one:

\[\mathbb{E} _{\xi _j} \left[ \| \xi _j(\hat{x} _i) - \xi _j(x _i)\|^2 \right] \, .\]

11 Detailed Analysis for Existing Information Extraction-Based Approaches↩︎

11.0.0.1 Results without relabel.

We evaluate the data with and without employing relabeling. The results are presented in Table 1. The visualizations of the images synthesized by state-of-the-art methods are presented in Section 17.

12 Detailed Framework of Our CIM↩︎

The structure of our CIM framework is outlined in Algorithm 6.

13 Experiment Details↩︎

13.0.0.1 Datasets.

In addition to the datasets described in Section 5.1, we note that prevalent dataset distillation techniques struggle to scale to large, high-resolution datasets. In Table 2, we present some information about the dataset, including the number of classes, the number of images per class in the training set, and the test set.

13.0.0.2 Models.

The experiment employed a multitude of pre-trained models, and we delineated their accuracies in Table 3. These results are furnished solely for reference purposes.

13.0.0.3 Baselines.

We benchmark our proposed CIM against a range of SOTA distillation techniques capable of handling large, high-resolution datasets.

• G-VBSM [@shao2023generalized] surpasses one-sided methods like SRe\(^2\)L by creating synthetic datasets with richer information and better generalization across various backbones, layers, and statistics.

• SRe\(^2\)L [@yin2023squeeze] is a novel entrant that efficiently handles ImageNet-1K, significantly outpacing other methods in managing large, high-resolution datasets and serving as our primary comparison point.

• RDED [@sun2024diversity] enables the compression of large-scale, high-resolution datasets while maintaining diversity and realism, significantly reducing the time required for training neural networks like ResNet-18 on ImageNet-1K.

• DataDAM [@sajedi2023datadam] efficiently distills images across multiple resolutions and scales by matching spatial attention maps between real and distilled samples at various layers within families of randomly initialized neural networks.

• ADD [@zhang2023accelerating] demonstrates effective scalability across varying dataset resolutions, enhancing distillation speed through model augmentation.

• IDM [@zhao2023improved] presents an efficient dataset condensation technique utilizing distribution matching, offering a scalable alternative to computationally demanding optimization-focused methods [@zhao2020dataset; @cazenavette2022dataset].

• CDA [@yin2023dataset] achieves superior accuracy on large-scale datasets like ImageNet-1K and 21K and significantly narrows the performance gap compared to full-data training counterparts.

• WMDD [@liu2024dataset] presents a novel dataset distillation approach that employs the Wasserstein distance to enhance distribution matching, achieving state-of-the-art performance by effectively capturing the essential representations of extensive datasets in synthetic forms.

• DATM [@guo2023towards] stands out by occasionally surpassing the training performance of the full original dataset, for instance, achieving \(\texttt{IPC}=100\) on CIFAR-100.

• DREAM [@liu2023dream] introduces an efficient technique while also delivering the most remarkable results.

• FreD [@shin2023frequency] is a novel parameterization method for dataset distillation that operates in the frequency domain, significantly reducing the budget for synthesizing a small-sized synthetic dataset while preserving the original dataset’s information and consistently improving the performance of existing distillation methods.

13.0.0.4 Evaluating main results.

For both dataset distillation and performance evaluation, we employ identical neural network architectures. Consistent with previous studies [@cazenavette2022dataset; @cui2023scaling; @zhao2023improved], we use Conv-3 for CIFAR-10 and CIFAR-100 distillation tasks and Conv-4 for Tiny-ImageNet (with the exception of DREAM, which utilizes Conv-3) and ImageNet-1K distillation. In line with [@cazenavette2022dataset; @cui2023scaling], MTT and TESLA apply a reduced resolution for distilling \(224 \times 224\) images. According to [@yin2023squeeze], for retrieving and evaluating distilled datasets, SRe\(^2\)L and CIM adopt ResNet-18.

13.0.0.5 Evaluating the distilled dataset.

We detail the hyperparameter configurations for distilling datasets in Table [tbl:tab:settings95distill]. Consistent with recent works [@yin2023dataset; @yin2023squeeze; @shao2023generalized], the evaluation on the distilled dataset follows the parameters outlined in Table [tbl:tab:settings95eval]. Furthermore, we implement Differentiable Siamese Augmentation (DSA) as described by [@zhao2021DSA] to enhance images during both the distillation and evaluation phases of our experiments.

13.0.0.6 Differentiable Siamese Augmentation (DSA).

We employ DSA (Differentiable Siamese Augmentation) as a method for image augmentation. To enhance clarity, we outline the DSA operations utilized in Table 4, along with their corresponding transformations and probabilities.

13.0.0.7 Factor Crop.

We have integrated a cropping method termed ‘factor crop’, which is applied prior to DSA. This technique enhances sample diversity by precisely extracting regions from specific areas of the images and resizing them to their original dimensions. As an augmentation method, it bolsters the model’s generalizability, as depicted in Figure 7. This approach functions as a substitute for RandomCrop, thereby preserving the semantic integrity of compressed images within the distilled dataset.

14 Experiment Results↩︎

14.0.0.1 Comparison with more datasets and baselines.

In addition to the experiments discussed in Section 5.2, we further benchmark our proposed CIM against a broader set of baselines, encompassing recent contributions [@yu2023dataset; @shao2023generalized; @sun2023diversity; @liu2023dataset]. The outcomes, presented in Table ¿tbl:tab:results95subset95imagenet? and Table 5, consistently affirm the superior performance of CIM in dataset distillation tasks.

14.1 Efficiency Comparison↩︎

Beyond assessing performance in Section [sec:efficiencycomp], we expand our evaluation to include additional baselines. Results presented in Table 6 underscore the exceptional efficiency of our proposed CIM, which also requires the least GPU memory.

15 Ablation↩︎

15.0.0.1 Selection.

To compare our selection strategy with others, namely Random, K-means [@forgy1965cluster], and Herding [@welling2009herding], we trained ResNet-18 using datasets distilled through various strategies, as delineated in Tables 7.

16 Continual Learning↩︎

In comparison to SRe\(^2\)L, our study implements a five-step class-incremental learning approach using ResNet-18 across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, each with an IPC setting of 10. The results of these experiments are depicted in Figures 8, 9, and 10 for CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively.

17 Visualization↩︎

17.0.0.1 Baselines.

Within the scope of CIFAR-10 distillation under the \(\texttt{IPC}=10\) setting, we illustrate the visual representations of distilled datasets. This includes visualizations for ADD [@zhang2023accelerating] in Figure 13, DataDAM [@sajedi2023datadam] in Figure 14, SRe\(^2\)L [@yin2023squeeze] in Figure 16, and DREAM [@liu2023dream] in Figure 15. Distilled images of each method are generated starting from actual images, showcased in Figures 11 and 12.

17.0.0.2 A simple squeezing-based method.

The image-squeezing process entails resizing and concatenating images to facilitate dataset distillation. For example, consider the manipulation of \(4\) images, each originally sized at \(224 \times 224\) pixels. The initial step involves downsizing each image to \(112 \times 112\) pixels. Subsequently, these reduced images are merged into a single composite image, effectively reverting to the original resolution of \(224 \times 224\) pixels. This approach underpins a simplistic, squeezing-based dataset distillation method, whereby \(N\) randomly selected original images are compressed into one distilled image to compose a condensed dataset. In the context of distilling CIFAR-10 with an \(\texttt{IPC}=10\) configuration, we exhibit the visual outcomes of this process for diverse settings: \(N=1\) in Figure 17, \(N=4\) in Figure 18, \(N=9\) in Figure 19, \(N=16\) in Figure 20, and \(N=25\) in Figure 21.

17.0.0.3 Our proposed CIM.

With an IPC setting of 10, we illustrate the distilled datasets generated by our proposed CIM. These include visualizations for CIFAR-10 in Figure 22, CIFAR-100 in Figure 23, Tiny-ImageNet in Figure [fig:visualization95lic95tinyimagenet], and ImageNet-1k in Figure [fig:visualization95lic95imagenet1k].

---

1. Features are extracted using a model trained on the full dataset, followed by visualization with T-SNE [@van2008visualizing].↩︎

2. See App. C for details of RDED.↩︎

3. In the context of image data, these transformations encompass a range of nonlinear and linear operations, including techniques for image data augmentation.↩︎

4. Traditional distillation techniques necessitate the concurrent synthesis of a set of images to ensure collective quality [@yin2023squeeze; @cazenavette2022dataset; @guo2023towards]. This issue stems from the nature of these methods, which involve using a distilled dataset to approximate the original dataset with a similarity metric.↩︎

5. This threshold is used for selecting \(\texttt{IPC}\times N\) samples for further processing.↩︎

6. We use the default size of this uniform-randomly selected subset in [@sun2024diversity]. see details in Section 5.↩︎