1 Introduction↩︎

The rapid progress of machine learning has been driven by increasingly large datasets and models, enabling state-of-the-art performance but posing severe challenges for researchers with limited computational or storage resources [1]. While model pruning [2]–[4] and quantization [5], [6] improve model efficiency, scalability from the data perspective is primarily addressed through data distillation [7], [8] and coreset selection [9]–[11]. In modern computer vision, the explosive growth of labeled datasets [12] makes training large models on millions of images computationally prohibitive [1], [13]. Dataset distillation tackles this by “compressing” the full dataset \(\mathcal{D}\) into a much smaller synthetic subset \(\mathcal{S}\) such that training on \(\mathcal{S}\) yields performance comparable to \(\mathcal{D}\). Earlier approaches, including coreset selection [9]–[11] and gradient-matching methods [14]–[19], optimize \(\mathcal{S}\) so that the training trajectory or feature statistics of a model trained on \(\mathcal{S}\) approximate those from \(\mathcal{D}\). Despite their promise, these methods often rely on costly bi-level optimization, exhibit sensitivity across architectures, and struggle to capture rare modes of the data distribution [19], [20].

The advent of large pre-trained generative models [21], [22] has transformed distillation: generative priors allow synthetic sampling and representational coverage of complex image manifolds. The generative model acts like a container storing the knowledge of \(\mathcal{D}\) and using it to generate \(\mathcal{S}\). By leveraging generative models, recent distillation works achieve higher fidelity, greater diversity and better downstream utility. Within this trend, diffusion-based distillation [20], [23]–[26] has emerged as state-of-the-art. Yet, many generative approaches remain training-based: they require fine-tuning the generator [24], [25], optimizing synthetic images via bi-level loops [23], which raises cost and complexity. Moreover, none explicitly enforce that synthetic trajectories remain faithful to the latent data manifold, i.e., once guided toward data distribution modes, samples may drift off-manifold [27], [28].

We explore a training free generative model based data distillation paradigm where achieve comparable performance training based methods while outperforming most of the existing methods. Our core contributions are as follows considering the aforementioned issues:

• A fully training-free dataset distillation pipeline that uses only a pre-trained generator and is inference-only.

• A layer‐wise hierarchical clustering of VAE latent features to select IPC (images‐per‐class) centroids that represent the class modes in the VAE feature space. The IPC centroid selection from each level of the tree generated during hierarchical clustering covers coarse (levels closer to the root node) to fine modes (levels closer to the leaf nodes) without optimization.e generative model based data distillation.

• A manifold‐guidance strategy to offer trajectory correction introduced by mode‐guidance: guiding towards IPC centroids while constraining updates to the local latent tangent subspace to maintain data-manifold fidelity.

2 Related Works↩︎

Dataset distillation has found importance for diverse applications such as continual learning [15], [29], privacy preservation [30], [31], neural architecture search [29], [32], and model interpretability [33]. Early approaches relied on coreset selection or gradient matching to identify representative subsets [34]–[37]. While computationally efficient, these methods are often architecture-specific and limited in capturing complex, multimodal distributions.

Generative-model–based distillation. The advent of pre-trained diffusion and latent generative models reshaped dataset distillation by enabling high-fidelity, conditional synthesis directly from generative priors. Training-based diffusion methods such as DM [32], Min–Max Diffusion [23], D\(^{4}\)M [24], and GLAD [37] fine-tune the generative model or optimize synthetic images via a min–max or gradient-matching objective to balance diversity and representativeness. Although effective, these approaches are computationally expensive and often require additional optimization stages for each dataset.

Training-free diffusion methods. More recent works have shifted toward training-free paradigms that avoid fine-tuning. Diffusion Transformers (DiT) [38] and Latent Diffusion Models (LDM) [39] can generate synthetic datasets directly, but unguided sampling often yields semantically diffuse or redundant samples. Mode-Guided Diffusion (MGD) [20] introduces cluster-based guidance to steer sampling toward class centroids, improving sample diversity and semantic consistency. However, MGD and similar methods rely purely on Euclidean attraction, leading to potential off-manifold drift and degraded fidelity.

A few concurrent efforts address more principled guidance mechanisms. Information-Guided Diffusion [40] uses an information-theoretic objective that depends on a separately trained classifier; Influence-Guided Diffusion [26] computes influence scores from pre-trained classifiers to steer the denoising trajectory; and text-guided methods [25] or OOD-aware generators [41] rely on additional conditioning or discriminators. While these methods enhance control, they require auxiliary networks beyond the diffusion backbone. In contrast, our proposed ManifoldGD is a fully training-free method that uses only a pre-trained diffusion model and its accompanying VAE feature space. By constraining the guidance direction to the tangent space of a locally estimated diffusion manifold, ManifoldGD enforces geometric consistency, preventing off-manifold drift and improving representativeness, diversity, and without any additional supervision or model fine-tuning.

3 Method↩︎

3.1 Preliminaries↩︎

Let the real dataset be \(\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N}, \quad x_i \in \mathbb{R}^D,\; y_i \in \{1, \dots, C\}\), where \(x_i\) denotes the input sample and \(y_i\) the associated class label. Under the standard Empirical Risk Minimization (ERM) [42], a classifier \(f_\theta\) with parameters \(\theta\) is trained by minimizing the expected loss \(\mathcal{L}_{\mathcal{D}}(\theta) = \frac{1}{N}\sum_{i=1}^N \ell\big(f_\theta(x_i), y_i\big)\), where \(\ell(\cdot,\cdot)\) denotes the task-specific loss function.

Dataset distillation seeks to construct a significantly smaller synthetic dataset \(\mathcal{S} = \{(\tilde{x}_j, \tilde{y}_j)\}_{j=1}^{M}\), \(M \ll N\), such that training on \(\mathcal{S}\) yields parameters \(\theta^{*}_{\mathcal{S}} = \arg\min_{\theta}\frac{1}{M}\sum_{j=1}^{M}\ell\big(f_\theta(\tilde{x}_j), \tilde{y}_j\big)\), whose downstream performance approximates that of \(\theta^{*}_{\mathcal{D}}\). Formally, synthetic dataset minimizes the discrepancy in population risk \[\begin{align} \mathcal{S}^{*} =\arg\min_{\mathcal{S}} \left| \operatorname*{\mathbb{E}}_{(x,y)\sim\mathcal{D}} \!\ell\!\left(f_{\theta_{\mathcal{S}}^{*}}(x),y\right)-\operatorname*{\mathbb{E}}_{(x,y)\sim\mathcal{D}} \!\ell\!\left(f_{\theta_{\mathcal{D}}^{*}}(x),y\right) \right| \end{align}\] In this way, the distilled dataset is explicitly chosen to preserve the predictive behavior of a model trained on the full dataset.

A diffusion model defines a forward noising process \(\{x_t\}_{t=0}^{T}\) that gradually perturbs data samples \(x_0 \!\sim\! p_{\mathrm{data}}(x)\) into Gaussian noise \(x_T \!\sim\! \mathcal{N}(0, I)\). The corresponding reverse (denoising) process is parameterized by a time-dependent Stein’s score function \(s_\theta(x_t, t)\) [43] that is trained to match the log-probability density of \(x_t\), that is \(\nabla_{x_t} \log p_t(x_t)\). The score \(s_\theta(x_t, t)\) defines a vector field on the sample space that points towards regions of higher probability under the marginal distribution \(p_t(x_t)\) at noise level \(t\). At each denoising step, updating \(x_t\) in the direction of \(s_\theta(x_t, t)\) incrementally increases its likelihood, projecting the sample onto the evolving data manifold and thereby performing progressive denoising.

For conditional generation, we consider the conditional density \(p_t(x_t|c)\), where \(c\) denotes an external conditioning signal. In the context of data distillation, \(c\) corresponds to an IPC centroid, that is, a representative \(\mathrm{mode}\) associated with a class. Such centroids (or \(\mathrm{mode}\)s) may be obtained through a variety of procedures, one example being the extraction of class-representative modes in a variational autoencoder (VAE) feature space (see 3.3 for a detailed discussion). Under this formulation, the score of the conditional distribution admits the standard decomposition: \[\begin{align} \label{eq:score95decomp} \nabla_{x_t} \log p_t(x_t \mid c) &= \underbrace{ \nabla_{x_t}\log p_t(x_t) }_{\textcolor{myblue}{\text{(1) Marginal Denoising}}}+ \underbrace{ \nabla_{x_t}\log p_t(c \mid x_t) }_{\textcolor{myred}{\text{(2) Mode Guidance}}} \end{align}\tag{1}\] Here, the term drives the dynamics toward regions of high data density. In the limit \(t \to 0\), as the injected noise vanishes, the transition density \(p_t\) converges to the underlying data distribution \(p_{\mathrm{data}}\) [44]. The term introduces a semantic correction by enforcing attraction toward the conditional mode associated with the signal \(c\). Weighted, classifier-based, and prototype-based guidance can be interpreted as variants of this mechanism [45], [46]. Conceptually, recovers the coarse geometric structure induced by the diffusion prior, whereas steers the sampling trajectory toward the conditional distribution, that is, the semantic mode of the corresponding class. When this attraction is excessively strong or misaligned with the intrinsic manifold geometry, the resulting trajectory may deviate from the true data manifold [27], [28] (see  1).

3.2 Manifold-Aware Mode Guidance in Conditional Diffusion↩︎

The mode guidance term in 1 can be expressed as a kernel affinity between between the diffusion sample \(x_t\) and the modes \(c\). That is, \(k_\phi(x_t, c) = \exp\!\big(-\phi(\|x_t - c\|_2)\big)\) where \(\phi(\cdot)\) defines a positive semi-definite potential function that governs the structure of the affinity ¹. The resulting log-posterior gradient is expressed as \(\nabla_{x_t} \log p_t(c|x_t)= -\phi'(\|x_t - c\|_2)\, \frac{x_t - c}{\|x_t - c\|_2}\), yielding the generalized mode-guidance vector, \(g_{\mathrm{mode}}^t = -\phi'(\|x_t - c\|_2)\, \frac{x_t - c}{\|x_t - c\|_2}\). Under the quadratic potential \(\phi(r)=\tfrac{r^2}{2\sigma_t^2}\), the expression reduces to the standard Gaussian form \(g_{\mathrm{mode}}^t = -\tfrac{1}{\sigma_t^2}(x_t - c)\), aligning with existing mode-guided diffusion formulations [20], [24]. The complete denoising step can be expressed as, \[\begin{align} \label{eq:kernel95overall95denoising} x_{t-1} = x_t + \eta_t \big[ s_\theta(x_t, t) + g^t_{\mathrm{mode}} \big] + \sqrt{\beta_t}\,\epsilon_t \end{align}\tag{2}\]

Mode guidance, using vectors like \(g^t_{\mathrm{mode}} = -\tfrac{1}{\sigma_t^2}(x_t - c)\) assumes that this direction is meaningful in the ambient Euclidean space. The true generative manifold is typically a curved, lower-dimensional subset of the ambient space [27], [28]. Prior work shows that modeling group actions tied to nuisance variables enables representations that separate signal from nuisance structure on such manifolds [1]. Let \(\mathcal{M} \subseteq \mathbb{R}^d\) denote the ambient data space. We define:

• The true data manifold \(\mathcal{M}_{\mathrm{data}}\), a low-dimensional submanifold on which \(p_{\mathrm{data}}(x)\) is supported.

• The diffusion manifold \(\mathcal{M}_t\) at timestep \(t\), supporting the noisy data marginal \(p_t(x_t)\). By construction, \(\mathcal{M}_{\mathrm{data}} \subseteq \mathcal{M}_t\), and \(\mathcal{M}_t \to \mathcal{M}_{\mathrm{data}}\) as \(t \to 0\).

Then, at any \(x_t \in \mathcal{M}_t\) we have the tangent space \(\mathcal{T}_{x_t}\mathcal{M}_t\) and normal space \(\mathcal{N}_{x_t}\) defined using,

The off-manifold component is quantified by projection onto the normal space: \[\langle g_t^{\mathrm{mode}}, n_t \rangle = -\phi'(\|x_t - c\|_2)\, \bigg\langle \frac{x_t - c}{\|x_t - c\|_2},\, n_t \bigg\rangle, n_t \in \mathcal{N}_{x_t},\] where \(\phi(\cdot)\) is the mode potential in affinity-based guidance. As diffusion proceeds (\(t \to 0\)), \(p_t(x_t)\) becomes sharply concentrated near \(\mathcal{M}_t \to \mathcal{M}_{\mathrm{data}}\), so even small normal components can substantially reduce likelihood under \(p_{\mathrm{data}}(x)\).

Unconstrained guidance \(g_{\mathrm{mode}}^t\) enforces semantic attraction but violates manifold geometry, yielding invalid generations [27], [28]. Manifold-corrected guidance, typically via tangent space projection, maintains local fidelity to \(\mathcal{M}_t\) and \(\mathcal{M}_{\mathrm{data}}\). We decompose guidance into tangent and normal components via orthogonal projection onto the estimated diffusion manifold \(\mathcal{M}_t\). Let \(P_{\mathcal{T}_t}\) and \(P_{\mathcal{N}_t} = I - P_{\mathcal{T}_t}\) [47] denote orthogonal projectors onto \(\mathcal{T}_{x_t}\mathcal{M}_t\) and its orthogonal complement, respectively. The tangential component \(P_{\mathcal{T}_t} g^t_{\mathrm{mode}}\) aligns the update with the manifold while preserving local validity of the denoising process.

Alternatively, explicit normal component subtraction provides independent control over semantic attraction and manifold correction: \[\begin{align} g^t_{\mathrm{manifold}}(x_t; c) = g^t_{\mathrm{mode}}(x_t; c) - P_{\mathcal{N}_t} g^t_{\mathrm{mode}}(x_t; c) \end{align}\] where the weighting parameters control the relative influence of semantic guidance and geometric correction, respectively. Thus, the full manifold-conditioned scoring function is given by \(s^t_{\mathrm{manifold}}(x_t) = s_\theta(x_t, t) + g^t_{\mathrm{manifold}}\) and the corresponding sampling step at each reverse diffusion iteration is \(x_{t-1} = x_t + \eta_t s^t_{\mathrm{manifold}}(x_t) + \sqrt{\beta_t} \epsilon_t\). This framework enables fine-grained control over manifold consistency and semantic alignment.

3.3 Algorithmic Details for Manifold-Projected Correction↩︎

In this section, we summarize the practical steps of our manifold-projected correction method for enforcing tangent-aligned guidance in diffusion-based dataset distillation. A full algorithmic specification is provided in the supplementary material.

: VAE Latent Encoding and Diffusion. Following prior works [1], [20], [23], each input image is encoded into a latent feature via a variational autoencoder (VAE), yielding \(x_t\) that follows the diffusion process in latent space. A pretrained diffusion model (e.g., DiT) then performs the reverse denoising to map \(x_t\) back toward the clean latent distribution approximating \(p_{\mathrm{data}}\).

: Building Mode-Guided Coresets (IPCs) and Local Manifolds. We construct a coreset of IPC centroids \(c_s\) via divisive (bisecting \(k\)-means) hierarchical clustering on class-wise VAE latents and use these centroids to build time-dependent local manifold \(\mathcal{M}_t\) aligned with the diffusion noise level. Let the divisive tree leaves at depth \(d\) be \(\mathcal{L}_d\) for \(d\!=\!0,\dots,L\) (root depth \(0\)). Given a start level \(s_{\mathrm{start}}\in[0,L]\) (controls coarse→fine bias) and IPC budget \(K\), we (i) pick one node per level in a coarse→fine sweep beginning at \(s_{\mathrm{start}}\) and going down to the root, and (ii) if more IPCs are required, we fill the remaining quota by randomly selecting from the leaf nodes. This yields a deterministic coreset that favors global/generic modes when classes heavily overlap (higher \(s_{\mathrm{start}}\)) and progressively adds finer/specific modes. For each selected \(c_s\), we define its neighborhood \(\mathcal{N}_s\) (local region in latent space) which captures structure similar to \(c_s\), so as to cover both broad and fine-grained variations of the data.

Constructing the Noisy Local Manifold at Diffusion Timestep \(t\). To match the current noise level in the diffusion process (at time \(t\)), we “forward-diffuse" all points in \(\mathcal{N}_s\) by adding Gaussian noise with variance \((1-\bar{\alpha}_t)\). The resulting set, \(\mathcal{M}_t^{(s)}\), approximates the structure of the data manifold at this particular noise scale. \[\begin{align} \label{eq:Mt95construct} \mathcal{M}_t^{(s)} = \mathcal{N}_s + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}\!\big(0,\,(1-\bar{\alpha}_t)I\big) \end{align}\tag{3}\] \(\mathcal{M}_t^{(s)}\) represents the diffusion manifold at timestep \(t\) that smoothly embeds \(\mathcal{N}_s \subseteq \mathcal{M}_{\mathrm{data}}\) as \(t\!\to\!0\).

Estimating Local Geometry (Tangent/Normal Subspaces) and Manifold Correction. Given a latent sample \(x_t\), we identify its \(K_t\) nearest neighbors within the local manifold patch \(\mathcal{M}_t^{(s)}\) and compute the empirical covariance \(C_t\) of these neighbors. The leading \(d\) eigenvectors of \(C_t\) define a basis for the tangent subspace \(\mathcal{T}_{x_t}\mathcal{M}_t\), while the remaining orthogonal directions form the normal subspace \(\mathcal{N}_{x_t}\). We construct the projection operators \(P_{\mathcal{T}_t}\) and \(P_{\mathcal{N}_t}\) accordingly. Since the generic mode-guidance vector \(g^t_{\mathrm{mode}}(x_t;c_s)\) may possess components orthogonal to the manifold, we perform a manifold correction by subtracting its normal projection. This ensures that guidance remains tangent-aligned, thereby preserving both semantic consistency and geometric validity during the diffusion process.

A detailed algorithmic explanation of the entire process can be seen in Supplementary 6.

4 Experimental Results↩︎

Setup. We evaluate our method using the hard-label protocol [20], [23], [24], the most challenging and unbiased setting for dataset distillation. Unlike soft-label or feature-matching variants that leak teacher information [48], this setup trains student networks solely on \(\mathcal{S}\) and discrete labels, ensuring improvements reflect true data fidelity. Experiments are conducted at image resolution \(256\times256\) using ConvNet-6, ResNetAP-10, and ResNet-18 classifiers, with IPC = 10, 20, and 50 (10 being the most challenging due to least number of samples per class). We distill datasets for ImageNette [49], ImageWoof [50], and ImageNet-100 [51], and train classifiers from scratch for fair comparison. Each setting is repeated three times with different seeds, and mean accuracy is reported. Baselines. We follow relevant baselines in 2 for comparison.

Metrics. Following prior works [20], [23], [24], we assess both downstream performance and distributional fidelity using five complementary metrics: Classification Accuracy. We train the classifiers (ConvNet-6, ResNetAP-10, and ResNet-18) from scratch on \(\mathcal{S}\) and test on \(\mathcal{D}_{\text{test}}\). \(\mathrm{Acc}_{\mathcal{S}\to\mathcal{D}}\) reflects task fidelity and the semantic validity of the distilled samples. Fréchet Inception Distance (FID). FID [52] is computed between Inception-V3 embeddings of real (\(\mathcal{D}_{\text{train}}\)) and synthetic images (\(\mathcal{S}\)) to quantify global distribution alignment and perceptual realism. Lower values indicate higher visual fidelity. \(\ell_2\) and MMD. We compute feature-space \(\ell_2\) distance and Maximum Mean Discrepancy (MMD) between ResNet-18 embeddings (pretrained on ImageNet1k) of \(\mathcal{D}_{\text{test}}\) and \(\mathcal{S}\), capturing local and global manifold alignment respectively [53], [54]. Smaller values imply better consistency of \(\mathcal{S}\) with \(\mathcal{D}_{\text{test}}\). Representativeness and Diversity. To measure coverage and spread, we use cosine-based minimax scores computed from ResNet-18 (pretrained on ImageNet1k) embeddings: \(\text{Rep} = \min_i \max_j \cos(f_{\mathcal{S}}^i, f_{\mathcal{D_{\text{train}}}}^j)\), \(\text{Div} = 1 - \max_{i\neq j} \cos(f_{\mathcal{S}}^i, f_{\mathcal{S}}^j)\), \(\cos(,)\) represents cosine similarity. Higher indicates better coverage of real data and higher reflects reduced redundancy.

4.1 Comparison with Existing Methods↩︎

ImageNette and ImageNet-100: As seen in 1, ManifoldGD outperforms the existing training free data distillation methods for all IPCs. Even, compared to the training-based methods, we see comparable (in some cases even superior) performance demonstrating the effectiveness of the course correction of \(g^t_{\text{manifold}}\). ImageWoof: Similar to ImageNette and ImageNet-100, 2 shows that ManifoldGD performs better than existing techniques. The common expected trend of increase in \(\mathrm{Acc}_{\mathcal{S}\to\mathcal{D}}\) with increase in number of samples (IPC) is seen across all datasets for all classifiers (1 and 2).

In addition to \(\mathrm{Acc}_{\mathcal{S}\to\mathcal{D}}\), we evaluate FID across DiT [38], MGD [20], and our ManifoldGD on ImageNette and ImageNet-100 with IPC = 10, 20, and 50. As shown in Figure 5, DiT yields the highest (worst) FID, MGD improves both FID and accuracy, while ManifoldGD achieves the lowest FID and highest \(\mathrm{Acc}_{\mathcal{S}\to\mathcal{D}}\) across all settings. This consistent trend supports the link between generative fidelity and downstream generalization: higher FID indicates synthetic shortcuts [57] that hinder real-data performance, while manifold-consistent guidance preserves semantic alignment and improves generalization. Moreover, ManifoldGD exhibits superior representativeness and diversity (5), confirming that hierarchical divisive clustering yields a compact yet expressive coreset that balances coarse semantic coverage with fine intra-class variability.

4 reports the \(\ell_2\) and \(\mathrm{MMD}\) distances between the synthetic and real distributions, serving as complementary measures of distribution-level fidelity between the \(\mathcal{S}\) (train set) and \(\mathcal{D_{\text{test}}}\) [19], [53], [54]. ManifoldGD consistently attains the lower \(\ell_2\) and \(\mathrm{MMD}\) values, indicating that its generated samples are both closer to individual real samples and better aligned with the overall data distribution.

We also see in Supplementary Section 8 Table 8 that ManifoldGD outperforms MGD [20] while using DDIM scheduler as well, showing effectiveness irrespective of the scheduler configuration.

4.2 Ablation Studies↩︎

4.2.0.1 Impact of Divisive-levelwise clustering.

We analyze the proposed divisive level-wise IPC selection for capturing global and local semantic variations in VAE latent space. 3 shows our strategy consistently outperforms \(k\)-means and other hierarchical clustering baselines, while 6 illustrates that our level-wise traversal spans the feature cloud more uniformly versus \(k\)-means’ limited sub-regions. Agglomerative clustering performs worst, placing centroids near outer boundaries (Supplementary 8). Optimal \(s_{\text{start}}\) depends on class granularity: deeper splits for well-separated classes, shallower splits for overlapping classes to avoid noisy leaf regions. Takeaway: Hierarchical clustering outperforms \(k\)-means via coarse-to-fine hierarchy, with top-down divisive selection being more appropriate than bottom-up agglomerative approaches. Level-wise centroid selection enhances coverage, with \(s_{\text{start}}\) adapting to dataset class separation.

4.2.0.2 Impact of \(g^t_{\text{manifold}}\).

3 further highlights the effectiveness of the proposed manifold guidance (\(g^t_{\text{manifold}}\)), which refines the coarse semantic attraction of (\(g^t_{\text{mode}}\)) by re-projecting samples back onto the estimated tangent manifold. When combined with divisive-levelwise clustering, this correction consistently yields improved performance, validating that manifold-consistent attraction effectively complements hierarchical IPC centroid selection. Furthermore, 4 analyzes the influence of different radius decay schedules for the local neighborhood (\(\mathcal{N}_s\)) used in tangent estimation. We find that an exponential annealing of the neighborhood radius achieves the best performance. This faster decay emphasizes progressively finer local structures of the manifold as diffusion proceeds, aligning with the intuition that higher-noise timesteps should rely on broader geometric context, while later steps should adapt to tighter, more locally linear approximations of (\(\mathcal{M}_t\)). Takeaway: The introduction of \(g^t_{\text{manifold}}\) further enhances the performance achieved by applying hierarchical clustering. Annealing the radius of \(\mathcal{N}_s\) operationalizes the trade-off in 1: larger initial radii accommodate necessary exploration when manifold estimates are uncertain, while progressive tightening enforces geometric constraints as the manifold becomes better defined, preventing over smoothing while maintaining consistency.

4.2.0.3 Ablation on varying \(T_{STOP}\).

8 illustrates the effect of varying \(T_{\text{STOP}}\) (the timestep at which guidance stops) on both visual fidelity and downstream accuracy. Increasing \(T_{\text{STOP}}\) up to 25 (out of 50 total denoising steps) improves FID and classification performance, suggesting that extending guidance through the early-to-mid denoising stages enables sufficient manifold correction before entering the low-noise regime. However, beyond this point, both metrics deteriorate, implying that excessive guidance interferes with the natural score-based refinement, constraining the generation to overly restricted regions of the diffusion manifold. Takeaway: FID and \(\mathrm{Acc}_{\mathcal{S}\to\mathcal{D}}\) increase steadily till \(T_{STOP}\) = 25 and decreases rapidly beyond that point. Since most structural formation occurs in early timesteps, guidance during later, fine-detail stages leads to overfitting and hinder natural denoising dynamics [26].

4.2.0.4 Ablation on varying \(t\).

We investigate the interaction between denoising steps \(t\) and \(T_{\text{STOP}}\) in 7. For smaller \(t\), \(g^t_{\text{mode}}\) dominates as \(x_t\) lies far from the data manifold, requiring strong semantic alignment when manifold structure is weakly defined. As \(t\) increases and samples approach high-density regions, \(g^t_{\text{manifold}}\) becomes critical for refining details and preventing off-manifold drift. Takeaway: Better \(\mathrm{Acc}{\mathcal{S}\to\mathcal{D}}\) favors \(g^t_{\text{mode}}\) at lower \(t\) and \(g^t_{\text{manifold}}\) at higher \(t\), directly addressing 1’s balance: early mode guidance enables exploration, while later manifold correction ensures consistency without oversmoothing.

4.2.0.5 Ablation on varying \(\phi\).

The results in 5 validate that \(g^t_{\text{manifold}}\) is kernel-agnostic (see 3.2). While RBF kernel (corresponds to MGD [20]) provides a smooth isotropic attraction toward \(c_s\), both Laplace (with sharper, local gradients) and IMQ (with heavier tails) exhibit comparable improvements when combined with our manifold correction term \(g^t_{\text{manifold}}\). Takeaway: Consistent gain across ConvNet-6, ResNet-18, and ResNetAP-10 reinforces that \(g^t_{\text{manifold}}\) provides a robust geometric correction independent of classifier capacity and is kernel-agnostic.

4.2.0.6 Qualitative Analysis.

3 illustrates that ManifoldGD produces sharper and more stable generations than MGD [20] across denoising steps. At smaller \(t\), the manifold correction constrains updates along the local tangent of the diffusion manifold \(\mathcal{M}_t\), maintaining geometric consistency while preserving class-level alignment through mode guidance. This yields cleaner, high-contrast details (e.g., facial textures, structural edges) that persist throughout sampling. In contrast, MGD [20] shows visible texture blurring and geometric drift at later steps, indicating off-manifold deviations caused by unconstrained Euclidean updates. Similar trends appear in 2 where ManifoldGD achieves higher quality and semantically sound images resembling the original samples more, whereas MGD [20] and DiT [38] fail.

5 Conclusion↩︎

We introduce Manifold-Guided Diffusion (ManifoldGD), a geometry-aware dataset distillation framework that unifies semantic mode attraction with manifold-consistent correction. By decomposing the conditional score into mode-aligned and geometry-constrained components, ManifoldGD ensures synthetic samples remain faithful to both class semantics and intrinsic data geometry. Leveraging tangent-normal decomposition on diffusion manifolds derived from hierarchical IPC centroid clustering, our approach preserves local structural validity while maintaining global diversity. Empirical results across multiple datasets, backbones, and kernel variants show consistent gains in classification accuracy, FID, and perceptual fidelity over prior methods, demonstrating that balancing semantic alignment and geometric consistency yields more representative distilled datasets. Limitations and future work. At high noise levels, diffusion corrupts local neighborhoods, biasing tangent estimates and weakening manifold reconstruction. Similarly, low-rank approximations struggle with highly curved manifolds, risking over smoothing and limited diversity (1). We mitigate these via (i) adaptive \((r, K_t)\), (ii) ridge-regularized covariance, and (iii) annealing \(\lambda_{\mathrm{man}}\) (6) to balance consistency and diversity. While empirically stable (Sec. 4.2), formal analysis of projection error and curvature sensitivity remains future work.

Acknowledgment Prof. Lokhande acknowledges support from University at Buffalo startup funds, an Adobe Research Gift, an NVIDIA Academic Grant, and the National Center for Advancing Translational Sciences of the NIH (award UM1TR005296 to the University at Buffalo). Dr.Chakraborty performed this work under the auspices of the U.S.Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

6 Algorithmic explanation of ManifoldGD↩︎

Given an IPC centroid \(c_s\) and its static latent neighborhood \(\mathcal{N}_s\) (A detailed algorithmic explanation of the formation of \(\mathcal{N}_s\) can be seen in 9.), the time-aligned manifold patch at diffusion timestep \(t\) is obtained by forward-diffusing every \(z \in \mathcal{N}_s\) using the DDPM noise schedule \(\varepsilon_t^{(k)} \sim \mathcal{N}(0,(1-\bar\alpha_t)I)\): \[\mathcal{M}_t^{(s)} = \Big\{\, x_t^{(k)} = \sqrt{\bar\alpha_t}\, z_k + \varepsilon_t^{(k)} \;\Big|\; z_k \in \mathcal{N}_s \Big\}\] Here the centroid \(c_s\) is fixed per synthetic sample: every reverse trajectory is conditioned on the specific \(c_s\) assigned at initialization.

At each timestep we compute local geometry relative to the current latent \(x_t\) by taking its \(K_t\) nearest neighbors (\(K_t=300\) empirically works the best) inside its own time-aligned patch \(\mathcal{M}_t^{(s)}\): \[\mathcal{N}_t = \text{KNN}(x_t;\,\mathcal{M}_t^{(s)}, K_t).\] This produces a time-dependent covariance \[C_t = \frac{1}{|\mathcal{N}_t|}\sum_{x\in\mathcal{N}_t} (x-\bar{x})(x-\bar{x})^\top,\] whose top-\(d\) (\(d=3\) empirically works the best) eigenvectors define the tangent and normal projectors, \[P_{\mathcal{T}_t} = U_{1:d}U_{1:d}^\top, \qquad P_{\mathcal{N}_t} = I - P_{\mathcal{T}_t}.\] The manifold-corrected guidance is obtained by canceling the normal component of the mode-guidance vector, \[g^{t}_{\mathrm{manifold}}(x_t;c_s) = -\,P_{\mathcal{N}_t}\,g^t_{\mathrm{mode}}(x_t;c_s),\] thereby restricting conditional attraction to the tangent directions of the evolving class manifold and preventing off-manifold drift.

7 Baselines used for comparison↩︎

We categorize existing dataset distillation methods into three primary groups: i) traditional coreset selection, ii) training-based diffusion methods, and iii) training-free diffusion methods. This categorization provides a structured landscape against which we position ManifoldGD. i)Coreset selection: Coreset selection techniques, such as Herding [34] and K-Center [35], aim to identify a representative subset of the original data that captures the underlying data distribution. While efficient, these methods often struggle to capture the full complexity of the data distribution and exhibit limited cross-architecture generalization. ii)Training-based diffusion methods: Training-based methods fine-tune or optimize within the generative model to produce synthetic datasets. DM [32] utilizes a latent diffusion model to distill datasets, showing strong generalization. Min-Max Diffusion [23] employs a min-max objective to enforce diversity and representativeness, though it requires generator optimization. Methods like D\(^{4}\)M [24] and Zou et al. [25] also fall into this category, achieving high performance but incurring the computational cost associated with model training or fine-tuning. IDC-1 [36] and GLAD [37] are also training-based methods that optimize synthetic images through gradient matching and generative modeling respectively. iii)Training-free diffusion methods: Random [56] simply selects random real images, providing a lower-bound performance benchmark. Latent Diffusion Models (LDM) [39] and DiT [38] can be directly sampled from, but without guidance, the resulting datasets may lack semantic focus. MGD [20] proposes a mode-guided diffusion pipeline that identifies class prototypes via clustering and guides the sampling process towards them, significantly improving upon unguided sampling.

Several concurrent works have explored enhanced guidance strategies for diffusion-based dataset distillation. [40] employs an information-theoretic framework that requires a separately trained classifier to guide the sampling process. [26] utilizes a pre-trained classifier to compute influence functions for guiding the generation trajectory. Similarly, other methods leverage text encoders for enhanced semantic conditioning [25], additional classifiers for OOD detection [41], or other external guidance mechanisms. Unlike these approaches that rely on extra components beyond the diffusion model itself, our ManifoldGD framework operates with only a single pre-trained diffusion model and a VAE for feature extraction. To maintain a fair comparison, we do not directly compete with these more complex architectures in our main experiments. Nevertheless, as seen in Section 4.1, our method achieves comparable performance to the results reported in these papers.

8 Ablation Experiments↩︎

Agglomerative vs. divisive clustering for IPC centroid selection. 10 highlights a key structural difference between agglomerative clustering and divisive (bisecting) clustering when operating in the VAE latent space. Agglomerative clustering begins with individual samples and progressively merges them ("bottom up" approach), causing early merges to be dominated by peripheral or low-density points. As a result, the resulting IPC centroids tend to lie near the boundary of the feature cloud, often reflecting outlier- or noise-driven directions. In contrast, divisive clustering recursively splits clusters, producing progressively purer, high-density partitions. This top-down refinement naturally places IPC centroids closer to the mean or high-density core of the feature manifold, yielding more stable and semantically representative prototypes. For IPC=10 on ImageNette, this difference is visually evident: divisive-levelwise centroids align with class structure, whereas agglomerative ones drift toward noisy edges. Takeaway: Divisive clustering provides centrally anchored, density-aligned IPC centroids, while agglomerative clustering often produces edge-biased, noisier centroids.

Level analysis for Divisive-levelwise clustering. As shown in 11, ImageNette selects a higher \(s_{\text{start}}\) compared to ImageNet-100, indicating that datasets with fewer classes benefit from starting IPC selection deeper in the hierarchy. With fewer classes, the VAE latent space is less congested, and class clusters occupy more separated, fine-grained regions, thus making leaf-level centroids reliable representatives. In contrast, ImageNet-100 exhibits substantial feature overlap due to its larger number of classes, causing leaf nodes to contain noisy, outlier-like samples that blur class boundaries. Starting closer to the root therefore avoids oversampling these unstable leaf clusters and yields more semantically coherent centroids. This validates the design of divisive-levelwise IPC selection, i.e., controlling \(s_{\text{start}}\) naturally adapts the granularity of the centroid set to the intrinsic structure of each dataset’s feature manifold. Takeaway: The behavior of \(s_{\text{start}}\) serves as a diagnostic for class-specific feature overlap. i) Low-overlap datasets (e.g., ImageNette): deeper, fine-level picks are effective. ii) High-overlap datasets (e.g., ImageNet-100): shallower levels preserve separability and avoid noise-dominated leaves.

Effect of Neighborhood Radius \(r\) in Local Manifold Estimation. In our method, each IPC centroid \(c_s\) defines a local latent neighborhood \(\mathcal{N}_s = \{ z \in Z : \|z - c_s\|_2 \le r \}\), where \(r\) is a radius in the VAE latent space. Intuitively, a smaller \(r\) yields a tightly localized neighborhood capturing fine-grained geometric structure, while a larger \(r\) aggregates a broader region of the latent feature cloud, mixing together points with potentially different local curvature or class-specific variations. Since the tangent space and normal projector are computed from the covariance of \(\mathcal{N}_s\), the choice of \(r\) directly affects the quality of the estimated local diffusion manifold \(\mathcal{M}_t^{(s)}\). For ImageNette (10-class subset), we observe that a small radius \(r=0.05\) performs best, while performance degrades for \(r=0.1\) and \(r=0.2\). Conversely, for ImageNet-100 (100 classes), the trend reverses: \(r=0.05\) tends to fail, and moderate radii \(r=0.1\) or \(r=0.2\) yield better performance. This pattern is expected because \(r\) interacts with dataset granularity, class overlap, and latent density. In ImageNette, classes occupy well-separated, compact regions of the VAE space, so a small radius correctly isolates coherent local geometry. In ImageNet-100, however, the latent space becomes more crowded (classes have heavier overlap and higher intrinsic variation) so \(r=0.05\) often yields too few neighbors to estimate a stable covariance, causing unreliable tangent spaces. Moderate radii provide enough samples to obtain statistically stable, noise-robust local geometry. Takeaway: \(r\) must match the density and geometry of the feature space. Small \(r\) excels when class clusters are compact and well-separated (e.g., ImageNette). Larger \(r\) becomes necessary in denser, higher-class settings (e.g., ImageNet-100) to obtain stable covariance and reliable tangent estimation. This reinforces that effective manifold guidance depends on capturing locally coherent structure, i.e., not too narrow (noisy), not too broad (geometrically inconsistent).

Effect of Tangent space dimension \(d\) in Local Manifold Estimation. Tab. 9 analyzes the effect of tangent subspace dimension d. We observe a consistent peak at d=3. \(d<3\) under-represents the local manifold, restricting valid tangent directions. \(d>3\) introduces noisy/off-manifold directions. Takeaway: Performance is stable across a reasonable range of d, indicating low sensitivity to this hyperparameter. This is expected since manifold correction is local and only requires principal variation directions (not full geometric reconstruction).

Comparison using different schedulers. From 8 we can see that ManifoldGD outperforms MGD [20] while using a DDIM noise scheduler as well. It was already seen in 1 2 that ManifoldGD achieves better performance with DDPM scheduler as well. Takeaway: Superior performance of ManifoldGD over training-free methods is scheduler agnostic.

Time complexity/comparison. We measure inference time for training-free methods when generating one ImageNette sample (IPC=10) on an NVIDIA A6000 (Fig. 12) ManifoldGD adds modest overhead compared to pure inference-time guidance, yet remains fully training-free, which would otherwise have incurred substantially higher compute and memory costs. In practice, ManifoldGD trades a small increase in sampling time (-\(\Delta0.15s\) vs. MGD [34], -\(\Delta5.2s\) vs. DiT [28]) for improved data quality and downstream performance (+\(\Delta2.2\) vs. MGD [34], +\(\Delta5.0\) vs. DiT [28]).

Analysis of denoising trajectories. From 8. We visualize the per-step denoising trajectories by tracking the Euclidean norm of the latent prediction \(||\hat{x}||\) (the VAE-latent decoded estimate returned by the diffusion step) for multiple IPC centroid-conditioned samples (10 trajectories for IPC=10) and computing the variance of those norms across samples at each timestep. Concretely, for each conditioned sample we record \(||\hat{x}(t)||\) at every denoising step, then plot the sample-wise trajectories (left) and the timestep-wise variance of those norms (right). The plotted final variance (v) refers to the variance of \(||\hat{x}(50)||\) at the end of the reverse process (t=50). 13 shows these trajectories for DiT, MGD, and ManifoldGD on ImageNette (IPC=10). ManifoldGD consistently produces a larger final trajectory variance than both MGD [20] and DiT [38]. This is an expression of meaningful intra-mode variability preserved by our manifold correction. Intuitively, MGD’s [20] plain Euclidean mode-pull strongly concentrates samples toward the centroid direction and therefore reduces final variability (mode-collapse within a prototype). DiT [38] being unguided often yields lower variance as well because it samples broadly from the prior but lacks targeted, class-anchored diversity. By contrast, ManifoldGD combines semantic attraction (mode guidance) with an explicit cancellation of the off-manifold normal component, i.e., we remove the normal projection and keep tangent-aligned variations. This lets samples remain close to class modes while preserving the valid degrees of freedom along the manifold (higher tangent-aligned spread), which increases the measured final variance. Takeaway: ManifoldGD’s larger final trajectory variance reflects useful, manifold-consistent intra-class diversity rather than noise. By eliminating the off-manifold normal component while retaining tangent-aligned variations, ManifoldGD avoids the two failure modes of plain mode-pulling (i) collapse onto a tight prototype and (ii) drift off the data manifold, so samples remain class-relevant yet diverse. Because FID and downstream accuracy also improve, the increased variance serves as a compact diagnostic of the desired balance between representativeness and geometric fidelity.

Class-wise analysis. 6 presents a detailed class-wise comparison between MGD and ManifoldGD on ImageWoof for IPC = 10 and IPC = 50 across three architectures. ImageWoof is intentionally challenging for dataset distillation because all classes are dog breeds with highly overlapping visual structure (textures, fur patterns, pose similarity), making the latent space significantly more homogeneous than ImageNette or ImageNet-100. In such settings, mode-collapse and insufficient intra-class variation are more likely, and class separation depends heavily on preserving subtle geometric cues. This makes ImageWoof an especially sensitive benchmark for evaluating whether a distillation method can retain fine-grained, within-class diversity. Across IPC = 10, ManifoldGD improves over MGD on a majority of classes and architectures. For example, on ResNet-AP, ManifoldGD achieves notable gains on Beagle (+3.1), Shih-Tzu (+10.5), Ridgeback (+7.2), and Golden Retriever (+11.2), showing that manifold-aligned variation helps recover class-specific fine-grained details that Euclidean mode-pulling suppresses. Similar trends appear for ConvNet (Beagle +12.9, Samoyed +4.8) and ResNet-18 (Samoyed +4.5, Beagle +6.7, Ridgeback +9.8). These improvements emerge specifically in classes with subtle visual boundaries, where preserving tangent directions of the manifold is more critical than strict centroid attraction. At IPC = 50, both methods improve (as expected with increased supervision) but ManifoldGD continues to outperform MGD on most architectures. In particular, for ResNet-18, large gains appear for Beagle (+13.9), Shih-Tzu (+3.2), Ridgeback (+5.9), and Golden Retriever (+4.1). ConvNet and ResNet-AP show similar improvements, especially on texture-heavy breeds such as Shih-Tzu, Ridgeback, and Golden Retriever. These gains indicate that even with a larger IPC budget, enforcing manifold-consistent guidance preserves meaningful degrees of freedom within each breed’s local geometry, preventing over-regularization and enabling better fine-grained discrimination. Takeaway: ManifoldGD is especially effective on fine-grained, homogeneous datasets like ImageWoof, where class distinctions rely on subtle manifold-level variations rather than coarse semantic separation. The consistent improvements across architectures and IPC budgets demonstrate that combining semantic attraction with manifold-consistent tangent preservation yields more discriminative and diverse synthetic data, ultimately improving downstream classification accuracy.

9 Qualitative Analysis↩︎

14 15 shows a qualitative comparison between MGD [20], DiT [38] and ManifoldGD. GPT-4.0 was used as an unbiased evaluator, prompted to assess conceptual clarity, visual sharpness, and background separation without revealing method identities. The responses indicate that ManifoldGD generates sharper, more coherent images where the concept “fish” and “monkey” are clearly distinguished from the background, exhibiting better texture and lighting consistency. In contrast, MGD [20] samples display moderate sharpness and weaker subject–background separation. This supports that \(g_t^{\mathrm{man}}\) promotes denoising along intrinsic manifold directions, yielding semantically faithful yet geometrically consistent generations. Furthermore, ManifoldGD achieves variations (See 15) without compromising image quality which is lacking for DiT [38] and MGD [20].

16 17 compare DiT [38], MGD [20], and ManifoldGD samples on ImageNette and ImageNet-100. For ImageNette, the visual gap between methods is more pronounced: MGD often exhibits over-smoothed textures or centroid-induced collapse, while DiT occasionally drifts off-class or produces weaker semantic detail. ManifoldGD, by contrast, consistently produces sharper boundaries, richer textures, and more stable object geometry reflecting the benefit of tangent-aligned guidance and the removal of off-manifold drift. For ImageNet-100, all methods generate reasonably strong images because the underlying data distribution is more visually diverse and less constrained, making minor errors less perceptually salient. Nevertheless, ManifoldGD still improves FID (see 5) and yields slightly cleaner, more coherent samples, even though the perceptual gap appears narrower. Overall, the qualitative results mirror our quantitative trends: on simpler, low-entropy datasets (ImageNette), manifold-corrected guidance yields visibly stronger structure and detail, while on high-entropy datasets (ImageNet-100), improvements remain measurable but more subtle.

10 Results on ImageNet-1k↩︎

We additionally benchmark DiT [38], MGD [20], and ManifoldGD on the full ImageNet-1k dataset under the most challenging hard-label evaluation protocol for IPC = 1 and 50. As shown in 10, ManifoldGD consistently outperforms both baselines across both IPC regimes. These gains reflect the effectiveness of our manifold-aligned guidance in preserving class-consistent structure even when only a single distilled exemplar per class is available.

Furthermore, ImageNet-1k represents a significantly more challenging testbed compared to ImageNette or ImageNet-100. The dataset contains 1000 classes, many of which are fine-grained, visually similar, and highly imbalanced in the VAE latent space. This dramatically increases mode density and inter-class overlap, making it difficult for training-free methods to avoid collapsing onto overly generic prototypes. Methods relying solely on Euclidean mode attraction (e.g., MGD [20]) tend to over-concentrate samples around coarse class directions, while unguided diffusion sampling (DiT [38]) lacks the semantic anchoring required to generate discriminative exemplars. In contrast, ManifoldGD’s tangent-aligned guidance preserves intra-class variability while suppressing off-manifold drift, leading to more faithful and well-separated prototypes across a large number of classes.

Overall, these results demonstrate that ManifoldGD is not only effective in small-scale settings but also scales robustly to the full ImageNet-1k challenge, maintaining strong sample fidelity and discriminative power even at extremely low IPC budgets.

References↩︎