No title


Abstract

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.

Jinseong Park Mijung Park
jinseong@kias.re.kr mijung.park@ubc.ca
Korea Institute for Advanced Study University of British Columbia

1 Introduction↩︎

Figure 1: Illustration of quality degradation of unlearned images after data unlearning. Left: two images to unlearn. Middle: Gradient Ascent (GA, first column) and GA with our approach (second column). Right: SISS [1] (first) and SISS with our approach (second). Our approach improves the quality of existing unlearning methods.

The ability to remove the influence of some training samples from a learned model, often referred to as machine unlearning [2], has become increasingly important. Regulatory frameworks such as the “right to be forgotten” in the General Data Protection Regulation (GDPR) by the European Union and growing concerns about sensitive or proprietary data have created demand for methods that allow models to forget without costly retraining from scratch. Recently, with the development of diffusion models [3], unlearning the unsafe concept or memorization has been actively explored through finetuning [4], output filtering [5], and training-free sampling [6], [7].

In machine unlearning with generative models, two major objectives exist: (a) concept or class unlearning, which refers to preventing the generation of a particular concept or class in samples [8], and (b) data unlearning, which focuses on removing individual samples rather than concepts. As noted by [1], data unlearning is less studied than concept unlearning. Furthermore, removing a single individual while preserving the generation of similar outputs is more challenging than deleting the entire mode of similar examples. Finetuning-based unlearning methods, such as gradient ascent [1] or negative preference optimization [4], rely on loss maximization objectives. The generated images often become noisier and aesthetically degraded, while forgetting might not be complete nor precise, as shown in Figure 1.

To mitigate the drawbacks of the data unlearning in diffusion models, we analyze the dynamics of data unlearning. We argue that forgetting does not occur uniformly, but rather unevenly across time and spectral domains. Diffusion models learn phase-dependent behaviour during training [9]: later diffusion time steps close to Gaussian noise capture coarse semantics, while earlier time steps close to the data refine fine-grained details. However, previous unlearning methods treat all diffusion time steps equally, potentially leading to the inadvertent forgetting of unintended details, as illustrated in Figure 2. Furthermore, we investigate the spectral domain to selectively remove unwanted high-frequency components in the forget set. Overall, our framework effectively implements forgetting and preserves sample quality by focusing on selective components in both the time and frequency domains. We summarize our contributions below:

  • We provide a systematic study of data unlearning in diffusion models in time and spectral space, considering both conditional and unconditional scenarios.

  • Our time- and frequency-selective unlearning approach is simple, compatible with existing unlearning objectives, and performs effective unlearning across various settings.

  • We devise an improved metric, SSCD\(^{\text{norm}}\). Unlike the original SSCD [10], which only measures dissimilarity, SSCD\(^{\text{norm}}\) standardizes perturbation distance for measuring quality.

2 Background↩︎

2.1 Diffusion Models↩︎

Diffusion models [3] aim to generate samples from an unknown data distribution by considering forward (diffusion) and reverse (denoising) processes. The forward process gradually injects noise into samples \({\mathbf{x}}_0\) drawn from the data distribution \(p_0\), forwarding them into a fixed standard Gaussian distribution \(p_T = \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\). The corresponding forward stochastic differential equation (SDE) is: \[d{\mathbf{x}}_t = f({\mathbf{x}}_t,t)dt + g(t)d{\mathbf{w}}_t,\] where \(f:\mathbb{R}^d \times [0,T] \rightarrow \mathbb{R}^d\) represents the drift function, \(g:[0,T] \rightarrow \mathbb{R}\) denotes the diffusion coefficient, and \({\mathbf{w}}_t\) is the Wiener process. As diffusion models learn to reverse the above process, the reverse SDE [11] is formulated as follows: \[d{\mathbf{x}}_t = [f({\mathbf{x}}_t,t) - g(t)^2\nabla_{{\mathbf{x}}_t} \log p({\mathbf{x}}_t)]dt + g(t)d\bar{{\mathbf{w}}}_t,\] where \(p({\mathbf{x}}_t)\) refers to the probability density of \({\mathbf{x}}_t\). Diffusion models learn to match the score function \(\nabla_{{\mathbf{x}}_t} \log p({\mathbf{x}}_t)\) (parameterized by a score network \(s_\theta({\mathbf{x}}_t,t)\)) for denoising [12]. With standard DDPM forward noising process \(q({\mathbf{x}}_t|{\mathbf{x}}_0) = \mathcal{N}\left(\sqrt{\bar\alpha_t}{\mathbf{x}}_0,(1-\bar\alpha_t)\mathbf{I}\right)\), the conditional score function can be written as \(\nabla_{{\mathbf{x}}_t}\log q({\mathbf{x}}_t|{\mathbf{x}}_0)\). With a weighting factor \(w_t\), the training objective \(\mathcal{L}_\mathcal{D}(\theta)\) is \[\mathbb{E}_{{\mathbf{x}}_0\sim p_0}\mathbb{E}_{t\in[0,T]}\Bigl[ w_t\bigl\| s_\theta({\mathbf{x}}_t,t) - \nabla_{{\mathbf{x}}_t}\log q({\mathbf{x}}_t|{\mathbf{x}}_0) \bigr\|_2^2 \Bigr]. \label{eq:diffusion95loss}\tag{1}\]

2.2 Data Unlearning in Diffusion Models↩︎

2.2.0.1 Machine Unlearning.

Suppose a model \(\theta^{*}\) is already trained on a dataset \(\mathcal{D}\). Then, our goal is to delete the influence of the forget set \(\mathcal{D}_F\) from \(\theta^{*}\), while maintaining the model utility on the retain set \(\mathcal{D}_R=\mathcal{D} \setminus \mathcal{D}_F\). Specifically, our goal is to fine-tune the trained model \(\theta^{*}\) using the forget dataset \(\mathcal{D}_F\), often using the retain dataset \(\mathcal{D}_R\) to mitigate quality degradation [2].

2.2.0.2 Diffusion Unlearning.

Unlearning in diffusion models can be categorized as concept unlearning and data unlearning. Concept unlearning [13] refers to prohibiting a diffusion model from producing images categorized in a particular type of high-level concept, including not safe for work (NSFW) material (nude and violent images, for example), or copyrighted content [13][15].

Due to the high flexibility of diffusion models in sampling, unlike GANs [16], training-free steering methods have been actively investigated [6], [17], [18]. These methods leverage guidance techniques to repel the generation from specific points or embeddings, achieving concept erasure without finetuning.

Data unlearning, on the other hand, is closely related to individual data memorization and aims to remove specific data examples in accordance with the “Right to be Forgotten" by the aforementioned GDPR. For example, if a user requests the deletion of a particular face image used to train a diffusion model, the goal is to eliminate the influence of that image from the trained model. In contrast to concept erasing, for which various practical solutions have been developed, data unlearning remains relatively underexplored [1]. In general, the objective of diffusion unlearning \(\min_\theta \mathcal{L}_{\text{UL}}(\theta)\) is \[\begin{cases} \max_\theta \mathcal{L}_{\mathcal{D_F}}(\theta) = \min_\theta -\mathcal{L}_{\mathcal{D_F}}(\theta), & \text{\mathcal{D}_F only} \\ \min_\theta \bigl( - \mathcal{L}_{\mathcal{D_F}}(\theta)+ \mathcal{L}_{\mathcal{D_R}}(\theta) \bigr), & \text{\mathcal{D}_F and \mathcal{D}_R. } \end{cases} \label{eq:unlearn}\tag{2}\]

Gradient ascent (GA) or negative gradient on forget data samples \(\mathcal{D}_F\) is a key component for data unlearning [19]. For GA, the loss function on the forget set \(\mathcal{L}_{\text{GA}}(\theta)\) is formulated as follows: \[-\underbrace{\mathbb{E}_{ {\mathbf{x}}_0 \sim \mathcal{D}_F}\mathbb{E}_{t\in[0,T]} \Bigl[ w_t\bigl\| s_\theta( {\mathbf{x}}_t,t) - \nabla_{ {\mathbf{x}}_t}\log q( {\mathbf{x}}_t| {\mathbf{x}}_0) \bigr\|_2^2 \Bigr]}_{\text{Negative loss in Equation~(\ref{eq:diffusion95loss}) on forget set }\mathcal{D}_F}. \label{eq:ga}\tag{3}\] EraseDiff [20] replaces the true score of the forget samples with a randomly sampled noisy image. SISS [1] investigated the unlearning as a mixture distribution of forget and retain data with importance sampling. In conclusion, previous methods investigate how to formulate Equation (2 ) well.

Figure 2: Illustration of unlearning a specific stage of time steps. See Section 3.1.1 for details.

2.3 Evaluation of Data Unlearning in Diffusion Models↩︎

We follow the evaluation of [1] based on unconditional or conditional scenarios.

Memorization on Unconditional Reconstruction. In unconditional generation, memorization is assessed by the diffusion model’s ability to reconstruct the training data. In particular, we noise a clean image \({\mathbf{x}}_0\) via the DDPM forward process to an intermediate time step \({\mathbf{x}}_t\) (e.g., \(t=250\)) where the model can easily reconstruct, and measure the similarity between the original \({\mathbf{x}}_0\) and the reconstructed \(\hat{\mathbf{x}}_0\) using model \(\theta\) from \({\mathbf{x}}_t\).

Memorization on Conditional Guidance. In conditional generation, conditioning on text or class embeddings can lead to stronger overfitting to the conditional distribution over the images given those conditions [21]. Therefore, measuring the effects of condition \(\epsilon_\theta({\mathbf{x}}_t, e_p)\) or its guidance norm \(\|\epsilon_\theta({\mathbf{x}}_t, e_p) - \epsilon_\theta({\mathbf{x}}_t, e_\emptyset)\|^2_2\) is crucial to track conditional memorization [1], [22], [23], with conditional embedding \(e_p\) and null embedding \(e_\emptyset\). Thus, unlike unconditional memorization, its evaluation starts by sampling from pure noise \(t=T=1000\) with the conditions.

3 Time and Frequency Selection for Data Unlearning in Diffusion Models↩︎

In this section, we first empirically investigate the effects of time steps and frequency on diffusion unlearning and propose a selective framework for data unlearning, followed by a theoretical analysis.

3.1 Not Every Time Step Needs to Be Forgotten↩︎

3.1.1 Motivation↩︎

Several theoretical papers analyze the dynamics of diffusion models [24][27]. Starting from the Gaussian noise, diffusion models learn distinct attributes across different phases. Phase I involves learning coarse features at late time steps near time \(T\). Phase II involves generating content at intermediate steps, and Phase III involves refining details toward convergence at early time steps near \(t=0\) [9]. [25] describes the diffusion dynamics in terms of these three distinct phases divided by two important transition points in time: (a) speciation time \(t_S\); and (b) collapse time \(t_C\). The denoising trajectories begin to separate by semantic class at the speciation time \(t_S\) (phase I-II), for example via embedding similarity or consistency scores. For \(t > t_S\), trajectories are nearly class-agnostic noise, whereas for \(t < t_S\), they encode semantic identity. Afterwards, trajectories converge toward individual training samples at the collapse time \(t_C\) (phase II-III), which quantifies instance-level attraction, such as nearest-neighbor consistency. For \(t < t_C\), diffusion paths are drawn to a single point and lose their generalization capability, indicating memorization. Empirically, as shown in Figure 31, [29] observed that the likelihood that the noisy data \({\mathbf{x}}_t\) is classified into a certain class in speciation time \(t_S\). Also, [28] observed that data converges into a stable point, indicating collapse time \(t_C\).

Figure 3: Empirical evidence on transition points of speciation time t_S and collapse time t_C.

To investigate data unlearning in the aforementioned view, we will exploit the use of the transition points in data unlearning. We conduct a toy experiment with a shallow diffusion model on a half-moon dataset, as visualized in Figure 2. We only change the unlearning time steps while using the same data samples, with the loss function of Equation (3 ). Deleting only the early time steps (phase III) does not help forget data samples; instead, it leads to quality degradation in real samples. Unlearning the middle time steps (phase II) is most aligned with the instance-level attraction before reaching the collapse time \(t_C\). For stronger forgetting of semantic conditioning, targeting the late time steps (phase I) preceding speciation at \(t_S\) is effective.

3.1.2 Time-selective Diffusion Forgetting↩︎

Based on the previous analysis, we suggest a simple framework of selective time steps as \[\mathbb{E}_{{\mathbf{x}}_0\sim p_0} \underbrace{\mathbb{E}_{t \sim P(t)}}_{\text{Time selection}} \Bigl[ w_t \| s_\theta({\mathbf{x}}_t,t) - \nabla_{{\mathbf{x}}_t} \log q({{\mathbf{x}}_t}| {\mathbf{x}}_0) \|^{2} \Bigr]. \label{eq:ours95time}\tag{4}\] Here, \(P(t)\) is a time step weighting. In detail, we utilize the time scales that give a higher probability to a specific interval \(t_1 \leq t \leq t_2\) for the forget data samples as follows: \[P(t) = \begin{cases} {1-k}{/(t_2 - t_1)}, & t_1 \le t \le t_2, \\ {k}/({T - (t_2 - t_1))}, & \text{otherwise}, \end{cases} \label{eq:time}\tag{5}\] where \(0 \le k \le 1\) is the suppression intensity and \(t_1,t_2 \in (0,T]\) with \(t_1 < t_2\) and we use \(k=0\) for simplicity. As explained in Section 2.3, the evaluation scenarios for unconditional and conditional settings differ significantly. Therefore, we investigate appropriate time steps \(P(t)\) in Equation (5 ).

3.1.2.1 Unconditional Unlearning.

The gradient norm \(\|\nabla_\theta\mathcal{L}({\mathbf{x}};\theta)\|_2^2\) reflects the magnitude of the update required to forget a sample \({\mathbf{x}}\) [30], [31], allowing us to identify which stages are most affected by the unlearning. In Figure [fig:distribution95time95b], to compare the actual changes after standard unlearning with uniform time steps in the image-level unlearning with CelebA-HQ, we unlearn a pre-trained model with SISS. The gradient norm increases significantly after unlearning, with the gap being most pronounced in the middle time steps (Phase II). The gradient gap indicates that unlearning induces a substantial shift in the model’s parameter space during Phase II.

Furthermore, as shown in Figure [fig:distribution95time95a], when we calculate the similarity on DINOv3 [32] of the noisy data \({\mathbf{x}}_t\) between other clean data samples, the similarity towards the original data \({\mathbf{x}}_0\) is drastically increased in the refinement stage, compared to other nearest samples. Thus, to prevent the diffusion trajectory from heading toward memorization, we need to control the preceding memorization at \(t_C\). Note that the evaluation scheme also compares the difference between the original and reconstructed images from intermediate time steps, such as \(t=250, 500\).

3.1.2.2 Conditional Unlearning.

In conditional memorization, the effect of conditional guidance, such as class or text embeddings, acts more strongly than its unconditional counterpart [21]. Therefore, calculating the guidance magnitude \(\|\epsilon_\theta({\mathbf{x}}_t, e_p) - \epsilon_\theta({\mathbf{x}}_t, e_\emptyset)\|_2^2\) is key for conditional memorization [22], [33]. In Figure [fig:adaptive95a], we measured the trajectory of conditional guidance with memorized prompts in Stable Diffusion v1.4 [34], with detailed settings provided in Section 4.1. We can observe a peak in guidance norms in phase I before speciation begins [23], [33].

Based on the experiments, we can hypothesize that controlling the diffusion trajectory for \(t>t_S\) before specification begins is important for the conditional case. Similarly, training-free methods observed that guiding only late time steps \(t>t_S\) is sufficient to steer the generation away from harmful concepts. For example, [6] applied guidance within \(t>780\), while [35] noted that their repellency terms are significant in late time steps.

In summary, \(t_C<t<t_S\) (phase II) is critical for controlling unconditional behavior, while \(t>t_S\) (phase I) matters for conditional generation, where conditional guidance dominates the diffusion dynamics after speciation, and the trajectory should be regulated before each critical time step begins.

Figure 4: Adaptive conditional memorization.
Table 1: Summary of diffusion transition points \(t_S\) and \(t_C\) (\(T=1000\)) across existing studies. \(^\dagger\) denotes intervals.
Reference Basis of Analysis \(\mathbf{t_S}\) (phase I-II) \(\mathbf{t_C}\) (phase II-III)
[25]\(^\dagger\) Spectral / Entropy \([500,800]\) \([100,250]\)
[9] SNR (\(10^{-2}, 10^0\)) \(675\) \(259\)
[36] Structure Pruning \(750\) \(250\)
[6] Conditional Sampling \(780\)
[23]\(^\dagger\) Conditional Sampling \([600,800]\)

r0.55

3.1.2.3 Optimal Selection of Transition Time \(t_S\) and \(t_C\).

Finding the optimal \(t_S,t_C\) is an open question in diffusion models. We summarize previous researches in Table 1, where \(t_S\) and \(t_C\) are approximately similar to the transition points \((t_C,t_S)=(250,750)\) used in Figure 2.

Since the guideline in Table 1 is not sample-wise, we further consider an adaptive method on conditional unlearning based on [23], analogue to the speciation time \(t_S\): (i) the guidance magnitude \(\|\epsilon_\theta({\mathbf{x}}_t, e_p) - \epsilon_\theta({\mathbf{x}}_t, e_\emptyset)\|_2^2\) is computed, and (ii) set the transition point \(\tau\) as the first local minimum observed in sampling, detailed in Algorithm 9 (Appendix 7). As the \(\tau\) searching method is prompt adaptive, we can consider the transition \(\tau\) as \(t_S\). As shown in Figure [fig:adaptive95b], although the optimal \(\tau\) varies across prompts, the median of \(\tau\) is \(761.1\), similar to the fixed \(t_S=750\).

3.2 Not Every Frequency Mode Needs to Be Forgotten↩︎

3.2.1 Motivation↩︎

From a time-selective perspective, Phase I (\(t > t_S\)) and Phase II (\(t_C < t < t_S\)) represent the primary stages where semantic- and individual-level memorization are established, respectively. Maintaining time steps \(t < t_C\) (Phase III) is essential for quality preservation because this phase only refines the quality of the generated sample determined in earlier stages. This motivates the use of low-pass filters in diffusion models to avoid forgetting fine-grained and high-frequency details [37], [38].

We evaluate the impact of unlearning in the frequency domain via the power spectral density in Figure [fig:psa]. For the retain set, the high-frequency distribution is preserved post-unlearning. Conversely, the forget set exhibits unwanted high-frequency components that risk degrading sample quality.

3.2.2 Frequency-selective Diffusion Forgetting↩︎

Thus, to preserve fine-grained image quality and make less perturbation in the manifold during unlearning, we add a low-pass filter to remove the high-frequency components of the forget set as \[\mathbb{E}_{\mathbf{x}_0\sim p_0}\mathbb{E}_t\Bigl[w_t\bigl\|\underbrace{\mathcal{F}\bigl(s_\theta(\mathbf{x}_t,t)-\nabla{\mathbf{x}_t}\log q(\mathbf{x}_t\mid\mathbf{x}_0)\bigr)}_{\text{Low-pass filter}}\bigr\|^2\Bigr] \label{eq:ours}\tag{6}\] Notice that the denoiser input remains the noisy natural sample \(\mathbf{x}_t\) during unlearning; only the loss residual is projected to low frequencies. Hence, the input distribution during unlearning and inference is identical, and only the gradient signal used for unlearning is projected. In practice, we employ the discrete Fast Fourier Transform (FFT) as follows: \[\mathcal{T}(u,v) = \text{FFT}({\mathbf{x}}) = \sum_{x=1}^{H}\sum_{y=1}^{W} {\mathbf{x}}(x,y) e^{-j2\pi\left(\frac{u}{H}x + \frac{v}{W}y\right)},\] where \({\mathbf{x}}(x,y)\) is the pixel intensity at position \((x,y)\), \(\mathcal{T}(u,v)\) is the complex coefficient at frequency \((u,v)\), and \(e\) and \(j\) represent Euler’s constant and the imaginary unit, respectively. To remove only the high-frequency terms, we apply a masking function to the FFT results and then reconstruct the image using the inverse FFT with radius \(r\) as \(\mathcal{F}({\mathbf{x}}_{i,t}) = \text{IFFT} \left(\text{FFT}({\mathbf{x}}_{i,t}) \odot \beta_{i,t}(r)\right),\) where \(\beta_{i,t}(r) = s \text{ if the radius } r > r_t, 1 \text{ otherwise}\). \(r_t\) is a radius cutoff frequency threshold. \(0 \leq s \leq 1\) is a high frequency weight, where we set \(s=0\) to exclude high frequency in unlearning.

3.3 Theoretical Justification↩︎

Data unlearning must remove the influence of specific samples while preserving the overall data manifold. To this end, we analyze the enhanced forgetting within the selected region and the minimized distortion in the non-selected region, showing the strength in the forgetting–preservation trade-off.

Setup. Let \(\mathbb{P}_{\mathrm{orig}}\) and \(\mathbb{P}_m\) (\(m \in \{\mathrm{full}, \mathrm{sel}\}\)) be the path measures of the original and unlearned diffusion models. We denote the score perturbation as \(\Delta s_m(x_t, t) := s_{\theta_m}(x_t, t) - s_{\theta_{\mathrm{orig}}}(x_t, t)\). By Girsanov’s theorem [39], the KL divergence between the original and unlearned path measures is \[D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) = \frac{1}{2}\int_0^T g(t)^2 \mathbb{E}_{x_t} \left[ \|\Delta s_m(x_t, t)\|_2^2 \right] dt.\] To evaluate unlearning efficacy, we decompose this total unlearning-induced distortion into a selected target window \(\mathcal{S}\) (e.g., selected time-steps or frequencies) and its complement \(\mathcal{S}^c\). Let \(D_{\mathrm{KL}}^{\mathcal{W}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) := \frac{1}{2}\int_{\mathcal{W}} g(t)^2 \mathbb{E}_{x_t} [\|\Delta s_m\|_2^2] dt\) denote the regional divergence over a window \(\mathcal{W}\). Thus, the total divergence is \(D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) = D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) + D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m).\) For \(\beta > 0\), we define the forgetting–preservation utility as: \[\label{eq:trade95off} \mathcal{J}_{\beta}(m) := D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) - \beta D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m).\tag{7}\] Note that \(D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m)\) is a proxy for the intended forgetting and, \(D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m)\) is a proxy for collateral distortion without intend to unlearn, detailed in Appendix 6.

Assumptions. (i) Finite divergence: \(D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) < \infty\). (ii) Focused updates in \(\mathcal{S}\): Due to the frequent update on the selected region under the same training budget, selective unlearning amplifies changes in the target region (\(\|\Delta s_{\mathrm{sel}}\|_2 \ge \mu \|\Delta s_{\mathrm{full}}\|_2\) for \(\mu \ge 1\) in \(\mathcal{S}\)) and weaker changes in \(\mathcal{S}^c\) (\(\|\Delta s_{\mathrm{sel}}\|_2 \le \eta \|\Delta s_{\mathrm{full}}\|_2\) for \(\eta \in [0,1]\) in \(\mathcal{S}^c\)). See Appendix 6 for empirical validation and proofs.

Proposition 1. Under the assumptions above, for every \(\beta>0\), and \(\lambda := \frac{D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}})}{D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}})} \in [0,1]\), the trade-off utility gap of \(\mathcal{J}_{\beta}(\mathrm{sel})\) and \(\mathcal{J}_{\beta}(\mathrm{full)}\) satisfies: \[\mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full}) \ge \left[ (\mu^2-1)(1-\lambda) + \beta(1-\eta^2)\lambda \right] D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}),\] where \(\lambda\) indicates the proportion of collateral distortion compared to the full-unlearning.

Remark 1. The bracketed term \((\mu^2-1)(1-\lambda) + \beta(1-\eta^2)\lambda\) is always non-negative under our assumptions (\(\mu \ge 1, \eta \le 1, \beta>0, \lambda \in [0,1]\)). Then, the improvement in utility is strictly positive.

Remark 2. The utility gap \(\mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full})\geq0\) indicates our selective mechanism focuses score updates on the selected forget-region while mitigating unwanted distortion in non-selected parts.

4 Experiments↩︎

4.1 Experimental Setups↩︎

4.1.0.1 Datasets and Baselines.

We follow the settings of [1]. For CelebA-HQ [40], we evaluate the deletion of individual samples at the image level in unconditional image generation. For Stable Diffusion v1.4 [34], we evaluated the memorization of the LAION dataset corresponding to each prompt in text-to-image generation. Further experimental details are in the Appendix 7.

4.1.0.2 Hyperparameters.

We use both time selection (4 ) and frequency selection (6 ) for experiments. Although we acknowledge that the choice of thresholds might vary, we fix \((t_C,t_S)=(250,750)\) based on Table 1. Thus, we mainly use \((t_1,t_2)=(250,750)\) for the unconditional case and \((t_1,t_2)=(750,1000)\) for the conditional case. Based on the observation in Figure [fig:psa], as our goal is to filter out the high-frequency, we tune \(r_t \in [0.1, 0.15]\), and unless otherwise specified, we set \(r_t=0.15\). For CelebA-HQ and Stable Diffusion from the Hugging Face Diffusers, we fine-tune the model with Adam with a batch size of 64 and 16 with a learning rate of \(5\cdot10^{-6}\) and \(10^{-5}\), respectively.

4.1.0.3 Evaluation Metrics.

The traditional evaluation of unlearning methods is two-fold: the quality of retained data samples, measured by Frechet Inception Distance (FID) [41], and the dissimilarity of forgotten samples from the originals, measured by SSCD [10]. However, SSCD ignores the quality of the generated samples, e.g., a blurry image achieves a very low SSCD, though it is not a meaningful outcome. To address this, we introduce a normalized SSCD score that considers the normalized distance between the generated image \(\hat{\mathbf{x}}_0({\mathbf{x}}_t,\theta)\) and the original image \({\mathbf{x}}_0\). Motivated by adversarial attacks [42], we project the difference onto \(\ell_2\) ball with radius \(\rho\) as \[\small \mathrm{SSCD}^\text{norm} = \mathrm{SSCD}({\mathbf{x}}_0,{\mathbf{x}}_0 + \rho \frac{\hat{\mathbf{x}}_0({\mathbf{x}}_t,\theta) - {\mathbf{x}}_0}{\|\hat{\mathbf{x}}_0({\mathbf{x}}_t,\theta) - {\mathbf{x}}_0\|_2^2+\varepsilon}). \label{eq:sscd95norm}\tag{8}\] Here, \(\varepsilon\) is a small constant to avoid division by zero, and we use \(\rho=100\) with a normalized image with 256x256 resolution. As a quality metric, we compute sample-wise aesthetic scores using the LAION-Aesthetic V2 predictor2. For Stable Diffusion, we evaluate generation quality using CLIP-IQA [43] and measure unlearning performance through the unlearning success rate, defined as the proportion of cases where all 16 memorized samples are removed. We tested with the five runs and report their mean. Variance is comparably small, shown in Table 9.

Figure 5: Ablation study with different diffusion time steps and cutoff for low-pass filter.
Table 2: Comparison of unconditional unlearning. We show the baseline scores, the scores with our method applied (+ Ours), and the resulting relative gain (%) in a separate row. For the ’Gain (%)’ row, a higher value always indicates better results. Positive gains are in blue, and degradations in red.
Method Denoising from \(t=250\) Denoising from \(t=500\)
4-6 (lr)7-9 FID-10K \(\downarrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesthetic \(\uparrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesthetic \(\uparrow\)
Pre-trained 30.3 1.257 - - - - -
Naive deletion 19.61 0.874 0.607 6.118 0.726 0.641 6.077
EraseDiff 117.81 0.133 0.551 3.702 0.096 0.517 4.359
GA Base 359.79 0.131 0.548 3.079 0.002 0.783 3.470
+ Ours 375.18 0.113 0.499 3.699 0.062 0.600 4.577
Gain (%) -4.28 +13.74 +8.94 +20.14 -3\(\cdot10^{3}\) +23.37 +31.90
SISS Base 23.42 0.336 0.430 4.094 0.299 0.501 4.845
+ Ours 23.65 0.345 0.349 5.520 0.282 0.399 6.095
Gain (%) -0.98 -2.68 +18.84 +34.83 +5.69 +20.36 +25.80
DPO Base 20.58 0.369 0.404 5.128 0.313 0.459 5.058
+ Ours 21.15 0.344 0.332 5.614 0.292 0.393 6.138
Gain (%) -2.77 +6.78 +17.82 +9.48 +6.71 +14.38 +21.35
KTO Base 23.48 0.363 0.366 5.355 0.289 0.442 5.879
+ Ours 23.58 0.373 0.340 5.470 0.280 0.377 6.274
Gain (%) -0.43 -2.75 +7.10 +2.15 +3.11 +14.71 +6.72

4.2 Experimental Results↩︎

4.2.0.1 Unconditional Data Unlearning.

For CelebA-HQ, our objective is to delete six randomly sampled celebrity faces selected by [1] from pre-trained unconditional diffusion models. Unlearning is applied image-by-image in a continual setting. As GA erases all details in a continual manner, we initialize the model as a pretrained point after each individual deletion. As explained in Section 2.3, we first inject noise to certain time steps \(t=250,500\) (only \(t=250\) in [1]) and compare the denoised image using the unlearned model and the original image.

In Figure 5, similar to the toy results in Figure 2, we focus on the middle steps, which show stable forgetting and better quality preservation in image-level unconditional unlearning. For a low-pass filter, a cutoff threshold \(r=0.15\) effectively maintains the quality. Thus, we apply our selective unlearning framework to various optimization methods (GA, SISS, DPO, and KTO) in Table 2. To ensure that a positive value consistently indicates an improvement over metric \(V\), the relative gain (%) is calculated as \((V_{\text{ours}} - V_{\text{base}}) / V_{\text{base}}\) for higher is better metric and \((V_{\text{base}} - V_{\text{ours}}) / V_{\text{base}}\) for lower is better metric. The results indicate that both time and frequency selection achieve forgetting and preserve the quality of unlearned data examples. At \(t=250\), it maintains the SSCD while improving SSCD\(^{\text{norm}}\), and at \(t=500\), it improves both metrics compared to base models. Since our selective method is optimized for the forget data, we observe a minor increase in FID for retained data.

The comparison between SSCD and SSCD\(_{\text{norm}}\) highlights the limitations of using raw similarity to evaluate unlearning. For instance, deletion-focused methods like EraseDiff and GA achieve low SSCD scores. While the results are well-aligned with target forgetting, it is merely an artifact of severe image quality degradation. Their high SSCD\(_{\text{norm}}\) scores correctly reveal that their perturbation direction is ineffective for true unlearning. In contrast, our selective framework demonstrates a superior unlearning direction by achieving significant SSCD\(_{\text{norm}}\) gains.

4.2.0.2 Conditional Text-to-Image Data Unlearning.

For Stable Diffusion, we evaluated to unlearn 45 memorized prompts selected by [1] from the LAION datasets. Fully-memorized prompts consistently generate exact replicas of the training images, whereas partially-memorized prompts contain weaker memorization prompts, thus easier to unlearn. For conditional cases, we unlearn only time steps \(t>t_S\), utilizing the condition on a text embedding of “memorized prompt". As shown in Figure 7, our method achieves a higher unlearning success rate and faster convergence than the baseline SISS throughout the training process. Figure 7 (c) further confirms that our approach yields superior unlearning (lower attack success) and better image quality (higher CLIP-IQA).

Figure 6: Visualization of generated images from the memorization with fully-memorized prompt “Mothers influence on her young hippo” at early unlearning steps [0,1,2]. The proposed method shows faster forgetting performance while maintaining quality.
a
b
c

Figure 7: Unlearning attack success rate (ASR) and CLIP-IQA when unlearning each of the memorized 45 prompts. As shown in (a) and (b), selective unlearning shows a faster and higher unlearning rate. The adaptive method does not show a big difference from the fixed one. As shown in (c), our method shows strong forgetting while preserving quality compared to existing unlearning methods.. a — Unlearning Ratio (Full ASR), b — CLIP-IQA (Quality), c — Overall Comparison

4.3 Additional Experiments↩︎

Adaptive method.\(None\) As illustrated in Figure 7 (a-b), the adaptive method in Algorithm 9 shows similar performance with the fixed window. However, the adaptive method introduces computational overhead in a pre-sampling step to determine \(t_S=\tau\). Therefore, we mainly adopt the fixed interval, leaving the adaptive method for future exploration. Furthermore, we also observed that frequency threshold scheduling can also improve the quality, as shown in Table 8.

Computation.\(None\) As shown in Table 8 using Stable Diffusion, our method requires an additional computational cost for FFT of 5.99% per epoch. Additionally, the adaptive method requires a sampling step before training begins. However, since our method converges significantly faster, the actual total training time can be reduced. Inference time remains unchanged for our methods.

Membership inference attack (MIA).\(None\) We further conduct a membership inference attack (MIA) [44], [45] evaluation based on diffusion loss, testing on a member set, hold-out dataset, and unlearned set. As shown in Table [tab:mia], existing methods suffer from over-forgetting, with relative errors reaching \(\times 107.98\) (SecMI) and \(\times 278.22\) (PIAN) on the unlearned set, far exceeding the hold-out baseline (\(\times 1.32\) and \(\times 1.12\)). Our method substantially mitigates this collateral distortion (\(\times 5.76\) and \(\times 53.20\)).

Supplementary Experiments.\(None\) Refer to Appendix 8 for supplementary experiments, including additional base analyses (Figures 11-14), evaluations on different backbones such as transformer-based architectures and Stable Diffusion v3 (Tables 3 and 4), concept unlearning results (Figure 10), LLM-based evaluations for anomaly detection (Table 5), the effect of unlearning on nearby samples (Table 6), filter band and frequency cutoff scheduling ablations (Tables 7 and 8), and statistical reliability across independent runs (Table 9). Refer to Appendix 9 for additional visualizations.

Figure 8: image.

5 Conclusion↩︎

In this paper, we address the critical issues of quality degradation and slow convergence in data unlearning for diffusion models. Our analysis reveals that forgetting is not a uniform process, but an effective region exists across time steps and spectral domains. As a limitation, this work does not provide a method for automatically discovering the optimal unlearning regions. We believe that this work assists the social impact of machine unlearning within generative models.

6 Justification of Proposition 1↩︎

Setup. Let \(\mathbb{P}_{\mathrm{orig}}\) and \(\mathbb{P}_m\) (\(m \in \{\mathrm{full}, \mathrm{sel}\}\)) be the path measures of the original and unlearned diffusion models. We denote the score perturbation as \[\Delta s_m(x_t, t) := s_{\theta_m}(x_t, t) - s_{\theta_{\mathrm{orig}}}(x_t, t).\] By Girsanov’s theorem [39], the KL divergence between the original and unlearned path measures is \[D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) = \frac{1}{2}\int_0^T g(t)^2 \mathbb{E}_{x_t} \left[ \|\Delta s_m(x_t, t)\|_2^2 \right] dt.\] To evaluate unlearning efficacy, we decompose this total unlearning-induced distortion into a selected target window \(\mathcal{S}\) (e.g., selected time-steps or specific frequencies) and its complement \(\mathcal{S}^c\). Let \[D_{\mathrm{KL}}^{\mathcal{W}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) := \frac{1}{2}\int_{\mathcal{W}} g(t)^2 \mathbb{E}_{x_t} [\|\Delta s_m\|_2^2] dt\] denote the regional divergence over a window \(\mathcal{W}\). Thus, the total divergence can be formulated as \[D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) = D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) + D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m).\] For \(\beta > 0\), we define the forgetting–preservation utility as: \[\label{qefivnbw} \mathcal{J}_{\beta}(m) := D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) - \beta D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m).\tag{9}\] Note that \(D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m)\) is a proxy for the forgetting action. If \(\mathcal{S}\) is chosen to contain the components that encode the forget-set information, a larger divergence on \(\mathcal{S}\) means the method is actively modifying the targeted information. Conversely, \(D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m)\) is a proxy for collateral distortion. Since \(\mathcal{S}^c\) is not the region we intend to unlearn, score changes there are undesirable side effects that can degrade retained content and sample quality.

Assumptions.

  1. Finite divergence: \(D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_m) < \infty\).

  2. Focused updates in \(\mathcal{S}\): Due to the frequent update on the selected region under the same training budget, selective unlearning amplifies changes in the target region (\(\|\Delta s_{\mathrm{sel}}\|_2 \ge \mu \|\Delta s_{\mathrm{full}}\|_2\) for \(\mu \ge 1\) in \(\mathcal{S}\)) and weaker changes in \(\mathcal{S}^c\) (\(\|\Delta s_{\mathrm{sel}}\|_2 \le \eta \|\Delta s_{\mathrm{full}}\|_2\) for \(\eta \in [0,1]\) in \(\mathcal{S}^c\)). Empirical validations are presented in Appendix 6.1.

Proposition 2. Under the assumptions above, the trade-off utility of \(\mathcal{J}_{\beta}(\mathrm{sel})\) is larger than \(\mathcal{J}_{\beta}(\mathrm{full)}\). For every \(\beta>0\) and \(\lambda := \frac{D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}})}{D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}})} \in [0,1]\), their gap satisfies: \[\mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full}) \ge \left[ (\mu^2-1)(1-\lambda) + \beta(1-\eta^2)\lambda \right] D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}),\] where \(\lambda\) indicates the proportion of collateral distortion compared to the full-unlearning.

Proof. From the definition of the utility function given in Eq. 7 , the difference is: \[\begin{align} \mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full}) &= \left( D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{sel}}) - D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) \right) \\ &\quad - \beta \left( D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{sel}}) - D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) \right). \end{align}\] By Assumption 2, the condition \(\|\Delta s_{\mathrm{sel}}\|_2 \ge \mu \|\Delta s_{\mathrm{full}}\|_2\) in \(\mathcal{S}\) ensures the divergence in the forget-window satisfies: \[D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{sel}}) \ge \mu^2 D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}).\] Moreover, the condition \(\|\Delta s_{\mathrm{sel}}\|_2 \le \eta \|\Delta s_{\mathrm{full}}\|_2\) in \(\mathcal{S}^c\) (Assumption 3) ensures the divergence in the non-target window satisfies: \[D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{sel}}) \le \eta^2 D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}).\] Substituting these bounds into the utility difference yields: \[\mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full}) \ge (\mu^2-1) D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) + \beta(1-\eta^2) D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}).\] By the definitions of \(\lambda\) and total \(D_{\mathrm{KL}}\), we can express the regional divergences as: \[D_{\mathrm{KL}}^{\mathcal{S}^c}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) = \lambda D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}), \quad D_{\mathrm{KL}}^{\mathcal{S}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) = (1-\lambda) D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}).\] Substituting these exact expressions into the lower bound directly yields the stated inequality: \[\begin{align} \mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full}) &\ge (\mu^2-1)(1-\lambda)D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) + \beta(1-\eta^2)\lambda D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}) \\ &= \left[ (\mu^2-1)(1-\lambda) + \beta(1-\eta^2)\lambda \right] D_{\mathrm{KL}}(\mathbb{P}_{\mathrm{orig}} \,\|\, \mathbb{P}_{\mathrm{full}}). \end{align}\] ◻

Remark 1. The bracketed term \((\mu^2-1)(1-\lambda) + \beta(1-\eta^2)\lambda\) is always non-negative under our assumptions (\(\mu \ge 1, \eta \le 1, \beta>0, \lambda \in [0,1]\)). Then, the improvement in utility is strictly positive.

Remark 2. The utility gap \(\mathcal{J}_{\beta}(\mathrm{sel}) - \mathcal{J}_{\beta}(\mathrm{full})\geq0\) indicates our selective mechanism focuses score updates on the selected forget-region while mitigating unwanted distortion in non-selected parts.

6.1 Empirical Validation of Assumptions↩︎

Our theoretical analysis relies on two key assumptions: enhanced forgetting in the target region (Assumption 2, \(\mu \ge 1\)) and suppressed distortion in the non-target region (Assumption 2, \(\eta \le 1\)). We empirically validate both assumptions across the time and frequency domains.

6.1.0.1 Enhanced Forgetting (\(\mu \ge 1\)).

To verify that targeted training induces stronger score updates within the specified region compared to full unlearning, we evaluate the \(\mu\)-ratio (\(\|\Delta s_{\mathrm{sel}}\|_2^2 / \|\Delta s_{\mathrm{full}}\|_2^2\)) strictly within the targeted window \(\mathcal{S}\). As shown in Table ¿tbl:tab:mu95ratio?, the relative ratios are strictly greater than 1 across all targeted time windows (Table [tab:window95ratio95time95mu]) and frequency bands (Table [tab:window95ratio95freq95mu]). This confirms that selective unlearning strengthens its update energy on the selected region.

0.45

Empirical validation of \(\mu^2 \ge 1\). We report the ratio \(\|\Delta s_{\mathrm{sel}}\|_2^2 / \|\Delta s_{\mathrm{full}}\|_2^2\) strictly within the target region \(\mathcal{S}\). Values \(>1\) indicate that targeted training induces stronger score updates compared to full unlearning. \(W_1\), \(W_2\), and \(W_3\) denote the early \([0, 250)\), middle \([250, 750)\), and late \([750, 1000]\) diffusion time windows. Low, Mid, and High correspond to the spatial frequency bands.
Target Region (\(\mathcal{S}\)) Ratio (\(\|\Delta s_{\mathrm{sel}}\|_2^2 / \|\Delta s_{\mathrm{full}}\|_2^2\))
\(W_1\) (Early) 1.2515
\(W_2\) (Middle) 1.4950
\(W_3\) (Late) 1.3095

0.45

Empirical validation of \(\mu^2 \ge 1\). We report the ratio \(\|\Delta s_{\mathrm{sel}}\|_2^2 / \|\Delta s_{\mathrm{full}}\|_2^2\) strictly within the target region \(\mathcal{S}\). Values \(>1\) indicate that targeted training induces stronger score updates compared to full unlearning. \(W_1\), \(W_2\), and \(W_3\) denote the early \([0, 250)\), middle \([250, 750)\), and late \([750, 1000]\) diffusion time windows. Low, Mid, and High correspond to the spatial frequency bands.
Target Region (\(\mathcal{S}\)) Ratio (\(\|\Delta s_{\mathrm{sel}}\|_2^2 / \|\Delta s_{\mathrm{full}}\|_2^2\))
Low-pass 1.2403
Mid-pass 13.3800
High-pass 1.3850

6.1.0.2 Suppressed Distortion (\(\eta \le 1\)).

To quantify the bounded out-of-region effect, we measure the normalized perturbation across different evaluation domains when the model is trained on a specific target region. The results in Table ¿tbl:tab:eta95ratio? indicate that selective unlearning predominantly affects the intended phase or band compared to non-target counterparts. The off-diagonal elements in both the time domain (Table [tab:window95ratio95time95eta]) and the frequency domain (Table [tab:window95ratio95freq95eta]) experimentally support \(\eta < 1\).

0.45

Empirical validation of \(\eta^2 \le 1\). We report the normalized perturbation \(\|\Delta s_{\mathrm{sel}}\|_2^2\) across evaluation regions (rows) when trained on a specific target \(\mathcal{S}\) (columns). Diagonal elements are scaled to 1, while off-diagonal elements show the bounded effects of collateral distortion.
Target Region (\(\mathcal{S}\))
2-4 Eval Region \(W_1\) \(W_2\) \(W_3\)
\(W_1\) 1.0000 0.2511 0.0009
\(W_2\) 0.7532 1.0000 0.0010
\(W_3\) 0.4939 0.7037 1.0000

0.45

Empirical validation of \(\eta^2 \le 1\). We report the normalized perturbation \(\|\Delta s_{\mathrm{sel}}\|_2^2\) across evaluation regions (rows) when trained on a specific target \(\mathcal{S}\) (columns). Diagonal elements are scaled to 1, while off-diagonal elements show the bounded effects of collateral distortion.
Target Region (\(\mathcal{S}\))
2-4 Eval Region Low Mid High
Low 1.0000 0.2535 0.1681
Mid 0.4116 1.0000 0.0511
High 0.0502 0.0814 1.0000

6.2 Time and Frequency Understanding for \(\lambda\)↩︎

6.2.0.1 Time-Selective Unlearning.

Defining the target region as a specific time window \(\mathcal{S} = \mathcal{T}_s \subset [0, T]\) formulates time-selective unlearning. In this case, the ratio \(\lambda_t\) represents the fraction of the full-unlearning distortion that occurs outside the targeted time window \(\mathcal{T}_s\): \[\label{eq:lambda95time95vC} \lambda_t \;:=\; \frac{\int_{[0,T]\setminus\mathcal{T}_s} g(t)^2\,\mathbb{E}_{x_t}\!\left[\|\Delta s_{\mathrm{full}}(x_t, t)\|_2^2\right] dt}{\int_0^T g(t)^2\,\mathbb{E}_{x_t}\!\left[\|\Delta s_{\mathrm{full}}(x_t, t)\|_2^2\right] dt} \;\in\;[0,1].\tag{10}\]

6.2.0.2 Frequency-Selective Unlearning.

Plancherel’s theorem [46] establishes an equivalence between spatial and spectral energy: \[\label{eq:plancherel95vC} \|\Delta s_m(x_t, t)\|_2^2 \;=\; \sum_{\boldsymbol{\omega}} \|\widehat{\Delta s}_m(\boldsymbol{\omega},t)\|_2^2,\tag{11}\] where \(\widehat{\Delta s}_m(\boldsymbol{\omega},t)\) is the discrete Fourier transform of the score perturbation across spatial dimensions at frequency \(\boldsymbol{\omega}\). By applying a low-pass filter with cutoff \(r\), we define our targeted forgetting region in the frequency domain as \(\mathcal{S} = \Omega_r := \{|\boldsymbol{\omega}|\le r\}\). Consequently, the collateral distortion ratio \(\lambda_\omega\), corresponding to the unselected high-frequency band, is defined as: \[\label{eq:lambda95freq95vC} \lambda_\omega \;:=\; \frac{\int_0^T g(t)^2\!\sum_{|\boldsymbol{\omega}|>r}\mathbb{E}_{x_t}\!\left[\|\widehat{\Delta s}_{\mathrm{full}}(\boldsymbol{\omega},t)\|_2^2\right] dt}{\int_0^T g(t)^2\!\sum_{\boldsymbol{\omega}}\mathbb{E}_{x_t}\!\left[\|\widehat{\Delta s}_{\mathrm{full}}(\boldsymbol{\omega},t)\|_2^2\right] dt} \;\in\;[0,1].\tag{12}\]

7 Experimental Details↩︎

7.0.0.1 Experimental Setups.

We basically follow the experimental setups in SISS [1]. We first make note of an important detail in their paper.

All the diffusion models were provided in the Huggingface diffusers library with a U-Net backbone. For the CelebA-HQ dataset, we used a pretrained checkpoint released by [3] at https://huggingface.co/google/ddpm-celebahq-256. For the Stable Diffusion experiments, we used version 1.4 at https://huggingface.co/CompVis/stable-diffusion-v1-4 as the pretrained checkpoint and used a 50-step DDIM sampler [47].

Regarding hyperparameters, we follow [1] without additional tuning.

For the experimental setups, we mainly use NVIDIA A40 GPUs with the PyTorch library and utilize NVIDIA A100 GPUs for parallel runs.

7.0.0.2 Memorized Prompts of Stable Diffusion.

[1] constructed 45 prompts in Stable Diffusion v1.4. Since only one LAION image corresponds to each prompt, synthetic datasets are generated by sampling 128 images and applying k-means clustering for classification. A “fully-memorized” prompt refers to a case where Stable Diffusion repeatedly reproduces the same outcome, whereas a “partially-memorized” prompt is obtained by manually adding or deleting tokens, producing more diverse outputs that are easier to unlearn.

7.0.0.3 Preference Optimization.

Direct Preference Optimization (DPO) [48] is widely used to evaluate preference alignment in language models. Originally developed for Reinforcement Learning from Human Feedback (RLHF), it has recently also been applied to diffusion fine-tuning [49]. For unlearning, Negative Preference Optimization (NPO) [50] has been proposed as an alternative to gradient descent. Unlike gradient ascent [50], NPO leverages the initial model as a reference point, which helps mitigate overfitting by keeping the optimization close to the initialization. In the diffusion setting, [4] applied NPO for alignment. Another line of work, Kahneman–Tversky Optimization (KTO) [51], has its own strength since the method does not require positive–negative pairs. [52] extended KTO to diffusion for pair-free feedback alignment. In this paper, we follow the formulation of forget loss with DPO [49] and KTO [52].

To formulate this, let the per-sample diffusion loss be \(\ell(\theta, {\mathbf{x}}) = \mathbb{E}_{t}[w_t \| s_\theta({\mathbf{x}}_t, t) - \nabla_{{\mathbf{x}}_t} \log q({\mathbf{x}}_t|{\mathbf{x}}) \|_2^2]\). We define the implicit reward as the loss reduction relative to the reference model: \(r(\theta, {\mathbf{x}}) = \ell(\theta_{\text{ref}}, {\mathbf{x}}) - \ell(\theta, {\mathbf{x}})\). By maximizing the implicit reward gap defined by the difference in diffusion loss relative to the reference model \(\theta_{\text{ref}}\), the framework effectively minimizes the likelihood of generating forget concepts while preserving general capabilities. The objectives for DPO and KTO are then given by for forget data \({\mathbf{x}}_f\) and retain data \({\mathbf{x}}_r\): \[\begin{align} \mathcal{L}_{\text{DPO}}(\theta) &= - \mathbb{E}_{{\mathbf{x}}_r \sim \mathcal{D}_R, {\mathbf{x}}_f \sim \mathcal{D}_F} \Bigl[ \log \sigma \Bigl( \beta \bigl( r(\theta, {\mathbf{x}}_r) - r(\theta, {\mathbf{x}}_f) \bigr) \Bigr) \Bigr]. \tag{13} \\ \mathcal{L}_{\text{KTO}}(\theta) &= \mathbb{E}_{{\mathbf{x}}_r \sim \mathcal{D}_R} \Bigl[ 1 - \sigma \bigl( \beta r(\theta, {\mathbf{x}}_r) - z_{\text{ref}} \bigr) \Bigr] \nonumber \\ &\quad + \lambda \mathbb{E}_{{\mathbf{x}}_f \sim \mathcal{D}_F} \Bigl[ 1 - \sigma \bigl( z_{\text{ref}} - \beta r(\theta, {\mathbf{x}}_f) \bigr) \Bigr]. \tag{14} \end{align}\] \(\sigma\) is the sigmoid function, \(\beta\) is a temperature hyperparameter, and \(z_{\text{ref}}\) is the reference point for the reward.

7.0.0.4 Algorithm of Adaptive Search.

Based on [23], we present pseudocode of the adaptive transition point \(\tau=t_S\) search method in Algorithm 9.

Figure 9: Adaptive Transition Time Search and Selective Unlearning

8 Supplementary Experiments↩︎

8.1 Different Backbones↩︎

To evaluate the adaptability of latent models for image-level generation, we conduct experiments using the Latent Diffusion Model (LDM) [34] pretrained on the CelebHQ dataset3. The results are presented in Table 3. Furthermore, we extend our evaluation to Stable Diffusion v3.0. built upon the Diffusion Transformer (DiT) architecture [53]. Since Stable Diffusion v3.0 has practical guardrail methods to prevent the memorization observed in previous Stable Diffusion v1.4, it is difficult to generate the memorized example in Stable 1.4. Thus, we rather calculate the SSCD and SSCD\(^{\text{norm}}\) while using the same prompts. The corresponding results are reported in Table 4. The gains of our method are smaller compared to Stable Diffusion v1.4 due to the flow-matching-based property, but we still observe that controlling time and frequency helps unlearning.

Tables 3 and 4 demonstrate the adaptability and effectiveness of our proposed method across diverse architectures. We did not perform additional hyperparameter tuning for these experiments.

Table 3: Comparison of unlearning methods on latent diffusion models with a relative Gain. For each method, we show the baseline scores, the scores with our method applied (+ Ours), and the resulting relative gain (%) in a separate row. For the ’Gain (%)’ row, a higher value always indicates better performance. Positive gains are colored in blue, and degradations are in red.
Method Denoising from \(t=250\) Denoising from \(t=500\)
4-6 (lr)7-9 FID-10K \(\downarrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesthetic \(\uparrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesthetic \(\uparrow\)
SISS Base 17.800 0.431 0.547 4.601 0.281 0.588 5.092
+ Ours 16.960 0.437 0.520 5.076 0.250 0.536 4.955
Gain (%) +4.72 -1.54 +4.95 +10.32 +10.90 +8.84 -2.69
DPO Base 28.630 0.428 0.573 4.505 0.372 0.630 4.880
+ Ours 28.630 0.372 0.630 4.880 0.295 0.546 4.996
Gain (%) 0.00 +13.02 -9.89 +8.34 +20.63 +13.34 +2.38
KTO Base 28.630 0.372 0.630 4.880 0.458 0.472 5.812
+ Ours 31.360 0.498 0.576 5.072 0.457 0.493 6.446
Gain (%) -9.54 -33.65 +8.58 +3.93 +0.19 -4.48 +10.90
Table 4: Quantitative comparison on Stable Diffusion v3.0. with Diffusion Transformer (DiT) backbone. Arrows indicate the direction of better performance (\(\uparrow\) for higher, \(\downarrow\) for lower).
Measure SISS SISS + Ours
CLIP-IQA (Partial) (\(\uparrow\)) 0.863 0.856 (-0.80%)
CLIP-IQA (Full) (\(\uparrow\)) 0.886 0.890 (+0.50%)
SSCD (Partial) (\(\downarrow\)) 0.349 0.302 (-13.67%)
SSCD (Full) (\(\downarrow\)) 0.421 0.362 (-14.06%)
SSCD Norm (Partial) (\(\downarrow\)) 0.704 0.698 (-0.87%)
SSCD Norm (Full) (\(\downarrow\)) 0.724 0.707 (-2.30%)

8.2 Concept Unlearning↩︎

As stated in the introduction, unlearning specific concepts, such as nudity or violence, is another major direction in unlearning research. We examine whether the choice of timesteps also affects concept generation in terms of “memorization of certain concepts.” We utilize the nudity prompts used in Ring-a-bell [54] using the GitHub4 of SAFREE [5].

We compare SISS and SISS+Ours using the same time window used SD 1.4, and compute the CLIP-IQA quality score. For the unlearning measure, we measure the nudity detection rate using NudeNet5, where an attack is counted if the NudeNet classifier probability exceeds 0.45. As shown in Figure 10, the proposed one shows faster convergence and superior forgetting behavior.

a
b

Figure 10: Unlearning with concept “nudity” using Ring-a-bell dataset. Our methods show faster unlearning convergence and a high nudity deletion rate.. a — Nudity Detection rate, b — CLIP-IQA (Quality)

8.3 Importance of Unlearned Quality↩︎

We argue that significant degradation or distinct artifacts in these outputs can create a privacy vulnerability. If the unlearned model generates evident anomalies for specific queries, it provides a clue to adversaries, allowing them to identify which data points were deleted (i.e., membership inference via outlier detection).

To verify this hypothesis, we conducted an experiment using advanced Large Language Models (LLMs) as anomaly detectors. We provided the LLMs with generated images and instructed them to evaluate whether the images are realistic (in-distribution) or abnormal (out-of-distribution) based on the following prompt:

Prompt: “You are an image quality and anomaly detection expert. You are given two image sets: The first set contains reference (in-distribution) images. Assign a score to each query image: 1.0 = Realistic / In-distribution (no visible generation problem), 0.0 = Abnormal / Out-of-distribution (clear generation failure or artifacts).”

As shown in Table 5, baseline methods like SISS exhibited low scores, indicating that their outputs are easily distinguishable as anomalies. In contrast, our method maintains higher generation quality, effectively masking the trace of unlearning. Furthermore, recent studies [55], [56] warn that unlearning mechanisms are vulnerable against relearning attacks, which can recover deleted data using residual information such as embedding similarity. Therefore, ensuring the quality of unlearned samples is crucial not only for aesthetics but also for preventing privacy attacks.

Table 5: Anomaly detection scores evaluated by various LLMs on unlearned outputs. A higher score indicates the generated image is perceived as realistic (in-distribution), making it harder to detect that unlearning has occurred.
LLM Evaluator SISS SISS + Ours
ChatGPT-5 0.22 0.67
Claude Sonnet 4.5 0.13 0.83
Gemini 2.5 Flash 0.00 0.50

To investigate the impact of unlearning on similar samples, we choose the five nearest neighbors for each forgotten image, and report the mean SSCD between their original and reconstructed images as well as the aesthetic score of the reconstructed samples in Table 6. As data unlearning focuses on individual deletion, the SSCD on these nearest images does not decrease. However, we do observe a slight drop in aesthetic quality after unlearning. By implementing our selective unlearning, this aesthetic degradation is mitigated. Consequently, selective unlearning helps preserve both the semantics and the overall quality of nearby examples.

8.4 Ablation on Frequency Filtering↩︎

Filter band ablation.\(None\) We evaluate low-pass (our method), mid-pass, and high-pass filters for unlearning in Table 7. The results indicate that deleting low-frequency information while preserving high-frequency details performs the best, similar to not deleting Phase III in time steps. Otherwise, both unlearning and quality preservation fail.

Frequency cutoff scheduling.\(None\) Our frequency selection can be further improved with different filtering schedules. Therefore, we test four additional variants during training, including Linear Increase (\(0.05 \to 0.2\)), Linear Decay (\(0.2 \to 0.05\)), and Stepwise strategies, to assess the impact of filtering high-frequency content over the unlearning fine-tuning. As shown in Table 8, the Linear Increase strategy, which retains a greater amount of high-frequency information during the later training stages, results in the highest image quality. This indicates that frequency filtering should be dynamically adjusted to preserve fine-grained details during the later stages of unlearning.

8.5 Statistical reliability↩︎

To verify that the gap is statistically significant, we further evaluate the SISS and KTO experiments in Table 9 across five independent runs, which yield small standard deviations.

Table 6: Reconstruction on the five nearest neighbors of each forgotten image. Selective unlearning preserves both semantics (SSCD) and visual quality (aesthetic score) of nearby samples.
\(t = 250\) \(t = 500\)
2-3(lr)4-5 Model SSCD \(\downarrow\) Aesthetic \(\uparrow\) SSCD \(\downarrow\) Aesthetic \(\uparrow\)
Orig. 0.8817 6.3894 0.7405 6.7483
SISS 0.8812 6.2441 0.7394 6.5261
+ Ours 0.8797 6.4467 0.7439 6.5873

6pt

Table 7: Effect of filter band on selective unlearning. The low-pass choice, which preserves high-frequency detail, yields the best trade-off across both timesteps.
\(t = 250\) \(t = 500\)
2-3(lr)4-5 Filter band SSCD \(\downarrow\) Aesthetic \(\uparrow\) SSCD \(\downarrow\) Aesthetic \(\uparrow\)
Low-pass (ours) 0.331 5.553 0.264 5.904
Mid-pass 0.364 4.374 0.329 5.043
High-pass 0.379 4.143 0.227 4.670

6pt

Table 8: Different schedules of the low-pass cutoff \(r_t\) during unlearning fine-tuning. Linear Increase (\(0.05 \to 0.2\)) gives the best quality at both evaluation timesteps.
\(t = 250\) \(t = 500\)
2-4(lr)5-7 Schedule SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesth.\(\uparrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesth.\(\uparrow\)
SISS 0.336 0.430 4.094 0.299 0.501 4.845
Fixed 0.345 0.349 5.520 0.282 0.399 6.095
Linear Increase 0.323 0.322 5.709 0.260 0.382 5.832
Linear Decay 0.429 0.422 5.296 0.317 0.448 5.722
Stepwise Increase 0.328 0.331 5.631 0.255 0.392 5.918
Stepwise Decay 0.431 0.419 5.186 0.317 0.438 5.535

3pt

Table 9: Mean \(\pm\) standard deviation over five independent runs on CelebA-HQ. Adding our selective scheme yields consistent gains on the perturbation-normalized metric SSCD\(^{\text{norm}}\) and on the aesthetic score, with negligible variance.
\(t = 250\) \(t = 500\)
3-5(lr)6-8 Method FID \(\downarrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesth.\(\uparrow\) SSCD \(\downarrow\) SSCD\(^{\text{norm}}\) \(\downarrow\) Aesth.\(\uparrow\)
SISS \(23.42_{\pm 0.31}\) \(0.336_{\pm 0.002}\) \(0.430_{\pm 0.003}\) \(4.094_{\pm 0.054}\) \(0.299_{\pm 0.006}\) \(0.501_{\pm 0.004}\) \(4.845_{\pm 0.051}\)
+ Ours \(23.65_{\pm 0.35}\) \(0.345_{\pm 0.003}\) \(0.349_{\pm 0.002}\) \(5.520_{\pm 0.063}\) \(0.282_{\pm 0.004}\) \(0.399_{\pm 0.003}\) \(6.095_{\pm 0.049}\)
KTO \(23.48_{\pm 0.15}\) \(0.363_{\pm 0.002}\) \(0.366_{\pm 0.001}\) \(5.355_{\pm 0.062}\) \(0.289_{\pm 0.007}\) \(0.442_{\pm 0.005}\) \(5.879_{\pm 0.089}\)
+ Ours \(23.58_{\pm 0.12}\) \(0.373_{\pm 0.003}\) \(0.340_{\pm 0.003}\) \(5.470_{\pm 0.049}\) \(0.280_{\pm 0.010}\) \(0.377_{\pm 0.006}\) \(6.274_{\pm 0.024}\)

3pt

8.6 Base Experiments↩︎

We illustrate some details through a toy example. For Figure 2, we construct a multi-layer perceptron diffusion model trained with the DDPM objective [3]. The model is trained until convergence. After training, we delete the forget samples (\(\times\)) using gradient ascent, while applying gradient descent to certain retain data points (◆).

Except for Figure 2, all experiments in Section 3 are conducted on the CelebHQ dataset used in the main table 2. We now provide additional results. In Figure [fig:distribution95time], we utilize the DINOv3 embedding [32] to calculate the similarity between data samples. We also measured in pixel-wise similarity in Figure 12, where we failed to observe similar patterns. The nearest images of individual data samples are in Figure 11. Individual gradient norms are in Figure [fig:gradient95norms95supp]. Finally, we report the remaining results of Figure [fig:psa] in Figures 13 and 14.

a

b

c

d

e

f

Figure 11: Forget samples and their nearest neighborhood on DINOv3 embedding [32]..

Figure 12: Similarity in pixel-level.

a

b

c

Figure 13: Additional results of Power Spectral Density (PSD) before and after unlearning on the forget dataset. As shown in the right figure, when the generated data from the unlearned model preserves aesthetic quality, the difference in high-frequency components is diminished..

a

b

c

Figure 14: Additional results of Power Spectral Density (PSD) before and after unlearning on the retain dataset..

9 Visualization↩︎

9.0.0.1 CelebA-HQ Results.

We first visualize the results for each method: SISS, SISS+Ours, and variational results on different time steps in Figures 15 to 19.

Figure 15: Unlearning with SISS.
Figure 16: Unlearing with SISS+Ours.
Figure 17: Unlearning on time steps [0, 500].
Figure 18: Unlearning on time steps [500, 1000].
Figure 19: Unlearning on time steps [750, 1000].

9.0.0.2 Stable Diffusion Results.

We visualize the results of text-to-image data unlearning in Figures 20 to 23.

Figure 20: Visualization of (left) Partially-memorized and (right) Fully-memorized results after unlearning of the prompt “Living in the Light with Ann Graham Lotz".
Figure 21: Visualization of (left) Partially-memorized and (right) Fully-memorized results after unlearning of the prompt “Living in the Light with Ann Graham Lotz" using different time steps for unlearning.
Figure 22: Visualization of (left) Partially-memorized and (right) Fully-memorized results after unlearning of the prompt “Rambo 5 und Rocky Spin-Off - Sylvester Stallone gibt Updates".
Figure 23: Visualization of (left) Partially-memorized and (right) Fully-memorized results after unlearning of the prompt “Daft Punk, Jay Z Collaborate on {"Computerized"}".

References↩︎

[1]
S. Alberti, K. Hasanaliyev, M. Shah, and booktitle=The. T. I. C. on L. R. Stefano Ermon, “Data unlearning in diffusion models,” 2025.
[2]
L. Bourtoule et al., “Machine unlearning,” 2021 , organization={IEEE}, pp. 141–159.
[3]
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
[4]
F.-Y. Wang, Y. Shui, J. Piao, K. Sun, and booktitle=The. T. I. C. on L. R. Hongsheng Li, “Diffusion-NPO: Negative preference optimization for better preference aligned generation of diffusion models,” 2025.
[5]
J. Yoon, S. Yu, V. Patil, H. Yao, and booktitle=The. T. I. C. on L. R. Mohit Bansal, “SAFREE : Training-free and adaptive guard for safe text-to-image and video generation,” 2025, [Online]. Available: https://openreview.net/forum?id=hgTFotBRKl.
[6]
M. Kim, D. Kim, A. Yusuf, S. Ermon, and booktitle=The. T. A. C. on N. I. P. S. Mijung Park, “Training-free safe denoisers for safe use of diffusion models,” 2025.
[7]
M. Kim, Y.-H. Kim, and booktitle=The. F. I. C. on L. R. Mijung Park, “SAFETY -GUIDED FLOW (SGF): A UNIFIED FRAMEWORK FOR NEGATIVE GUIDANCE IN SAFE GENERATION,” 2026, [Online]. Available: https://openreview.net/forum?id=EA80Zib9UI.
[8]
C. Fan, J. Liu, Y. Zhang, E. Wong, D. Wei, and booktitle=The. T. I. C. on L. R. Sijia Liu, “SalUn: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation,” 2024.
[9]
J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Yoon Sungroh, “Perception prioritized training of diffusion models,” 2022, pp. 11472–11481.
[10]
E. Pizzi, S. D. Roy, S. N. Ravindra, P. Goyal, and booktitle=Proceedings. of the I. C. on C. V. and P. R. Douze Matthijs, “A self-supervised descriptor for image copy detection,” 2022, pp. 14532–14542.
[11]
B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications, vol. 12, no. 3, pp. 313–326, 1982.
[12]
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and booktitle=International. C. on L. R. Ben Poole, “Score-based generative modeling through stochastic differential equations,” 2021.
[13]
R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and booktitle=Proceedings. of the I. international conference on computer vision Bau David, “Erasing concepts from diffusion models,” 2023, pp. 2426–2436.
[14]
K. Srivatsan, F. Shamshad, M. Naseer, V. M. Patel, and booktitle=Proceedings. of the C. V. and P. R. C. Nandakumar Karthik, “Stereo: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models,” 2025, pp. 23765–23774.
[15]
Y.-H. Park et al., “Direct unlearning optimization for robust and safe text-to-image models,” 2024, [Online]. Available: https://openreview.net/forum?id=UdXE5V2d0O.
[16]
I. Goodfellow et al., “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
[17]
R. Singhal et al., “A general framework for inference-time scaling and steering of diffusion models,” 2025.
[18]
F. Koulischer, J. Deleu, G. Raya, T. Demeester, and booktitle=The. T. I. C. on L. R. Luca Ambrogioni, “Dynamic negative guidance of diffusion models,” 2025.
[19]
A. Golatkar, A. Achille, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Soatto Stefano, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” 2020, pp. 9304–9312.
[20]
J. Wu, T. Le, M. Hayat, and booktitle=Proceedings. of the C. V. and P. R. C. Harandi Mehrtash, “Erasing undesirable influence in diffusion models,” 2025, pp. 28263–28273.
[21]
S. Zhai et al., “Membership inference on text-to-image diffusion models via conditional likelihood discrepancy,” Advances in Neural Information Processing Systems, vol. 37, pp. 74122–74146, 2024.
[22]
Y. Wen, Y. Liu, C. Chen, and booktitle=The. T. I. C. on L. R. Lyu Lingjuan, “Detecting, explaining, and mitigating memorization in diffusion models,” 2024.
[23]
A. Jain et al., “Classifier-free guidance inside the attraction basin may cause memorization,” 2025, pp. 12871–12879.
[24]
M. Li and S. Chen, “Critical windows: Non-asymptotic theory for feature emergence in diffusion models.” 2024 , eprint={2403.01633}, archivePrefix={arXiv}, primaryClass={cs.LG}, [Online]. Available: https://arxiv.org/abs/2403.01633.
[25]
G. Biroli, T. Bonnaire, V. de Bortoli, and M. Mézard, “Dynamical regimes of diffusion models,” Nature Communications, vol. 15, no. 1, p. 9957, Nov. 2024.
[26]
A. Sclocchi, A. Favero, and M. Wyart, “A phase transition in diffusion models reveals the hierarchical nature of data,” Proceedings of the National Academy of Sciences, vol. 122, no. 1, p. e2408799121, 2025, doi: 10.1073/pnas.2408799121.
[27]
M. Li, A. Karan, and booktitle =. P. of the 42nd. I. C. on M. L. Chen Sitan, “Blink of an eye: A simple theory for feature localization in generative models,” 2025 , editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, vol. 267 , series = Proceedings of Machine Learning Research, pp. 35047–35080, [Online]. Available: https://proceedings.mlr.press/v267/li25at.html.
[28]
G. Raya and booktitle=Thirty. C. on N. I. P. S. Luca Ambrogioni, “Spontaneous symmetry breaking in generative diffusion models,” 2023.
[29]
K. Georgiev, J. Vendrow, H. Salman, S. M. Park, and A. Madry, “The journey, not the destination: How data guides diffusion models,” arXiv preprint arXiv:2312.06205, 2023.
[30]
M. Paul, S. Ganguli, and G. K. Dziugaite, “Deep learning on a data diet: Finding important examples early in training,” Advances in neural information processing systems, vol. 34, pp. 20596–20607, 2021.
[31]
S. Pal, C. Wang, J. Diffenderfer, B. Kailkhura, and booktitle=Second. C. on L. M. Sijia Liu, “LLM unlearning reveals a stronger-than-expected coreset effect in current benchmarks,” 2025.
[32]
O. Siméoni et al., “DINOv3.” 2025 , eprint={2508.10104}, archivePrefix={arXiv}, primaryClass={cs.CV}, [Online]. Available: https://arxiv.org/abs/2508.10104.
[33]
D. Jeon, D. Kim, and booktitle=Forty. I. C. on M. L. No Albert, “Understanding and mitigating memorization in generative models via sharpness of probability landscapes,” 2025.
[34]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and booktitle=Proceedings. of the I. conference on computer vision and pattern recognition Ommer Björn, “High-resolution image synthesis with latent diffusion models,” 2022, pp. 10684–10695.
[35]
M. Kirchhof, J. Thornton, L. Béthune, P. Ablin, E. Ndiaye, and booktitle=Forty. I. C. on M. L. marco cuturi, “Shielded diffusion: Generating novel and diverse images using sparse repellency,” 2025.
[36]
S. Yang, Y. Chen, L. Wang, S. Liu, and booktitle=The. T. I. C. on L. R. Ying-Cong Chen, “Denoising diffusion step-aware models,” 2024.
[37]
P. Lian, Y. Cai, S. Li, and B. Bao, “Unveiling impact of frequency components on membership inference attacks for diffusion models,” arXiv preprint arXiv:2505.20955, 2025.
[38]
J. S. Choi, K. Lee, S. Yu, Y. Choi, J. Shin, and K. Lee, “Enhancing motion dynamics of image-to-video models via adaptive low-pass guidance,” arXiv preprint arXiv:2506.08456, 2025.
[39]
R. S. Liptser and A. N. Shiriaev, Statistics of random processes: General theory, vol. 394. Springer, 1977.
[40]
Z. Liu, P. Luo, X. Wang, and booktitle =. P. of I. C. on C. V. (ICCV). Tang Xiaoou, “Deep learning face attributes in the wild,” 2015.
[41]
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
[42]
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and booktitle=International. C. on L. R. Vladu Adrian, “Towards deep learning models resistant to adversarial attacks,” 2018.
[43]
J. Wang, K. C. Chan, and booktitle=Proceedings. of the A. conference on artificial intelligence Loy Chen Change, “Exploring clip for assessing the look and feel of images,” 2023, vol. 37, pp. 2555–2563.
[44]
J. Duan, F. Kong, S. Wang, X. Shi, and booktitle=International. C. on M. L. Xu Kaidi, “Are diffusion models vulnerable to membership inference attacks?” 2023 , organization={PMLR}, pp. 8717–8730.
[45]
F. Kong et al., “An efficient membership inference attack for the diffusion model by proximal initialization,” 2024.
[46]
K. Yosida, Functional analysis. Springer Science & Business Media, 2012.
[47]
J. Song, C. Meng, and booktitle=International. C. on L. R. Ermon Stefano, “Denoising diffusion implicit models,” 2020.
[48]
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, pp. 53728–53741, 2023.
[49]
B. Wallace et al., “Diffusion model alignment using direct preference optimization,” 2024, pp. 8228–8238.
[50]
R. Zhang, L. Lin, Y. Bai, and booktitle=First. C. on L. M. Song Mei, “Negative preference optimization: From catastrophic collapse to effective unlearning,” 2024.
[51]
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and booktitle=Forty. I. C. on M. L. Kiela Douwe, “Model alignment as prospect theoretic optimization,” 2024.
[52]
S. Li, K. Kallidromitis, A. Gokul, Y. Kato, and K. Kozuka, “Aligning diffusion models by optimizing human utility,” Advances in Neural Information Processing Systems, vol. 37, pp. 24897–24925, 2024.
[53]
W. Peebles and booktitle=Proceedings. of the I. international conference on computer vision Xie Saining, “Scalable diffusion models with transformers,” 2023, pp. 4195–4205.
[54]
Y.-L. Tsai et al., “Ring-a-bell! How reliable are concept removal methods for diffusion models?” 2024.
[55]
M. Bertran, S. Tang, M. Kearns, J. H. Morgenstern, A. Roth, and S. Z. Wu, “Reconstruction attacks on machine unlearning: Simple models are vulnerable,” Advances in Neural Information Processing Systems, vol. 37, pp. 104995–105016, 2024.
[56]
H. Gao, T. Pang, C. Du, T. Hu, Z. Deng, and booktitle=Proceedings. of the I. I. C. on C. V. Lin Min, “Meta-unlearning on diffusion models: Preventing relearning unlearned concepts,” 2025, pp. 2131–2141.

  1. We clarify that Figure 3 is a slight modification of figures from [28] and [29], adjusted to align the time steps with ours.↩︎

  2. https://github.com/christophschuhmann/improved-aesthetic-predictor↩︎

  3. https://huggingface.co/CompVis/ldm-celebahq-256↩︎

  4. https://github.com/jaehong31/SAFREE/blob/main/datasets/nudity-ring-a-bell.csv↩︎

  5. https://github.com/notAI-tech/NudeNet↩︎