0.0.1 Introduction↩︎

Generative models form a foundation of modern AI, and in particular, diffusion models have become a default backbone for image generation [@ho2020ddpm; @Song2020-kt; @Dhariwal2021-un; @Rombach2021-yj]. In recent years, generative models have increasingly been used not only after training from scratch, but also undergoing continual knowledge transfer, such as continual pretraining and fine-tuning in different domains. This brings continual learning (CL) of diffusion models to the foreground as a practical engineering problem and a fast-growing empirical literature [@Zajac2023-cldiffusion; @Smith2024-continualdiffusion; @GaoLiu2023-dr; @Masip2024-gd], Continual learning for diffusion models remains largely empirical: the mechanism of catastrophic forgetting is still poorly understood, and even basic questions that have been extensively studied in supervised models remain open– “which information of Task-1 will this model forget after training on Task 2?” and “which Task-1 samples should we replay to prevent it?”

Our approach to such fundamental questions is through Hopfield networks, which have played a central role in advancing the understanding and development of generative models [@hopfield1982neural; @krotov2021large]. A distinctive feature of Hopfield networks is that they admit an explicitly computable energy function. This enables systematic architecture design with the energy function serving as a guiding principle [@Hoover2023-mp; @dehmamy2025nrgpt], as well as the development of machine learning methods [@Zhang2022-vg; @Hofmann2024-hb; @scellier2017equilibrium]. In particular, modern Hopfield networks (MHNs) form a family of models that has been actively studied since the rise of deep learning and has been extended to deep learning architectures [@krotov2021large; @Ramsauer2021Hopfield]. Crucially, a growing literature identifies an underlying connection between MHNs and diffusion models: the score function of a Gaussian kernel-density estimator is exactly a Hopfield update [@Ambrogioni2024Associative; @Pham2025-kx], and backward diffusion dynamics can be read as gradient flows on an implicit log-sum-exp energy [@RayaAmbrogioni2023SSB; @Ambrogioni2025Thermo].

In this work, we reveal that the Hopfield energy serves as a theoretically grounded measure for quantitatively characterizing forgetting and prioritizing which samples to replay. Figure 1 illustrates the core idea of quantifying forgetting in generative models admitting the Hopfield energy, whose details we explain in Section 0.0.4. For such models, changes in generation probability can be tracked by observing the energy. By characterizing the forgetting of knowledge as a decrease in its generation probability, we can reduce the problem of quantifying forgetting to measuring an increase in the corresponding energy. Regions with densely clustered memory patterns form low energy, less sharp landscapes. Therefore, even if memory vectors shift from Task 1 to Task 2, the energy increase remains relatively small, so the corresponding states are robust to forgetting. By contrast, a Task-1 memory lying in a region with sparse or isolated memories undergoes a much larger energy increase and forgetting. This picture is analogous to the view that associates flat minima in parameter space with better generalization to test samples and out-of-distribution (OOD) samples [@keskar2017large; @cha2021swad].

Main contributions are summarized as follows:

• Hopfield energy view of forgetting supported by theoretical analysis (Theorem  1). We introduce intrinsic forgetting as the increase in Hopfield energy caused by a task transition. This provides an internal measure to quantify how the learned generative landscape changes, without relying solely on external classifiers or distributional metrics such as FID. To formalize the intrinsic forgetting shown in Figure 1, we analyze a modern Hopfield network with structured memories. The analysis shows that outlier-like memories, corresponding to the high-energy state and sharp basin, undergo a larger energy increase under a task perturbation than typical cluster memories.

• Analysis of energy-dependent replay (Theorem  2). We analyze how Hopfield energy controls the effect of memory replay. The analysis shows that high-energy samples can produce larger replay gains than low-energy samples, revealing an energy dependence in the effect of replay. This suggests that Hopfield energy may provide a useful signal for replay-buffer design. Controlled experiments with an MHN and its Boltzmann-machine counterpart support this energy-dependent replay picture.

• Empirical examination in diffusion models (Section  0.0.4.3,  0.0.5.3). We empirically confirm these theoretical predictions using two diffusion models in continual-learning settings: Stable Diffusion v1.5 fine-tuned on split CIFAR-10 and a pixel-space DDPM trained on the same one. In both settings, Hopfield energy tracks reconstruction-based forgetting, and replay experiments show energy-dependent changes in reconstruction error consistent with the MHN analysis.

Thus, we expect that this work will serve as a theoretical and empirical foundation for exploring forgetting and continual learning in modern generative models.

0.0.2 Related Work↩︎

0.0.2.0.0.1 Modern Hopfield networks.

Recent work on MHNs has developed various extensions of the classical Hopfield network [@hopfield1982neural], including energy functions beyond pairwise correlations [@Krotov2016-sr; @Demircigil2017-eh] and architectural generalizations [@krotov2021large]. In particular, [@Ramsauer2021Hopfield] showed that a Hopfield network with a log-sum-exp energy function corresponds to softmax attention in modern deep learning architectures. The term MHN is also commonly used to refer specifically to the model of [@Ramsauer2021Hopfield], and we follow this convention. A distinctive feature of MHNs is that they admit an explicit energy function of states. Recent studies have increasingly exploited this energy as an intrinsic model quantity, for example, to systematically design architectures from its gradient [@krotov2021large; @Hoover2023-mp], to detect out-of-distribution (OOD) samples [@Zhang2022-vg; @Hofmann2024-hb], and to derive biologically plausible algorithms [@scellier2017equilibrium]. In this work, we reveal that the energy also works as a useful indicator of forgetting in generative models.

0.0.2.0.0.2 Connection between MHNs and diffusion models.

[@RayaAmbrogioni2023SSB] showed that diffusion trajectories can be viewed as dynamics on an implicit potential landscape, with attractor-like behavior toward the data manifold. [@Ambrogioni2024Associative] further made the connection to MHNs explicit by showing that, for finite normalized patterns, the low-noise diffusion energy is asymptotically identical to the MHN energy. Recent work has more clarified this view [@Ambrogioni2025Thermo] and extended it to memorization/generalization behavior in diffusion models [@Ambrogioni2025Thermo; @Pham2025-kx]. Our work uses this established MHN–diffusion correspondence in a new setting: continual learning of generative models.

0.0.2.0.0.3 Continual learning in generative models.

Continual learning in generative models has mainly been studied from a practical perspective, with the aim of mitigating catastrophic forgetting, rather than from the viewpoint of theoretically understanding its mechanism. Early studies considered replay algorithms in Hopfield networks, while much of the recent work has focused on VAEs and GANs, including teacher–student generative modeling, memory-replay GANs, latent space consolidation, and distillation for conditional generation [@Ramapuram2020-lifelong; @Wu2018-mergan; @Zhai2019-lifelonggan; @Deja2021-qj]. More recently, continual learning has also been studied for diffusion models, including experimental demonstrations of forgetting and its mitigation by replay [@Zajac2023-cldiffusion], a regularization method tailored to diffusion models [@wang2025avoid], and parameter-efficient adaptation and distillation in text-to-image diffusion models [@Smith2024-continualdiffusion; @Masip2024-gd]. In contrast to these application-oriented studies, our work uses MHNs as a theoretical bridge to understand the mechanism of forgetting and to provide a basis for developing principled methods to mitigate forgetting.

0.0.3 Preliminaries↩︎

0.0.3.1 Modern Hopfield Energy

Let \(X=\{x_i\}_{i=1}^{N}\) be a set of stored memories and let \(\xi\in\mathbb{R}^d\) denote a state at which the memory landscape is evaluated. The continuous modern Hopfield energy is defined as \[E(\xi\mid X;\beta) = -\frac{1}{\beta}\log\sum_{i=1}^{N}\exp(\beta x_i^\top \xi) +\frac{1}{2}\|\xi\|^2 , \label{eq:mhn-energy}\tag{1}\] where \(\beta>0\) is an inverse-temperature parameter controlling the sharpness of the energy landscape [@Ramsauer2021Hopfield; @krotov2021large; @Lucibello2024-lo]. For \(\nabla_\xi E(\xi\mid X;\beta) =0\), its fixed points satisfy \[\xi = \sum_{i=1}^{N}p_i(\xi\mid X)x_i, \quad p_i(\xi\mid X) = \frac{\exp(\beta x_i^\top \xi)}{\sum_{j=1}^{N}\exp(\beta x_j^\top \xi)}. \label{eq:mhn-fixed-point}\tag{2}\] Throughout this paper, \(E(\xi\mid X)\) denotes the Hopfield energy evaluated with respect to the memory set \(X\), omitting \(\beta\) when it is clear from context. The energy is low when the state \(\xi\) lies near a cluster of memories, and high when it lies near an isolated memory. It is the scalar quantity we use below to characterize how the generative landscape changes across tasks.

0.0.3.2 Boltzmann Machine Formulation

Define the Boltzmann distribution of the MHN by \[p_\beta(\xi \mid X) = \exp(-\beta E(\xi \mid X;\beta))/Z, \label{eq:attnbm-distribution}\tag{3}\] where \(Z\) is the partition function. One can easily see that, for \(\|x_i\|=1\), \(Z\) is explicitly given by \(Z = N (2\pi/\beta)^{d/2}\), and this Boltzmann distribution is a Gaussian mixture \(p_\beta(\xi \mid X)= \sum_i \exp\left( -\frac{\beta}{2}\|\xi - x_i\|^2 \right)/Z.\) In the following section, as one of our synthetic experiments, we perform unsupervised learning of an MHN by maximum-likelihood estimation of this Boltzmann distribution. Specifically, we regard this distribution as a Boltzmann machine (BM) with a trainable weight matrix \(X\), and refer to it as MHN-BM. Since \(Z\) is analytically tractable, the model can be trained easily by gradient descent for maximum-likelihood learning. Note that a similar Boltzmann machine counterpart of MHNs was investigated by [@OtaKarakida2023-attnbm], but their treatment of the inverse temperature differs from our focus.

0.0.3.3 Diffusion Models and Implicit Energy

Diffusion models are not usually written as explicit energy-based models, but their score can be viewed as the negative gradient of an implicit, time-dependent energy. Let \(p_s(x)\) be the noisy data marginal at diffusion time \(s\), and write \[E_{\mathrm{DM},s}(x):=-\log p_s(x), \qquad \nabla_x\log p_s(x)=-\nabla_x E_{\mathrm{DM},s}(x).\] Thus, roughly speaking, reverse sampling moves a noisy sample \(x_s\) by noisy descent on \(E_{\mathrm{DM},s}\); under conservative-drift assumptions, this energy also includes the drift potential [@RayaAmbrogioni2023SSB]. Its stable equilibria therefore act as attractors for generation and memory recall [@Ambrogioni2024Associative; @Pham2025-kx]. For a finite set of patterns corrupted by additive Gaussian noise, \[p_s(x) \propto \sum_i \exp\!\left( -\|x-y_i\|^2/2\tau_s^2 \right).\] Here \(\tau_s>0\) denotes the standard deviation of the Gaussian corruption at diffusion time \(s\). Expanding the square shows that, for normalized memories, \(E_{\mathrm{DM},s}\) reduces up to constants and positive rescaling to \[E_{DM, s} (x) \propto -\frac{1}{\beta_s} \log\sum_{i=1}^{N}\exp(\beta_s y_i^\top x) + \frac{1}{2}\|x\|^2, \qquad \beta_s=\tau_s^{-2},\] which is the modern Hopfield energy in Eq. (1 ). We therefore use Hopfield energy as a tractable proxy for the implicit diffusion energy landscape. A more explicit derivation, including the variance-preserving case, is given in Appendix 0.0.7.

0.0.4 Energy as an Indicator of Forgetting↩︎

We first introduce forgetting with the internal Hopfield energy. Defining “forgetting” in an unsupervised generative model is subtle: label-based metrics via external classifiers are heuristic measures [@Zajac2023-cldiffusion; @Smith2024-continualdiffusion], but the generative objective of our interest is essentially unsupervised and class information is not necessarily relevant to the input data. We need an internal, model-native measure applicable to pure unsupervised settings.

0.0.4.0.0.1 Intrinsic forgetting via Hopfield energy.

Let \(\mathcal{X}_{\mathrm{old}}\) and \(\mathcal{X}_{\mathrm{new}}\) denote the stored memory set at task 1 and task 2. For any state \(x\), we define the intrinsic forgetting at \(x\) as \[\Delta E(x) = E(x\mid\mathcal{X}_{\mathrm{new}}) - E(x\mid\mathcal{X}_{\mathrm{old}}). \label{eq:internal-forgetting}\tag{4}\] Large positive \(\Delta E(x)\) means that the energy landscape at \(x\) has risen, so the model assigns a smaller probability to that state after the task transition. Under the assumption that the score function has sufficient expressive capacity to approximate the data distribution of each task nearly perfectly, this quantity captures forgetting induced by the intrinsic and unavoidable distributional differences between tasks. It is noteworthy that, for a fixed number of memories, \(p_\beta(\xi\mid\mathcal{X}_{\mathrm{new}})/ p_\beta(\xi\mid\mathcal{X}_{\mathrm{old}}) =\exp[-\beta\Delta E(\xi)]\). Thus, intrinsic forgetting naturally arises when distributional changes between tasks are measured in terms of probability density ratios.

0.0.4.1 Theoretical analysis: equal-angular cluster plus outlier under small rotation

We derived an analytical formulation of intrinsic forgetting within a toy setting.

0.0.4.1.0.1 Setup (Fig. 2).

Consider memory vectors (corresponding to training samples) \(x_1,\dots,x_N,x_o\in\mathbb{S}^{d-1}\) satisfying \[\begin{align} x_i^\top x_j=c\;(i\neq j;\;i,j\le N),\qquad x_i^\top x_o=0\;(i\le N),\qquad 0<c<1, \label{eq:eqang-setup} \end{align}\tag{5}\] so \(\{x_i\}_{i\le N}\) is an equal-angular cluster and \(x_o\) is the orthogonal outlier. A canonical realization in \(d\gg 1\) is the latent-Gaussian construction \(x_i=\sqrt{c}\,\mu+\sqrt{1-c}\,z_i\) with a fixed unit centroid \(\mu\) and i.i.d.unit-variance Gaussian residuals \(z_i\perp\mu\), for which \(\|x_i\|^2=1\) and \(x_i^\top x_j=c+O(d^{-1/2})\) concentrate at the desired geometry; the results below depend only on (5 ). Task 1 stores \(X=\{x_1,\dots,x_N,x_o\}\); Task 2 is provided by its rational domain shift: \[X_2=VX, \quad \text{with} \;V=\exp(\varepsilon\Omega), \quad \Omega=(W-W^\top)/2. \label{eq:rotation}\tag{6}\]

Note that the rotation identity \(E(\xi\mid VX)=E(V^\top\xi\mid X)\) converts a domain shift into a change of argument at the original landscape. This transformation facilitates our analysis. In the analysis, we assume a sufficiently small random rotation with \(W_{ij}\stackrel{\mathrm{iid}}{\sim}\mathcal{N}(0,1)\) and \(\varepsilon\ll 1\). In addition, we work at finite \(N\), large \(\beta\), where individual memories are still fixed points [@Lucibello2024-lo; @Achilli2025-cm].

0.0.4.1.0.2 Fixed points nearby each memory vector.

Let \(\xi_{cl}^*,\xi_o^*\) denote the cluster-side and outlier-side Task-1 fixed points. By the symmetry of (5 ), the softmax weights at each Task-1 fixed point reduce to scalar self-consistency conditions whose large-\(\beta\) asymptotics (proved in Appendix 0.0.8) give the closed-form expansions \[\xi_o^*\sim x_o+e^{-\beta}(s-Nx_o),\quad \xi_{cl}^* \sim x_{cl}+e^{-\beta(1-c)}\!\big(\textstyle\sum_{j\ge 2}x_j-(N-1)x_{cl}\big), \label{eq:fp-asym}\tag{7}\] with \(s:=\sum_{i\le N}x_i\). For clarity, here and below we use the informal shorthand “\(\sim\)” to show only the leading-order term; the precise lower-order terms and error bounds are given in Appendix 0.0.8 (Eqs. (?? ), (?? ) for the fixed points and Eqs. (67 )–(68 ) for the energy rises).

The dominant off-memory weights are therefore \(e^{-\beta(1-c)}\) on the cluster branch and \(e^{-\beta}\) on the outlier branch, with \(e^{-\beta(1-c)}\gg e^{-\beta}\) for \(0<c<1\). Geometrically, the cluster fixed point is supported by \(N-1\) near-neighbors at cosine \(c\), while the outlier sees only orthogonal competitors. Appendix 0.0.9 shows that these same off-memory weights also control the local energy level and the local sharpness of the basin, as measured by the Hessian trace of the energy landscape. This justifies the shorthand used below: the cluster branch is low-energy and broad, whereas the outlier branch is high-energy and sharp.

0.0.4.1.0.3 Intrinsic forgetting at the fixed points.

For a Task-1 fixed point \(\xi_*\), write \(\langle\,\cdot\,\rangle_V:=\mathbb{E}_{\Omega}[\,\cdot\,]\) for the expectation over the small-rotation ensemble of (6 ). We absorb this average into the definition and express the (rotation-averaged) intrinsic forgetting at \(\xi_*\) as \[\Delta E_*^{\mathrm{fp}}\;:=\;\big\langle E(\xi_*\mid VX)-E(\xi_*\mid X)\big\rangle_V. \label{eq:DEfp-def}\tag{8}\] Since \(\xi_*\) is stationary, Taylor expansion combined with the rotation identity gives, at leading order, \[\Delta E_*^{\mathrm{fp}}\;\sim\;\tfrac{\varepsilon^2}{4}\big(\|\xi_*\|^2\,\mathrm{tr}\,H_*-\xi_*^\top H_*\xi_*\big),\qquad H_*=I-\beta\Big(\textstyle\sum_i p_i^* x_ix_i^\top-\xi_*\xi_*^\top\Big). \label{eq:delta-hessian-body}\tag{9}\] Writing the off-memory weights as \(p_k^*=1-\sum_{j\neq k}\eta_j\), \(p_j^*=\eta_j\ll 1\) (with \(\alpha_j:=x_j^\top x_k\)) gives the following: \[\Delta E_*^{\mathrm{fp}}\;\sim\;\tfrac{\varepsilon^2}{4}\Big[(d-1)-\sum_{j\neq k}\eta_j\,\Lambda(\alpha_j)\Big],\qquad \Lambda(\alpha):=2(d-1)(1-\alpha)+\beta(1-\alpha^2). \label{eq:delta-eta}\tag{10}\] Specializing 10 to the two representative Task-1 fixed points, namely the outlier (\(N\) competitors at \(\alpha=0\), \(\eta=e^{-\beta}\)) and the cluster (\(N-1\) competitors at \(\alpha=c\), \(\eta=e^{-\beta(1-c)}\) plus one competitor at \(\alpha=0\), \(\eta=e^{-\beta}\)), we obtain the following.

Proof sketch. \(e^{-\beta(1-c)}\gg e^{-\beta}\) at large \(\beta\) and \(\Lambda(c)>0\) for \(0<c<1\), so the first term dominates. The genuine-cluster regime \(0<c<1\) is essential: at \(c=0\) both branches coincide at leading order, while \(c<0\) can flip the sign of (11 ). The precise lower-order terms and the full derivation are in Appendix 0.0.8.\(\square\)

0.0.4.2 Synthetic verification in MHN and Boltzmann-machine learning

We verify Theorem 1 in synthetic cluster-plus-outlier systems. The closed-form MHN uses an equal-angular cluster and an orthogonal outlier with \(N=12,d=50,c=0.35,\beta=8,\varepsilon=0.05\), averaged over 2000 random rotations. The outlier fixed point has a larger rotation-induced energy rise than the cluster fixed points, as predicted by Theorem 1; details are in Appendix 0.0.11.1. The same ordering also appears after likelihood training in the MHN-BM with \(N=12,d=50,c=0.35,\beta=16,\varepsilon_{\mathrm{rot}}=0.02\), shown in Fig. 3 (a).

0.0.4.3 Application to Diffusion models

We next examine whether the prediction of Theorem 1 extends beyond the tractable MHN setting to continual learning in diffusion models. The main text focuses on incremental class learning with split CIFAR-10, which is a standard benchmark for continual learning. We also report results for domain-incremental learning with rotated MNIST in Appendix 0.0.12. This setting most closely matches the rotation-based analysis in Sections 0.0.4.1 and 0.0.5.1, and yields the same qualitative conclusions.

We evaluate two practically relevant diffusion-model systems on split CIFAR-10: latent-space Stable Diffusion v1.5 fine-tuned on the task split (Fig. 3 (b)), and a pixel-space DDPM trained from scratch on the same split (Fig. 3 (c)). Following prior work [@RayaAmbrogioni2023SSB; @Ambrogioni2024Associative; @Pham2025-kx], we define the per-sample Hopfield energy of a diffusion model using Eq. (1 ), with the stored memories \(\{x_i\}\) given by the Task-1 training images.

0.0.4.3.0.1 Reconstruction metric for forgetting.

In the diffusion-model experiments, we additionally use a reconstruction-based measure of forgetting, which evaluates how well a Task-1 sample can still be recovered by the model after Task-2 training. Specifically, we corrupt a Task-1 sample \(x\) to a fixed diffusion timestep \(t^\star\), run the post-Task-2 reverse process from \(t^\star\) back to \(0\), and measure \[\mathcal{F}_{t^\star}(x) = \|x-\hat{x}_{0,t^\star}\|_2^2 . \label{eq:reconstruction-forgetting}\tag{12}\] A larger \(\mathcal{F}_{t^\star}(x)\) indicates that the post-Task-2 reverse dynamics is less able to reconstruct the original Task-1 sample. We use this heuristic metric to examine whether the energy-based tendency predicted by intrinsic forgetting is also reflected in the behavior of trained diffusion models. Details of this metric and robustness checks across other timesteps are given in Appendix 0.0.13. We also report a perturbation-based sharpness analysis on CIFAR-10 in Appendix 0.0.9.4, showing how local sharpness relates to both Hopfield energy and reconstruction-based forgetting.

0.0.4.3.0.2 Stable Diffusion on split CIFAR-10 (Fig. 3 (b)).

We fine-tune Stable Diffusion v1.5 (runwayml/stable-diffusion-v1-5, VAE frozen, only the UNet trained) on a split CIFAR-10 protocol where each task is composed of different 5 classes. After Task 1 we assign every Task-1 image a scalar Hopfield energy \(E_\theta(x)\) computed via the log-sum-exp expression (1 ) over all training set of Task-1 images. This is our operationalization of the diffusion model’s intrinsic energy at \(x\). After Task 2 training, we measure the forgetting by the per-sample reconstruction MSE at noise level \(t=0.8\). This is the main diffusion-model instantiation of Theorem 1.

0.0.4.3.0.3 Pixel-space diffusion (Fig. 3 (c)).

To check that the relation is not specific to the latent space learned by Stable Diffusion, we repeat the same protocol with a DDPM trained directly in pixel space on \(32\times 32\) split CIFAR-10 from scratch. The same energy-vs-forgetting ordering is reproduced: per-sample reconstruction MSE after Task 2 is a monotone-increasing function of the Task-1 Hopfield energy. Together, Fig. 3 (b) and Fig. 3 (c) show that Hopfield energy is a model-agnostic, sample-level forgetting indicator for diffusion CL.

0.0.5 Energy-guided replay prioritization↩︎

Given that high-energy memories are forgotten stronger, the natural question is whether they should be replayed more. We answer this affirmatively with a second theorem, then turn it into a selection rule of replay samples.

0.0.5.1 Theoretical analysis: baseline-subtracted replay gain

We add one Task-1 replay sample \(x_r\) to the rotated Task-2 dataset, \(X_2^{+r}(V):=VX\cup\{x_r\}\), with the replayed sample counted as an additional term in the log-sum-exp. Define the raw replay gain at the Task-1 fixed point \(\xi_k^*\) as \[\mathcal{R}_{r\to k}(V)\;:=\;E(\xi_k^*\mid X_2(V))-E(\xi_k^*\mid X_2^{+r}(V)). \label{eq:raw-gain-def}\tag{13}\] Writing \(\Delta E_k^{\mathrm{fp}}(V):=E(\xi_k^*\mid X_2(V))-E(\xi_k^*\mid X)\) for the intrinsic forgetting (8 ), the gain decomposes as \[\mathcal{R}_{r\to k}(V) =\underbrace{\Delta E_k^{\mathrm{fp}}(V)}_{\text{forgetting w/o replay}} \;-\;\underbrace{\bigl[E(\xi_k^*\mid X_2^{+r}(V))-E(\xi_k^*\mid X)\bigr]}_{\text{forgetting w/ replay}},\] so \(\mathcal{R}_{r\to k}(V)\) measures the reduction of intrinsic forgetting due to replay. With the softmax weight \(p_{r\mid k}=e^{\beta x_r^\top\xi_k^*}/\sum_{\mu\in\mathcal{I}}e^{\beta x_\mu^\top\xi_k^*}\), the log-sum-exp form of \(E\) gives \(\mathcal{R}_{r\to k}(V)=\tfrac{1}{\beta}\log(1+p_{r\mid k}\,e^{\beta\Delta E_k^{\mathrm{fp}}(V)})\), of which \(\mathcal{R}_{r\to k}(I)=\tfrac{1}{\beta}\log(1+p_{r\mid k})\) is the trivial duplicate-memory baseline. Subtracting this baseline and Taylor-expanding in \(\Delta E_k^{\mathrm{fp}}(V)=O(\varepsilon^2)\) gives, with \(\rho_{r\mid k}:=p_{r\mid k}/(1+p_{r\mid k})\) (full derivation in Appendix 0.0.10), \[\Delta_{r\to k}\;:=\;\big\langle\mathcal{R}_{r\to k}(V)-\mathcal{R}_{r\to k}(I)\big\rangle_V\;\sim\;\rho_{r\mid k}\,\Delta E_k^{\mathrm{fp}}. \label{eq:replay-master}\tag{14}\] We call \(\rho_{r\mid k}\) a replay susceptibility because it is the linear response of the baseline-subtracted replay gain to the energy rise \(\Delta E_k^{\mathrm{fp}}\). Replay gain \(=\) replay susceptibility \(\times\) rotation-induced energy rise. The first factor asks how much softmax weight the replay point already has at the target fixed point (geometry); the second is exactly the rotation-averaged rise \(\Delta E_k^{\mathrm{fp}}\) produced by Theorem 1 (drift). Let \(x_{\mathrm{cl}_1}\) be a cluster memory and let \(x_{\mathrm{cl}_2}\neq x_{\mathrm{cl}_1}\) be another member of the same cluster. The replay susceptibilities satisfy \[\rho_{\mathrm{cl}_1\mid \mathrm{cl}_1},\,\rho_{o\mid o}\sim \tfrac12,\qquad \rho_{\mathrm{cl}_1\mid \mathrm{cl}_2}\sim e^{-\beta(1-c)},\qquad \rho_{\mathrm{cl}_1\mid o},\,\rho_{o\mid \mathrm{cl}_1}\sim e^{-\beta}.\] The precise lower-order corrections are in Appendix 0.0.10 (Eqs. (93 ), (94 )). Combining these with the inequality of \(\Delta E_k^{\mathrm{fp}}\) in Theorem 11 , we obtain the following.

Intuition. Equation (14 ) says that a replay sample reduces forgetting in proportion to its softmax weight at the target fixed point. The first inequality, \(\Delta_{o\to o}>\Delta_{\mathrm{cl}_1\to\mathrm{cl}_1}\), compares self-replay: both have susceptibility \(\rho\simeq 1/2\), but the outlier target has the larger energy rise by Theorem 1, so high-energy self-replay gives the largest per-sample rescue. The gap \(\Delta_{\mathrm{cl}_1\to\mathrm{cl}_1}\gg\Delta_{\mathrm{cl}_1\to\mathrm{cl}_2}\) says that replacing self-replay by another cluster member loses the small overlap factor \(e^{-\beta(1-c)}\), although it can still help nearby low-energy states. Finally, \(\Delta_{\mathrm{cl}_1\to\mathrm{cl}_2}\gg\{\Delta_{\mathrm{cl}_1\to o},\Delta_{o\to\mathrm{cl}_1}\}\) and \(\Delta_{\mathrm{cl}_1\to o}\asymp\Delta_{o\to\mathrm{cl}_1}\) mean that cross-type replay between the cluster and the outlier is exponentially weaker, because their softmax overlap is only \(e^{-\beta}\). Thus high-energy replay gives a tall-but-narrow rescue, while low-energy cluster replay gives a shallower-but-broader rescue over neighboring low-energy states.

0.0.5.2 Synthetic verification of replay ordering

We verify Theorem 2 on the same synthetic systems. In the closed-form MHN with \(N=12,d=50,c=0.35,\beta=8,\varepsilon=0.05\), the replay-gain shows the predicted hierarchy: outlier self-replay is largest, cluster self-replay is next, within-cluster cross replay is smaller, and cross-type replay between the cluster and the outlier is weakest. The full result is reported in Appendix 0.0.11.2. The trained MHN-BM with \(N=12,d=50,c=0.35,\beta=16,\varepsilon_{\mathrm{rot}}=0.02\) shows the almost same qualitative ordering in Fig. 4.

0.0.5.3 Application to Diffusion models

Theorem 2 delivers a concrete selection rule of replay samples: among Task-1 samples, replay those that maximize their own energy (thereby inheriting a large \(\Delta E_k^{\mathrm{fp}}\)) and that self-replay (so that \(\rho_{k\mid k}\approx 1/2\)). The simplest realization is energy-guided top-\(K\) replay.

0.0.5.3.0.1 Replay buffer.

We pick the top-\(K\) highest-energy samples of Task 1 as the buffer, \(\mathcal{B}=\mathrm{TopK}_{x\in\mathcal{D}_1}E_\theta(x)\), with \(K=5000\) in all experiments. During Task-2 fine-tuning we augment each mini-batch with replay samples drawn from \(\mathcal{B}\), so that the effective training set is \(\mathcal{D}_2\cup\mathcal{B}\) (25,000 new + 5,000 replay images per epoch). We compare against random replay (uniform top-\(K\)) and no replay baselines.

We again focus on the two diffusion-model systems of Section 0.0.4.3 as the main empirical test.

0.0.5.3.0.2 Stable Diffusion (Fig. 5).

On split CIFAR-10, we compare three replay policies with identical buffer size \(K=5000\): energy-guided top-\(K\) (the \(K\) highest-energy Task-1 images), random, and energy-guided bottom-\(K\) (the \(K\) lowest-energy images). After Task-2 fine-tuning, per-sample reconstruction MSE at \(t=0.8\) is measured for every Task-1 image and binned by energy.

Eq. (14 ) predicts a mean–range tradeoff in the replay effect: \(\Delta_{r\to k}\sim \rho_{r\mid k}\,\Delta E_k^{\mathrm{fp}} .\) The energy-rise term \(\Delta E_k^{\mathrm{fp}}\) makes the per-sample replay gain larger for high-energy targets, whereas the relevance factor \(\rho_{r\mid k}\) determines how broadly a replay sample affects other targets. Thus, high-energy replay is expected to produce a large but localized reduction in the high-energy tail, while low-energy replay is expected to produce a smaller reduction spread over a broader set of low-energy samples. This is what we observe: the high-energy buffer (Fig. 5 (a)) yields a sharply peaked bin-mean reduction at high energy (Fig. 5 (c)), but its bin-sum reduction is confined to a narrow energy range (Fig. 5 (d)). In contrast, the low-energy buffer (Fig. 5 (b)) gives a smaller bin-mean reduction at low energy, but the affected range is broader, so the bin-sum reduction extends across a wider energy interval. This complementary pattern is consistent with the replay ordering in Eq. (15 ).

0.0.5.3.0.3 Pixel-space diffusion (Fig. 6).

We repeat the three-policy comparison with the pixel-space DDPM of Section 0.0.4.3. Every qualitative conclusion from Stable Diffusion is reproduced: (i) high-energy replay protects high-energy images, (ii) low-energy replay only helps low-energy images, (iii) aggregated reductions place energy-guided top-\(K\) above random above low-energy. The replay benefit is again concentrated in the high-energy tail, as predicted. Together, Fig. 5 and Fig. 6 constitute our main empirical case that Hopfield energy is a practical, diffusion-agnostic replay criterion.

0.0.6 Conclusion and Limitations↩︎

Through theoretical analyses of modern Hopfield networks and empirical confirmation on diffusion models, we find that Hopfield energy serves as an indicator of forgetting and also provides a criterion for selecting replay samples. These results suggest that forgetting in unsupervised generative models can be evaluated not only through external evaluation metrics, but also through the geometry of the learned state landscape. We expect that this work will serve as a theoretical and practical foundation for future studies of knowledge forgetting, replay analysis, and energy-based diagnostics of generative models.

0.0.6.0.0.1 Limitations.

(i) Forgetting through the learning of score functions. We consider intrinsic forgetting, that is, an unavoidable factor that arises when a generative model perfectly approximates task-dependent input distributions. In practical diffusion models, however, the score function is represented by a neural network, and additional forgetting may appear even for similar inputs. In supervised learning, theoretical analyses of such forgetting have been developed [@lee2022maslow], and extending the analyses to the score function is an important direction for future work. (ii) Finite sample sizes. Although our theory can handle structured data, it is currently restricted to the case of finitely many memories. Quantifying forgetting in regimes where the number of memories grows with the input dimension, and where the memories encode richer data manifolds [@Achilli2025-cm], remains an open problem. (iii) Broader experimental validation. For future practical development, it would also be useful to examine scenarios not covered in this study, such as domains beyond images and online continual learning without explicit task boundaries. A more elaborated indicator of internal states (e.g., probability-flow ODE marginals) could tighten the correlation of Fig. 3 (b), 3 (c).

Appendices↩︎

0.0.7 Diffusion–Hopfield Correspondence↩︎

We summarize the correspondence between diffusion energies and modern Hopfield energies. Consider a forward noising SDE \[dY_s=f(Y_s,s)\,ds+g(s)\,d\widehat W_s,\] and let \(p_s(x)\) be the marginal density of \(Y_s\). For additive noise and conservative drift, the reverse-time SDE can be written as \[dX_t = \left[ g^2(T-t)\nabla_x\log p_{T-t}(X_t) - f(X_t,T-t) \right]dt + g(T-t)dW_t .\] Equivalently, \[dX_t=-\nabla_x E_{T-t}(X_t)\,dt+g(T-t)dW_t,\] where the implicit energy is \[E_s(x) = -g^2(s)\log p_s(x) + \int_0^x f(z,s)\cdot dz ,\] up to an additive constant [@RayaAmbrogioni2023SSB]. To examine the attractor structure at a fixed diffusion time \(s\), we freeze the time-dependent coefficients \(g(s)\) and \(f(\cdot,s)\) and study the deterministic drift part of the reverse SDE. That is, we drop only the Brownian increment \(g(s)dW_t\), while keeping the \(g^2(s)\nabla_x\log p_s\) term in the drift. Under the conservative-drift assumption above, \(f(\cdot,s)=\nabla\Phi_s(\cdot)\) for a scalar potential \(\Phi_s\), so the line integral in \(E_s\) is path independent. Along the deterministic flow \[\frac{d x_\lambda}{d\lambda} = -\nabla_x E_s(x_\lambda),\] the energy decreases as \[\frac{d}{d\lambda}E_s(x_\lambda) = -\|\nabla_x E_s(x_\lambda)\|^2 \le 0,\] so stable equilibria satisfying \(\nabla_x E_s(x)=0\) define the attractor structure of the reverse dynamics.

Now assume that the data distribution is the empirical distribution over a finite set of patterns, \[p_0(y)=\frac{1}{M}\sum_{i=1}^{M}\delta(y-y_i).\] In the variance-exploding case, \[dY_s=g(s)d\widehat W_s, \qquad f=0, \qquad \tau_s^2=\int_0^s g^2(r)\,dr .\] Thus \[Y_s\mid y_i\sim\mathcal{N}(y_i,\tau_s^2I),\] and the noisy marginal is the Gaussian mixture \[p_s(x) \propto \sum_i \exp\!\left( -\frac{\|x-y_i\|^2}{2\tau_s^2} \right).\] Expanding the square gives \[p_s(x) \propto \exp\!\left(-\frac{\|x\|^2}{2\tau_s^2}\right) \sum_i \exp\!\left( \tau_s^{-2}y_i^\top x - \frac{\|y_i\|^2}{2\tau_s^2} \right).\] When the memories are normalized, \(\|y_i\|=r\), the last term is independent of \(i\) and can be absorbed into the normalization. Since \(f=0\), we have \(E_s(x)=-g^2(s)\log p_s(x)\). Multiplying by the positive factor \(\tau_s^2/g^2(s)\), which does not change the equilibria, yields \[-\tau_s^2\log p_s(x) = -\tau_s^2 \log\sum_i \exp(\tau_s^{-2}y_i^\top x) + \frac{1}{2}\|x\|^2 + \mathrm{const}.\] With \(\beta_s=\tau_s^{-2}\), this becomes \[E_{\mathrm{MHN}}(x;\beta_s) = -\frac{1}{\beta_s} \log\sum_{i=1}^{M} \exp(\beta_s y_i^\top x) + \frac{1}{2}\|x\|^2 + \mathrm{const},\] which is the continuous modern Hopfield energy [@Ambrogioni2024Associative]. Thus, for discrete normalized patterns, the Gaussian-mixture marginal induced by the forward diffusion defines the same log-sum-exp energy landscape as a modern Hopfield network.

For variance-preserving diffusions, the forward SDE is typically \[dY_s=-\frac{1}{2} b(s)Y_s\,ds+\sqrt{b(s)}\,d\widehat W_s .\] The transition kernel is still Gaussian, \[Y_s\mid y_i \sim \mathcal{N}(\theta_s y_i,\tau_s^2 I), \qquad \theta_s= \exp\!\left( -\frac{1}{2}\int_0^s b(r)\,dr \right), \qquad \tau_s^2=1-\theta_s^2 .\] Therefore \(p_s\) is again a Gaussian mixture, now centered at the shrunk memories \(\theta_s y_i\). Substituting this marginal into the implicit energy gives, for normalized memories, \[\frac{E_s(x)}{b(s)} = -\log\sum_i \exp\!\left( \frac{\theta_s}{\tau_s^2}y_i^\top x \right) + \left( \frac{1}{2\tau_s^2} - \frac{1}{4} \right)\|x\|^2 + \mathrm{const}.\] Thus the variance-preserving case differs at finite noise by pattern shrinkage and time-dependent coefficients. In the low-noise regime, \(\theta_s\to1\) and \(\tau_s\to0\), this reduces to the same log-sum-exp plus quadratic structure, with effective inverse temperature proportional to \(\theta_s/\tau_s^2\). More generally, for additive SDEs with conservative drift, the low-noise transition kernel is locally Gaussian around each pattern, so the same Gaussian-mixture argument gives the Hopfield fixed-point structure asymptotically [@Ambrogioni2024Associative].

0.0.8 Full derivation of Theorem 1↩︎

We organize the proof as follows. Subsection 0.0.8.1 states the equal-angular cluster-plus-outlier model and the rotation identity. Subsection 0.0.8.2 reduces the fixed-point equations by symmetry. Subsection 0.0.8.3 separates the existence of pattern-specific fixed points from the stronger localization condition used to control leakage corrections. Subsection 0.0.8.4 derives the large-\(\beta\) fixed-point asymptotics. Subsection 0.0.8.5 collects the Hessian calculation, the rotation average, and the first-order expansion in the off-memory weights that produce the formula for \(\Delta E_*^{\mathrm{fp}}\). Finally, Subsection 0.0.8.6 specializes that formula to the outlier and cluster branches and proves Theorem 1.

0.0.8.1 Problem setup and rotation identity

Let \(x_1,\dots,x_N,x_o\in\mathbb{S}^{d-1}\) satisfy (5 ); we assume \(0<c<1\) and that the ambient dimension \(d\) is large enough for the configuration to exist. A canonical large-\(d\) realization is the latent-Gaussian construction \(x_i=\sqrt{c}\,\mu+\sqrt{1-c}\,z_i\) with a fixed unit centroid \(\mu\in\mathbb{S}^{d-1}\) and i.i.d.Gaussian residuals \(z_i\perp\mu\) of unit variance, for which \(\|x_i\|^2=1\) and \(x_i^\top x_j=c+O(d^{-1/2})\) concentrate at the equal-angular geometry; the analysis below depends only on (5 ) and applies to any realization. The MHN energy and update rule are \[E(\xi\mid X)=-\tfrac{1}{\beta}\log\sum_{i=1}^{N+1}e^{\beta x_i^\top\xi}+\tfrac12\|\xi\|^2,\qquad T(\xi)=\sum_i p_i(\xi)x_i,\qquad p_i(\xi)=\frac{e^{\beta x_i^\top\xi}}{\sum_k e^{\beta x_k^\top\xi}}, \label{eq:app-E-T}\tag{16}\] so that a fixed point \(\xi_*=T(\xi_*)\) satisfies \(\xi_*=Xp_*\) with \(p_*=\mathrm{softmax}(\beta X^\top\xi_*)\). The rotation \(X_2=VX\) induces \[E(\xi\mid VX)=E(V^\top\xi\mid X), \label{eq:app-rotation-identity}\tag{17}\] reducing Task-2 evaluation at \(\xi\) to Task-1 evaluation at \(V^\top\xi\).

0.0.8.2 Symmetry reduction of the fixed-point equations

Define \[s:=\sum_{i=1}^N x_i,\qquad A:=1+(N-1)c. \label{eq:app-s-A}\tag{18}\] A direct calculation using (5 ) gives \[x_i^\top s=A\quad(i\le N),\qquad x_o^\top s=0,\qquad \|s\|^2=NA. \label{eq:app-s-properties}\tag{19}\]

0.0.8.2.0.1 Outlier-like branch.

The outlier-like fixed point is invariant under permutations of the cluster members, so its softmax weights are \[p^{\mathrm{out}}=(u,\dots,u,v),\qquad Nu+v=1, \label{eq:app-p-out}\tag{20}\] and \(\xi_{\mathrm{out}}=u\,s+v\,x_o\). The logits read \(x_i^\top\xi_{\mathrm{out}}=Au\) for \(i\le N\) and \(x_o^\top\xi_{\mathrm{out}}=v\). Substituting into the softmax, \[u=\frac{e^{\beta Au}}{Ne^{\beta Au}+e^{\beta v}},\qquad v=\frac{e^{\beta v}}{Ne^{\beta Au}+e^{\beta v}}, \label{eq:app-out-softmax}\tag{21}\] and taking the log-ratio together with \(u=(1-v)/N\) yields the scalar self-consistency \[\log\frac{Nv}{1-v}=\beta\Big(v-\frac{A}{N}(1-v)\Big). \label{eq:app-out-scalar}\tag{22}\]

0.0.8.2.0.2 Cluster-like branch near \(x_1\).

Permuting \(x_2,\dots,x_N\) leaves the cluster-like fixed point invariant, so \[p^{\mathrm{cl}}=(a,b,\dots,b,q),\qquad a+(N-1)b+q=1, \label{eq:app-p-cl}\tag{23}\] and \(\xi_{\mathrm{cl}}=a\,x_1+b\sum_{j\ge 2}x_j+q\,x_o\). The three distinct logits are \[z_1=a+(N-1)cb,\qquad z_-=ca+\{1+(N-2)c\}b,\qquad z_o=q. \label{eq:app-z-cl}\tag{24}\] The softmax equations and the identity \(z_1-z_-=(1-c)(a-b)\) give the two pattern-selection equations \[\log\frac{a}{b}=\beta(1-c)(a-b),\qquad \log\frac{a}{q}=\beta(z_1-q). \label{eq:app-cl-ratios}\tag{25}\] The first is a Curie–Weiss-type fixed-point equation with effective inverse temperature \[\beta_{\mathrm{eff}}:=\beta(1-c), \label{eq:app-beta-eff}\tag{26}\] matching the equal-angular capacity scale of [@Ramsauer2021Hopfield; @Achilli2025-cm].

0.0.8.3 Existence threshold and localization condition

We separate two requirements: (i) existence of pattern-specific cluster fixed points, and (ii) the stronger localization condition that keeps the leakage corrections small in the Hessian expansion below.

0.0.8.3.1 Existence threshold

Drop the outlier coordinate temporarily and consider the symmetric cluster sector \(p^{\mathrm{cl}}=(a,b,\dots,b)\) with \(a+(N-1)b=1\). Introduce the order parameter \[m:=a-b,\qquad a=\frac{1+(N-1)m}{N},\qquad b=\frac{1-m}{N}. \label{eq:app-m-def}\tag{27}\] The pattern-selection equation (25 ) becomes \[\log\frac{1+(N-1)m}{1-m}=\beta_{\mathrm{eff}}\,m,\qquad \beta_{\mathrm{eff}}=\beta(1-c). \label{eq:app-m-equation}\tag{28}\] The function on the left is convex in \(m\in(0,1)\), so (28 ) admits a non-trivial root iff the line of slope \(\beta_{\mathrm{eff}}\) touches the convex graph. Tangency at some \(m_c\in(0,1)\) defines the critical inverse temperature \[\beta_c(N):=\min_{m\in(0,1)}\frac{1}{m}\log\frac{1+(N-1)m}{1-m}, \label{eq:app-beta-c-def}\tag{29}\] characterised by \[\log\frac{1+(N-1)m_c}{1-m_c}=\frac{N\,m_c}{(1-m_c)\{1+(N-1)m_c\}},\qquad \beta_c(N)=\frac{N}{(1-m_c)\{1+(N-1)m_c\}}. \label{eq:app-mc-tangency}\tag{30}\] Equivalently, with \(t_c:=\{1+(N-1)m_c\}/(1-m_c)\), \[N\,t_c\log t_c=(t_c-1)(t_c+N-1),\qquad \beta_c(N)=\frac{(t_c+N-1)^2}{N\,t_c}. \label{eq:app-tc-equation}\tag{31}\] The closed form is implicit; the asymptotics is explicit. Elementary bounds on \(\log t_c\) yield \[\log N+1\le \beta_c(N)\le \log N+\log(1+\log N)+1+\frac{\log(1+\log N)}{\log N}, \label{eq:app-betac-bounds}\tag{32}\] so \(\beta_c(N)=\Theta(\log N)\), and as \(N\to\infty\), \[\beta_c(N)=\log N+\log\log N+1+o(1). \label{eq:app-betac-asym}\tag{33}\] Pattern-specific cluster fixed points therefore exist iff \[\beta(1-c)>\beta_c(N). \label{eq:app-existence-condition}\tag{34}\]

0.0.8.3.2 Leakage and the localization condition

Existence does not by itself bound the size of corrections in the Hessian expansion below. We use a single auxiliary quantity, the leakage: the total softmax mass assigned to memories other than the dominant one. For the outlier and cluster branches, define \[\eta_{\mathrm{out}}:=1-v=Nu,\qquad \eta_{\mathrm{cl}}:=1-a=(N-1)b+q. \label{eq:app-eta-defs}\tag{35}\] Using \(s\perp x_o\) for the outlier and the explicit form of the cluster fixed point, \[\xi_{\mathrm{out}}-x_o=u\,s-(1-v)\,x_o=\eta_{\mathrm{out}}\Big(\tfrac{s}{N}-x_o\Big),\qquad \xi_{\mathrm{cl}}-x_1=b\sum_{j=2}^N(x_j-x_1)+q(x_o-x_1), \label{eq:app-distance-form}\tag{36}\] so \(\|\xi_{\mathrm{out}}-x_o\|\le C_{\mathrm{out}}\eta_{\mathrm{out}}\) and \(\|\xi_{\mathrm{cl}}-x_1\|\le C_{\mathrm{cl}}\eta_{\mathrm{cl}}\) with \(c\)-dependent constants. The condition must control \(\beta\eta\), not only \(\eta\), because the leakage enters the Hessian multiplied by \(\beta\). Indeed, if \(p_k^*=1-\sum_{j\neq k}\eta_j\) and \(p_j^*=\eta_j\), then \[\sum_i p_i^*x_i x_i^\top-\xi_*\xi_*^\top = \sum_{j\neq k}\eta_j(x_j-x_k)(x_j-x_k)^\top+O(\eta^2), \label{eq:app-hessian-leakage-scale}\tag{37}\] and hence \(H_*=I-\beta\sum_{j\neq k}\eta_j(x_j-x_k)(x_j-x_k)^\top+O(\beta\eta^2)\). Thus the working localization condition is \[\beta\,\eta_{\mathrm{out}}\ll 1,\qquad \beta\,\eta_{\mathrm{cl}}\ll 1. \label{eq:app-localization-cond}\tag{38}\] The next two paragraphs derive the corresponding outlier and cluster forms of this same condition.

0.0.8.3.2.1 Outlier branch.

Set \(\delta_{\mathrm{out}}:=1-v=\eta_{\mathrm{out}}\) so \(u=\delta_{\mathrm{out}}/N\). Substituting into (22 ), \[\log\frac{N(1-\delta_{\mathrm{out}})}{\delta_{\mathrm{out}}}=\beta\Big[1-\big(1+\tfrac{A}{N}\big)\delta_{\mathrm{out}}\Big], \label{eq:app-out-delta-eq}\tag{39}\] or in fixed-point form \[\frac{\delta_{\mathrm{out}}}{1-\delta_{\mathrm{out}}}=N\,e^{-\beta}\exp\!\big[\beta(1+\tfrac{A}{N})\,\delta_{\mathrm{out}}\big]. \label{eq:app-out-delta-fixed}\tag{40}\] If \(\beta N e^{-\beta}\ll 1\), the right-hand side is the contraction \(N e^{-\beta}\,(1+O(\beta\delta_{\mathrm{out}}))\) and the leading balance reads \(\delta_{\mathrm{out}}=N e^{-\beta}(1+o(1))\). Hence the outlier form of the localization condition is \[\beta N\,e^{-\beta}\ll 1, \label{eq:app-out-localization-cond}\tag{41}\] or, for \(N\to\infty\), \(\beta-\log N-\log\beta\to+\infty\). Under (41 ), \[v=1-N\,e^{-\beta}(1+o(1)),\qquad u=e^{-\beta}(1+o(1)). \label{eq:app-out-localization-asym}\tag{42}\]

0.0.8.3.2.2 Cluster branch.

Equation (25 ) gives \(b/a=\exp[-\beta(1-c)(a-b)]\). Writing \(a=1-\eta_{\mathrm{cl}}\) and \(a-b=1-\eta_{\mathrm{cl}}-b\), \[b=a\,e^{-\beta(1-c)}\exp\!\big[\beta(1-c)(\eta_{\mathrm{cl}}+b)\big]. \label{eq:app-b-fixed}\tag{43}\] Similarly, from \(\log(a/q)=\beta(z_1-q)\) with \(z_1=a+(N-1)cb\), \[q=a\,e^{-\beta}\exp\!\big[\beta\{(1-c)(N-1)b+2q\}\big]. \label{eq:app-q-fixed}\tag{44}\] The leading values of these off-memory weights are therefore \((N-1)\,e^{-\beta(1-c)}\) from the other cluster memories and \(e^{-\beta}\) from the outlier. The exponential feedbacks in (43 )–(44 ) are negligible iff the cluster form of the localization condition holds: \[\beta(1-c)(N-1)\,e^{-\beta(1-c)}\ll 1,\qquad \beta\,e^{-\beta}\ll 1. \label{eq:app-cl-localization-cond}\tag{45}\] Under (45 ), \[\begin{align} & b=e^{-\beta(1-c)}(1+o(1)),\qquad q=e^{-\beta}(1+o(1)),\\ & a=1-(N-1)\,e^{-\beta(1-c)}-e^{-\beta}+o\big((N-1)\,e^{-\beta(1-c)}+e^{-\beta}\big). \label{eq:app-cl-localization-asym} \end{align}\tag{46}\] Equations (41 ) and (45 ) are simply the outlier and cluster forms of the same localization condition \(\beta\eta\ll1\). Since \(e^{-\beta(1-c)}\gg e^{-\beta}\) for \(0<c<1\), the binding cluster constraint is \(\beta(1-c)(N-1)\,e^{-\beta(1-c)}\ll 1\), equivalently \(\beta(1-c)-\log N-\log\beta(1-c)\to+\infty\), i.e. \[\beta(1-c)\ge \log N+\log\log N+\omega(1). \label{eq:app-beta-localization-final}\tag{47}\]

0.0.8.3.3 Working assumption \(\beta\gg\log N\)

The capacity threshold \(\beta_c(N)=\log N+\log\log N+1+o(1)\) obtained in (33 ) is the order at which the pattern-specific phase transition occurs. The boundary scale \(\beta\sim\log N\) is mathematically interesting and has been the focus of single-task associative-memory analyses, which study how patterns emerge as \(\beta\) crosses this threshold. The present paper, however, is about CL — a more involved setting in which we want to study forgetting and replay at a Task-1 fixed point that is already cleanly localized. We therefore simplify by working in the strictly stronger regime \[\beta\gg\log N, \label{eq:app-beta-strong-regime}\tag{48}\] which directly implies the working condition \[\beta\,N\,e^{-\beta}\ll 1. \label{eq:app-working-cond}\tag{49}\] For fixed \(0<c<1\), \(\beta\gg\log N\) also implies \(\beta(1-c)\gg\log N\), hence \(\beta(1-c)(N-1)\,e^{-\beta(1-c)}\ll 1\), and we obtain the inclusion chain \[\big\{\beta\gg\log N\big\}\;\subset\;\big\{\beta(1-c)\ge \log N+\log\log N+\omega(1)\big\}\;\subset\;\big\{\beta(1-c)>\beta_c(N)\big\}. \label{eq:app-beta-inclusion}\tag{50}\] The strict inclusions are useful: the outermost set is the existence threshold; the middle set enforces the localization condition; the innermost set is what we adopt here. Under (48 ), every Task-1 pattern-specific fixed point is localized around its corresponding memory and the Hessian expansion in the leakage variables is valid.

0.0.8.4 Large-\(\beta\) asymptotics of the Task-1 fixed points

We collect the closed-form expansions used in the body.

0.0.8.5 Hessian expansion for the rotation-averaged energy rise

This subsection collects the Hessian-related calculations used in the proof: the Taylor expansion at a fixed point, the average over small random rotations, and the first-order expansion in the off-memory weights.

0.0.8.5.0.1 Taylor expansion at a Task-1 fixed point.

Because \(\xi_*\) is stationary (\(\nabla E(\xi_*\mid X)=0\)), Taylor expansion gives \[\Delta E_*^{\mathrm{fp}}(V)=\tfrac12(V^\top\xi_*-\xi_*)^\top H_*(V^\top\xi_*-\xi_*)+O(\varepsilon^3),\qquad H_*:=\nabla^2 E(\xi_*\mid X). \label{eq:app-taylor}\tag{51}\] Direct differentiation of (16 ) yields \[H_*=I-\beta\Big(\sum_{i=1}^{N+1}p_i^*x_ix_i^\top-\xi_*\xi_*^\top\Big). \label{eq:app-Hessian}\tag{52}\] Using \(V^\top\xi_*-\xi_*=-\varepsilon\Omega\xi_*+O(\varepsilon^2)\), \[\Delta E_*^{\mathrm{fp}}(V)=\tfrac{\varepsilon^2}{2}(\Omega\xi_*)^\top H_*(\Omega\xi_*)+O(\varepsilon^3). \label{eq:app-delta-hessian}\tag{53}\] The covariance term in Eq. (52 ) also explains the \(\beta\eta\) scale used in the localization condition. Near a pattern-specific fixed point with \(p_k^*=1-\sum_{j\neq k}\eta_j\) and \(p_j^*=\eta_j\), it has the expansion \[\sum_i p_i^*x_i x_i^\top-\xi_*\xi_*^\top = \sum_{j\neq k}\eta_j(x_j-x_k)(x_j-x_k)^\top+O(\eta^2). \label{eq:app-hessian-leakage-expansion}\tag{54}\] Thus leakage contributes to the Hessian at order \(\beta\eta\), which is why Eq. (38 ) controls \(\beta\eta\) rather than merely \(\eta\).

0.0.8.5.0.2 Rotation average.

Substituting into (53 ) and using the main-text convention \(\Delta E_*^{\mathrm{fp}}:=\langle\Delta E_*^{\mathrm{fp}}(V)\rangle_V\) from (8 ), \[\Delta E_*^{\mathrm{fp}}=\tfrac{\varepsilon^2}{4}(\|\xi_*\|^2\,\mathrm{tr}\,H_*-\xi_*^\top H_*\xi_*)+O(\varepsilon^3). \label{eq:app-delta-avg}\tag{55}\]

0.0.8.5.0.3 First-order expansion near a pattern.

Suppose \(\xi_*\) is concentrated at a single stored pattern \(x_k\), with weights \[p_k^*=1-\sum_{j\neq k}\eta_j,\qquad p_j^*=\eta_j\ll 1,\qquad \alpha_j:=x_j^\top x_k. \label{eq:app-eta-leak}\tag{56}\] The fixed-point identity \(\xi_*=Xp_*\) then reads \[\xi_*=x_k+\sum_{j\neq k}\eta_j(x_j-x_k). \label{eq:app-xi-eta}\tag{57}\] We compute the three quantities entering (55 ) step by step, retaining all \(O(\eta)\) terms and dropping \(O(\eta^2)\).

(i) \(\|\xi_*\|^2\). Using \(\|x_k\|^2=1\), \(x_k^\top(x_j-x_k)=\alpha_j-1\), and \((x_i-x_k)^\top(x_j-x_k)=O(1)\), \[\|\xi_*\|^2=1+2\sum_{j\neq k}\eta_j(\alpha_j-1)+O(\eta^2)=1-2\sum_{j\neq k}\eta_j(1-\alpha_j)+O(\eta^2). \label{eq:app-leak-norm}\tag{58}\]

(ii) \(\mathrm{tr}\,H_*\). From (52 ), using \(\|x_i\|^2=1\) and \(\sum_i p_i^*=1\), \[\mathrm{tr}\,H_*=d-\beta\Big(\sum_{i=1}^{N+1}p_i^*\,\|x_i\|^2-\|\xi_*\|^2\Big)=d-\beta\big(1-\|\xi_*\|^2\big), \label{eq:app-leak-tr-step1}\tag{59}\] and substituting (58 ), \[\mathrm{tr}\,H_*=d-2\beta\sum_{j\neq k}\eta_j(1-\alpha_j)+O(\eta^2). \label{eq:app-leak-tr}\tag{60}\]

(iii) \(\xi_*^\top H_*\xi_*\). From (52 ), \[\xi_*^\top H_*\xi_*=\|\xi_*\|^2-\beta\Big(\sum_{i}p_i^*\,(x_i^\top\xi_*)^2-\|\xi_*\|^4\Big). \label{eq:app-leak-xHx-step1}\tag{61}\] Using (57 ), \(x_k^\top\xi_*=1-\sum_{j\neq k}\eta_j(1-\alpha_j)\), and \(x_j^\top\xi_*=\alpha_j+O(\eta)\) for \(j\neq k\). Therefore \[\begin{align} \sum_i p_i^*(x_i^\top\xi_*)^2 &=\Big(1-\sum_{j\neq k}\eta_j\Big)\Big(1-\sum_{j\neq k}\eta_j(1-\alpha_j)\Big)^{\!2}+\sum_{j\neq k}\eta_j\,\alpha_j^2+O(\eta^2)\\ &=1-2\sum_{j\neq k}\eta_j(1-\alpha_j)-\sum_{j\neq k}\eta_j(1-\alpha_j^2)+O(\eta^2), \label{eq:app-leak-sum-step} \end{align}\tag{62}\] where in the last step we used \(-\sum_j\eta_j+\sum_j\eta_j\alpha_j^2=-\sum_j\eta_j(1-\alpha_j^2)\) and absorbed cross-products of \(\eta\)’s into \(O(\eta^2)\). Combining (62 ) with \(\|\xi_*\|^4=1-4\sum_{j\neq k}\eta_j(1-\alpha_j)+O(\eta^2)\) from (58 ) gives \[\sum_i p_i^*(x_i^\top\xi_*)^2-\|\xi_*\|^4=\sum_{j\neq k}\eta_j\big\{2(1-\alpha_j)-(1-\alpha_j^2)\big\}+O(\eta^2)=\sum_{j\neq k}\eta_j(1-\alpha_j)^2+O(\eta^2), \label{eq:app-leak-bracket}\tag{63}\] using the algebraic identity \(2(1-\alpha)-(1-\alpha^2)=(1-\alpha)^2\). Substituting (63 ) into (61 ), \[\xi_*^\top H_*\xi_*=1-2\sum_{j\neq k}\eta_j(1-\alpha_j)-\beta\sum_{j\neq k}\eta_j(1-\alpha_j)^2+O(\eta^2). \label{eq:app-leak-xHx}\tag{64}\]

Substituting (58 ), (60 ), and (64 ) into (55 ), at order \(\eta\), \[\begin{align} \|\xi_*\|^2\,\mathrm{tr}\,H_*-\xi_*^\top H_*\xi_* &=(d-1)-2(\beta+d-1)\sum_{j\neq k}\eta_j(1-\alpha_j)+\beta\sum_{j\neq k}\eta_j(1-\alpha_j)^2+O(\eta^2)\nonumber\\ &=(d-1)-\sum_{j\neq k}\eta_j\Big\{2(d-1)(1-\alpha_j)+\beta(1-\alpha_j)\big[2-(1-\alpha_j)\big]\Big\}+O(\eta^2)\nonumber\\ &=(d-1)-\sum_{j\neq k}\eta_j\big\{2(d-1)(1-\alpha_j)+\beta(1-\alpha_j^2)\big\}+O(\eta^2), \label{eq:app-leak-trick} \end{align}\tag{65}\] where the last line uses the identity \((1-\alpha)\,\big[2-(1-\alpha)\big]=(1-\alpha)(1+\alpha)=1-\alpha^2\). Therefore \[\Delta E_*^{\mathrm{fp}}=\tfrac{\varepsilon^2}{4}\Big[(d-1)-\sum_{j\neq k}\eta_j\,\Lambda(\alpha_j)\Big]+O(\varepsilon^2\eta^2+\varepsilon^3),\quad \Lambda(\alpha):=2(d-1)(1-\alpha)+\beta(1-\alpha^2). \label{eq:app-delta-eta}\tag{66}\] Each competitor at cosine \(\alpha_j\) shaves \(\eta_j\Lambda(\alpha_j)\) off the bare \((d-1)\). The geometric weight \(\Lambda(\alpha)\) vanishes at \(\alpha=1\) (perfect alignment, no help) and is maximal at \(\alpha=0\) (orthogonal competitor, maximum reduction).

0.0.8.6 Specialization and proof of Theorem 1

This subsection applies the Hessian formula to the two Task-1 branches and then compares the resulting energy rises.

0.0.8.6.0.1 Outlier fixed point.

For the outlier-like fixed point all \(N\) competitors are orthogonal: \(\alpha_j=0\) and \(\eta_j=u=e^{-\beta}(1+o(1))\) by Proposition 3. Substituting into (66 ), \[\Delta E_o^{\mathrm{fp}}=\tfrac{\varepsilon^2}{4}\big[(d-1)-N\,e^{-\beta}\{2(d-1)+\beta\}\big]+O(\varepsilon^2 e^{-2\beta}+\varepsilon^3). \label{eq:app-DE-out}\tag{67}\]

0.0.8.6.0.2 Cluster fixed point.

For the cluster-like fixed point near \(x_1\) there are two competitor types (Proposition 4): the other \(N-1\) cluster members with \(\alpha=c\), \(\eta=b=e^{-\beta(1-c)}(1+o(1))\), and the outlier with \(\alpha=0\), \(\eta=q=e^{-\beta}(1+o(1))\). Substituting into (66 ), \[\begin{align} \Delta E_{\mathrm{cl}}^{\mathrm{fp}} &=\tfrac{\varepsilon^2}{4}\Big[(d-1)-(N-1)\,e^{-\beta(1-c)}\{2(d-1)(1-c)+\beta(1-c^2)\}\\ &-e^{-\beta}\{2(d-1)+\beta\}\Big]+O(\varepsilon^2\,e^{-2\beta(1-c)}+\varepsilon^3). \end{align} \label{eq:app-DE-cl}\tag{68}\]

0.0.8.6.0.3 Comparison.

Subtracting (68 ) from (67 ) with \(\Lambda(\alpha)=2(d-1)(1-\alpha)+\beta(1-\alpha^2)\), \[\Delta E_o^{\mathrm{fp}}-\Delta E_{\mathrm{cl}}^{\mathrm{fp}}=\tfrac{\varepsilon^2}{4}(N-1)\big[e^{-\beta(1-c)}\,\Lambda(c)-e^{-\beta}\,\Lambda(0)\big]+o\big(\varepsilon^2\,e^{-\beta(1-c)}\big). \label{eq:app-DE-diff}\tag{69}\] For \(0<c<1\) and \(\beta\gg\log N\) we have \(e^{-\beta(1-c)}/e^{-\beta}=e^{\beta c}\to\infty\), while \(\Lambda(c)=2(d-1)(1-c)+\beta(1-c^2)>0\). Hence the right-hand side is strictly positive at leading order, proving Theorem 1.\(\square\)

0.0.9 Energy–Sharpness Relation↩︎

In the main text, we describe isolated memories as forming high-energy and sharp basins, and cluster-supported memories as forming low-energy and broad basins. This appendix makes that statement precise in the equal-angular cluster-plus-outlier model. The key point is that both the local energy level and the local sharpness of a fixed point are controlled by the same off-memory softmax weights.

0.0.9.1 Definition of local sharpness

Let \(\xi_*\) be a Task-1 fixed point of the Hopfield energy \[E(\xi\mid X) = -\frac{1}{\beta} \log\sum_i \exp(\beta x_i^\top \xi) + \frac{1}{2}\|\xi\|^2 .\] The Hessian at \(\xi_*\) is \[H_* = \nabla^2_\xi E(\xi_*\mid X) = I - \beta \left( \sum_i p_i^* x_i x_i^\top - \xi_*\xi_*^\top \right),\] where \(p_i^*=p_i(\xi_*\mid X)\). We define the local sharpness of the basin by the average Hessian trace in random tangential directions induced by an infinitesimal rotation, \[\mathcal{S}_* := \|\xi_*\|^2 \operatorname{tr} H_* - \xi_*^\top H_* \xi_* . \label{eq:app-sharpness-def}\tag{70}\] Indeed, for \(V=\exp(\varepsilon\Omega)\) with \(\Omega=(W-W^\top)/2\) and \(W_{ij}\sim\mathcal{N}(0,1)\), one has \[\mathbb{E}_{\Omega} \left[ (\Omega\xi_*)^\top H_* (\Omega\xi_*) \right] = \frac{1}{2} \left( \|\xi_*\|^2 \operatorname{tr}H_* - \xi_*^\top H_*\xi_* \right) = \frac{1}{2}\mathcal{S}_* .\] Thus \(\mathcal{S}_*\) is the curvature factor that appears in the rotation-induced energy increase: \[\mathbb{E}_{\Omega} \left[ E(V^\top\xi_*\mid X)-E(\xi_*\mid X) \right] = \frac{\varepsilon^2}{4}\mathcal{S}_* + O(\varepsilon^3).\] Larger \(\mathcal{S}_*\) means that the basin rises more rapidly under small displacements, and we therefore use it as the local sharpness of the Hopfield basin.

0.0.9.2 First-order expansion of energy and sharpness

Consider a fixed point concentrated near a stored memory \(x_k\). Write the softmax weights as \[p_k^* = 1-\sum_{j\neq k}\eta_j, \qquad p_j^* = \eta_j, \qquad \eta_j\ll 1,\] and denote the cosine similarity between the competitor \(x_j\) and the dominant memory \(x_k\) by \[\alpha_j:=x_j^\top x_k .\] The fixed-point condition gives \[\xi_* = x_k+\sum_{j\neq k}\eta_j(x_j-x_k) + O(\eta^2).\]

First, the energy at the fixed point has the expansion \[E(\xi_*\mid X) = -\frac{1}{2} - \frac{1}{\beta} \sum_{j\neq k}\eta_j + O(\eta^2). \label{eq:app-energy-leakage}\tag{71}\] Thus a fixed point supported by more nearby memories has lower energy. In this sense, the total leakage \(\sum_{j\neq k}\eta_j\) measures how much additional support the dominant memory receives from neighboring memories.

Second, the sharpness defined in Eq. 70 satisfies \[\mathcal{S}_* = (d-1) - \sum_{j\neq k}\eta_j \Lambda(\alpha_j) + O(\eta^2), \label{eq:app-sharpness-leakage}\tag{72}\] where \[\Lambda(\alpha) := 2(d-1)(1-\alpha) + \beta(1-\alpha^2). \label{eq:app-lambda-sharpness}\tag{73}\] Since \(\Lambda(\alpha)>0\) for \(\alpha<1\), leakage from neighboring memories lowers the local curvature. Therefore the same off-memory softmax mass that lowers the energy also flattens the basin. This is the precise local sense in which low-energy basins are broad, whereas high-energy basins are sharp.

0.0.9.3 Cluster-plus-outlier specialization

We now apply Eqs. 71 and 72 to the equal-angular configuration of Section 0.0.4.1. For the outlier fixed point, all \(N\) competitors are orthogonal, so \(\alpha=0\) and \(\eta=e^{-\beta}(1+o(1))\). Therefore, \[E_o^* = -\frac{1}{2} - \frac{N}{\beta}e^{-\beta} + o(e^{-\beta}/\beta), \label{eq:app-energy-outlier}\tag{74}\] and \[\mathcal{S}_o = (d-1) - N e^{-\beta}\Lambda(0) + o(e^{-\beta}). \label{eq:app-sharpness-outlier}\tag{75}\]

For the cluster fixed point near \(x_1\), the \(N-1\) other cluster memories have cosine \(c\) and leakage \(e^{-\beta(1-c)}\), while the outlier has cosine \(0\) and leakage \(e^{-\beta}\). Thus, \[E_{\mathrm{cl}}^* = -\frac{1}{2} - \frac{1}{\beta} \left\{ (N-1)e^{-\beta(1-c)} + e^{-\beta} \right\} + o(e^{-\beta(1-c)}/\beta), \label{eq:app-energy-cluster}\tag{76}\] and \[\mathcal{S}_{\mathrm{cl}} = (d-1) - (N-1)e^{-\beta(1-c)}\Lambda(c) - e^{-\beta}\Lambda(0) + o(e^{-\beta(1-c)}). \label{eq:app-sharpness-cluster}\tag{77}\]

Subtracting Eq. 76 from Eq. 74 gives \[E_o^* - E_{\mathrm{cl}}^* = \frac{N-1}{\beta} \left( e^{-\beta(1-c)} - e^{-\beta} \right) + o(e^{-\beta(1-c)}/\beta) > 0, \label{eq:app-energy-order}\tag{78}\] because \(0<c<1\) implies \(e^{-\beta(1-c)}\gg e^{-\beta}\) at large \(\beta\). Hence the outlier fixed point has higher Hopfield energy than the cluster fixed point.

Likewise, subtracting Eq. 77 from Eq. 75 gives \[\mathcal{S}_o - \mathcal{S}_{\mathrm{cl}} = (N-1) \left[ e^{-\beta(1-c)}\Lambda(c) - e^{-\beta}\Lambda(0) \right] + o(e^{-\beta(1-c)}) > 0 . \label{eq:app-sharpness-order}\tag{79}\] Thus the same outlier fixed point is also sharper. Equations 78 and 79 justify the shorthand used in the main text: in this model, high-energy memories correspond to sharp isolated basins, whereas low-energy memories correspond to broad cluster-supported basins.

0.0.9.4 Sharpness in diffusion-model experiments

We further examine the relation between Hopfield energy, local sharpness, and reconstruction-based forgetting in the diffusion-model experiments. Here we focus on CIFAR-10 and compute all quantities sample-wise for Task-1 images.

For each Task-1 sample \(x\), we first compute its Hopfield energy with respect to the Task-1 reference set \(\mathcal{D}_1\), \[E_t(x\mid \mathcal{D}_1),\] using the same energy computation as in the main diffusion-model experiments. To quantify local sharpness, we do not explicitly form the Hessian. Instead, we use a perturbation-based finite-difference estimate: for each sample \(x\), we draw \(M\) small random perturbations \(\{\delta_m\}_{m=1}^M\) and measure how much the Hopfield energy increases around \(x\), \[\widehat{\mathcal{S}}_t(x) = \frac{1}{M} \sum_{m=1}^{M} \left[ E_t(x+\delta_m\mid\mathcal{D}_1) - E_t(x\mid\mathcal{D}_1) \right], \label{eq:perturbation-sharpness}\tag{80}\] up to the fixed perturbation-scale normalization used in the implementation. In the experiments, we use \(M=100\) random perturbations for each sample. Thus, \(\widehat{\mathcal{S}}_t(x)\) measures how steeply the Hopfield energy changes under local random perturbations: larger values indicate that the sample lies in a sharper local region of the energy landscape.

Figure 7 summarizes the result. First, we compare the perturbation-based sharpness \(\widehat{\mathcal{S}}_t(x)\) with the Hopfield energy \(E_t(x\mid\mathcal{D}_1)\). Higher-energy samples tend to have larger perturbation sharpness, consistent with the theoretical picture that weakly supported samples lie in sharper regions of the landscape. Second, we compare \(\widehat{\mathcal{S}}_t(x)\) with the reconstruction-based forgetting metric \(\mathcal{F}_{t^\star}(x)\) defined in Eq. 12 . Samples with larger perturbation sharpness also tend to show larger reconstruction error after Task-2 training. These results suggest that the energy-based ordering observed in the main text is accompanied by a corresponding difference in local sharpness, and that this difference is reflected in reconstruction behavior after continual learning.

0.0.10 Full derivation of Theorem 2↩︎

We derive the exact replay identity, isolate the duplicate-memory baseline, expand the baseline-subtracted gain to leading order in the small rotation, evaluate the five canonical gains, and prove the ordering of Theorem 2.

0.0.10.1 Exact replay identity

Let \(\mathcal{I}=\{1,\dots,N,o\}\), \(X=\{x_\mu:\mu\in\mathcal{I}\}\), and \(X_2(V)=VX\). When a replay point \(x_r\in X\) is added, write \(X_2^{+r}(V):=X_2(V)\cup\{x_r\}\), with the replayed sample counted as an additional term in the log-sum-exp. For a Task-1 fixed point \(\xi_k^*\), the energy rise without replay was already defined in Section 0.0.8.5 as \[\Delta E_k^{\mathrm{fp}}(V)=E(\xi_k^*\mid VX)-E(\xi_k^*\mid X),\qquad \Delta E_k^{\mathrm{fp}}(I)=0. \label{eq:app-DEfp-recall}\tag{81}\] We additionally need the Task-1 softmax weight of the replay point at \(\xi_k^*\), \[p_{r\mid k}:=\frac{e^{\beta x_r^\top\xi_k^*}}{\sum_{\mu\in\mathcal{I}}e^{\beta x_\mu^\top\xi_k^*}}. \label{eq:app-p-rk-def}\tag{82}\] The raw replay-induced energy decrease is \[\mathcal{R}_{r\to k}(V):=E(\xi_k^*\mid X_2(V))-E(\xi_k^*\mid X_2^{+r}(V)). \label{eq:app-raw-gain-def}\tag{83}\] Using the log-sum-exp form (16 ) of \(E\), the difference of the two log-sum-exp’s is \[\mathcal{R}_{r\to k}(V)=\tfrac{1}{\beta}\log\!\left(1+\frac{e^{\beta x_r^\top\xi_k^*}}{\sum_{\mu\in\mathcal{I}}e^{\beta(Vx_\mu)^\top\xi_k^*}}\right). \label{eq:app-raw-gain-log}\tag{84}\] Now observe that, by the definition (81 ) and the log-sum-exp form, \[e^{\beta\Delta E_k^{\mathrm{fp}}(V)}=\frac{\sum_\mu e^{\beta x_\mu^\top\xi_k^*}}{\sum_\mu e^{\beta(Vx_\mu)^\top\xi_k^*}}. \label{eq:app-DE-ratio-id}\tag{85}\] Multiplying numerator and denominator inside (84 ) by \(e^{\beta\Delta E_k^{\mathrm{fp}}(V)}\) converts the rotated denominator back into the Task-1 partition function: \[\frac{e^{\beta x_r^\top\xi_k^*}}{\sum_\mu e^{\beta(Vx_\mu)^\top\xi_k^*}}=\frac{e^{\beta x_r^\top\xi_k^*}\,e^{\beta\Delta E_k^{\mathrm{fp}}(V)}}{\sum_\mu e^{\beta x_\mu^\top\xi_k^*}}=p_{r\mid k}\,e^{\beta\Delta E_k^{\mathrm{fp}}(V)}. \label{eq:app-multiply-trick}\tag{86}\] Hence the exact identity \[\mathcal{R}_{r\to k}(V)=\tfrac{1}{\beta}\log\!\big(1+p_{r\mid k}\,e^{\beta\Delta E_k^{\mathrm{fp}}(V)}\big). \label{eq:app-raw-exact}\tag{87}\] At \(V=I\), \(\Delta E_k^{\mathrm{fp}}(I)=0\) and (87 ) reduces to the trivial duplicate-memory baseline \[\mathcal{R}_{r\to k}(I)=\tfrac{1}{\beta}\log(1+p_{r\mid k}), \label{eq:app-baseline}\tag{88}\] which is the energy decrease one obtains merely from replicating an already-stored Task-1 memory. Subtracting this baseline and using \(\log(a)-\log(b)=\log(a/b)\), \[\widehat{\Delta}_{r\to k}(V):=\mathcal{R}_{r\to k}(V)-\mathcal{R}_{r\to k}(I)=\tfrac{1}{\beta}\log\!\frac{1+p_{r\mid k}\,e^{\beta\Delta E_k^{\mathrm{fp}}(V)}}{1+p_{r\mid k}}. \label{eq:app-delta-exact}\tag{89}\] This separates the two effects: \(p_{r\mid k}\) measures the relevance of the replay point (geometry), and \(\Delta E_k^{\mathrm{fp}}(V)\) measures how much the rotated Task-2 dataset raises the energy at \(\xi_k^*\) (drift) — exactly the same object that drives Theorem 1.

0.0.10.2 Small-rotation expansion and master formula

Define \[\rho(p):=\frac{p}{1+p}. \label{eq:app-rho-def}\tag{90}\] This factor determines the response of forgetting mitigation to the energy rise, because \[\left. \frac{\partial}{\partial \Delta E} \left\{ \frac{1}{\beta}\log\frac{1+p\,e^{\beta\Delta E}}{1+p} \right\} \right|_{\Delta E=0} = \rho(p). \label{eq:app-rho-response}\tag{91}\] We therefore call \(\rho(p)\) the replay susceptibility. This name refers to the response to an energy change in Eq. (91 ). Taylor-expanding (89 ) in \(\Delta E_k^{\mathrm{fp}}(V)=O(\varepsilon^2)\), \[\widehat{\Delta}_{r\to k}(V)=\rho(p_{r\mid k})\,\Delta E_k^{\mathrm{fp}}(V)+\tfrac{\beta}{2}\rho(p_{r\mid k})(1-\rho(p_{r\mid k}))\,\Delta E_k^{\mathrm{fp}}(V)^2+O\big(\beta^2\rho(p_{r\mid k})\,|\Delta E_k^{\mathrm{fp}}(V)|^3\big). \label{eq:app-delta-expand}\tag{92}\] Averaging over \(V\) and setting \(\Delta_{r\to k}:=\langle\widehat{\Delta}_{r\to k}(V)\rangle_V\), and recalling \(\Delta E_k^{\mathrm{fp}}:=\langle\Delta E_k^{\mathrm{fp}}(V)\rangle_V\) from (8 ), \[\Delta_{r\to k}=\rho(p_{r\mid k})\,\Delta E_k^{\mathrm{fp}}+O\big(\beta\rho(p_{r\mid k})\varepsilon^4\big). \label{eq:app-master}\tag{93}\] The factor \(\Delta E_k^{\mathrm{fp}}\) is exactly the rotation-averaged energy rise computed in Section 0.0.8.6 (Eqs. (67 ),(68 )).

0.0.10.3 Susceptibility asymptotics

From Appendix 0.0.8, \[v=p_{o\mid o} = 1-N e^{-\beta}+O(\beta e^{-2\beta}),\] \[a=p_{1\mid1} = 1-(N-1)e^{-\beta(1-c)}-e^{-\beta} +o(e^{-\beta(1-c)}),\] \[b=p_{1\mid2} = e^{-\beta(1-c)}(1+o(1)), \qquad u=p_{1\mid o} = e^{-\beta}(1+o(1)), \qquad q=p_{o\mid1} = e^{-\beta}(1+o(1)).\] Applying \(\rho(p)=p/(1+p)\), \[\begin{align} &\rho(v)=\tfrac12-\tfrac{N\,e^{-\beta}}{4}+O(\beta\,e^{-2\beta}),\\ &\rho(a)=\tfrac12-\tfrac{(N-1)\,e^{-\beta(1-c)}+e^{-\beta}}{4} +o(e^{-\beta(1-c)}),\\ &\rho(b)=e^{-\beta(1-c)}(1+o(1)),\qquad \rho(u)=e^{-\beta}(1+o(1)),\qquad \rho(q)=e^{-\beta}(1+o(1)). \label{eq:app-rho-asym} \end{align}\tag{94}\]

Hence \[\rho(v)>\rho(a)\gg\rho(b)\gg\rho(u)\asymp\rho(q).\]

0.0.10.4 Five replay gains

Recall from (67 )–(68 ), with \(\Lambda(\alpha)=2(d-1)(1-\alpha)+\beta(1-\alpha^2)\), \[\begin{align} \Delta E_o^{\mathrm{fp}}&=\tfrac{\varepsilon^2}{4}\big[(d-1)-N\,e^{-\beta}\,\Lambda(0)\big]+O(\varepsilon^2\,e^{-2\beta}+\varepsilon^3),\\ \Delta E_{\mathrm{cl}}^{\mathrm{fp}}&=\tfrac{\varepsilon^2}{4}\big[(d-1)-(N-1)\,e^{-\beta(1-c)}\,\Lambda(c)-e^{-\beta}\,\Lambda(0)\big]+O(\varepsilon^2\,e^{-2\beta(1-c)}+\varepsilon^3),\\ \Delta E_o^{\mathrm{fp}}-\Delta E_{\mathrm{cl}}^{\mathrm{fp}}&=\tfrac{\varepsilon^2}{4}(N-1)\big[e^{-\beta(1-c)}\,\Lambda(c)-e^{-\beta}\,\Lambda(0)\big]+o\big(\varepsilon^2\,e^{-\beta(1-c)}\big)>0. \end{align}\] Multiplying (94 ) by \(\Delta E_o^{\mathrm{fp}}\) or \(\Delta E_{\mathrm{cl}}^{\mathrm{fp}}\) via the master formula (93 ) yields \[\begin{align} \Delta_{o \to o} &= \tfrac12\,\Delta E_o^{\mathrm{fp}}\,(1+o(1)) = \tfrac{\varepsilon^2}{8}(d-1) + O\big(\varepsilon^2 e^{-\beta}+\varepsilon^3+\beta\varepsilon^4\big), \\ \Delta_{\mathrm{cl}_1 \to \mathrm{cl}_1} &= \tfrac12\,\Delta E_{\mathrm{cl}}^{\mathrm{fp}}\,(1+o(1)) = \tfrac{\varepsilon^2}{8}(d-1) + O\big(\varepsilon^2 e^{-\beta(1-c)}+\varepsilon^3+\beta\varepsilon^4\big), \\ \Delta_{\mathrm{cl}_1\to \mathrm{cl}_2} &= e^{-\beta(1-c)}\,\Delta E_{\mathrm{cl}}^{\mathrm{fp}}\,(1+o(1)) = \tfrac{\varepsilon^2}{4}(d-1)e^{-\beta(1-c)}(1+o(1)), \\ \Delta_{\mathrm{cl}_1\to o} &= e^{-\beta}\,\Delta E_o^{\mathrm{fp}}\,(1+o(1)) = \tfrac{\varepsilon^2}{4}(d-1)e^{-\beta}(1+o(1)), \\ \Delta_{o\to \mathrm{cl}_1} &= e^{-\beta}\,\Delta E_{\mathrm{cl}}^{\mathrm{fp}}\,(1+o(1)) = \tfrac{\varepsilon^2}{4}(d-1)e^{-\beta}(1+o(1)). \label{eq:app-five-gains} \end{align}\tag{95}\]

0.0.10.5 Ordering

1. \(\Delta_{o\to o}>\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1}\). \[\Delta_{o\to o}-\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1} = \{\rho(v)-\rho(a)\}\Delta E_{\mathrm{cl}}^{\mathrm{fp}} + \rho(v)\{\Delta E_o^{\mathrm{fp}}-\Delta E_{\mathrm{cl}}^{\mathrm{fp}}\} + o\big(\varepsilon^2e^{-\beta(1-c)}\big).\] From (94 ), \(\rho(v)-\rho(a) = \tfrac{(N-1)(e^{-\beta(1-c)}-e^{-\beta})}{4} + o(e^{-\beta(1-c)})>0\), and Theorem 1 gives \(\Delta E_o^{\mathrm{fp}}>\Delta E_{\mathrm{cl}}^{\mathrm{fp}}>0\). Thus \(\Delta_{o\to o}>\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1}\).

2. \(\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1}\gg\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}\). \[\frac{\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}}{\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1}} = \frac{\rho(b)}{\rho(a)}(1+o(1)) = 2e^{-\beta(1-c)}(1+o(1)) \to0.\]

3. \(\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}\gg\Delta_{\mathrm{cl}_1\to o}\). \[\frac{\Delta_{\mathrm{cl}_1\to o}}{\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}} = \frac{\rho(u)}{\rho(b)} \frac{\Delta E_o^{\mathrm{fp}}}{\Delta E_{\mathrm{cl}}^{\mathrm{fp}}} = e^{-\beta c}(1+o(1)) \to0.\]

4. \(\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}\gg\Delta_{o\to \mathrm{cl}_1}\). \[\frac{\Delta_{o\to \mathrm{cl}_1}}{\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}} = \frac{\rho(q)}{\rho(b)} = e^{-\beta c}(1+o(1)) \to0.\] Moreover, \[\frac{\Delta_{o\to \mathrm{cl}_1}}{\Delta_{\mathrm{cl}_1\to o}} = \frac{\rho(q)}{\rho(u)} \frac{\Delta E_{\mathrm{cl}}^{\mathrm{fp}}}{\Delta E_o^{\mathrm{fp}}} = 1+o(1),\]

so \(\Delta_{o\to \mathrm{cl}_1}\) and \(\Delta_{\mathrm{cl}_1\to o}\) have the same exponential order. Combining the four comparisons gives \[\Delta_{o\to o}>\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1} \gg \Delta_{\mathrm{cl}_1\to \mathrm{cl}_2} \gg \{\Delta_{\mathrm{cl}_1\to o},\Delta_{o\to \mathrm{cl}_1}\}, \qquad \Delta_{\mathrm{cl}_1\to o}\asymp\Delta_{o\to \mathrm{cl}_1},\] which proves Theorem 2. \(\square\)

0.0.11 Synthetic experiments on MHNs↩︎

0.0.11.1 Intrinsic forgetting in synthetic MHNs

To complement the theory in Section 0.0.4.1, we instantiate the equal-angular MHN at moderate finite values (\(N=12\), \(d=50\), \(c=0.35\), \(\beta=8\), \(\varepsilon=0.02\), 2000 random rotations) – well outside the asymptotic regime \(\beta\gg\log N\), \(\varepsilon\ll 1\) used in the proof – and compute \(\Delta E^{\mathrm{fp}}\) in closed form. Even in this finite-parameter regime the outlier fixed point has a strictly larger rotation-induced energy rise than the cluster fixed points, matching the qualitative ordering predicted by Theorem 1. The corresponding learned MHN-BM result is shown in the main text in Fig. 3 (a).

0.0.11.2 Replay ordering in synthetic MHNs

For completeness, we also verify Theorem 2 in the exact equal-angular regime it was proved in. In the closed-form MHN, the full replay-gain \(\{\Delta_{r\to k}\}_{r,k}\) shows the predicted ordering: the outlier self-entry \(\Delta_{o\to o}\) is largest, cluster self-gains \(\Delta_{\mathrm{cl}_1\to \mathrm{cl}_1}\) are next, within-cluster cross entries \(\Delta_{\mathrm{cl}_1\to \mathrm{cl}_2}\) are substantially smaller, and cross-type entries \(\Delta_{\mathrm{cl}_1\to o},\Delta_{o\to \mathrm{cl}_1}\) are smallest. The corresponding learned MHN-BM corroboration is shown in the main text in Fig. 4.

0.0.12 Domain-incremental learning experiment: Rotated MNIST↩︎

The theoretical analysis of Sections 0.0.4.1 and 0.0.5.1 models continual learning as a small rotation \(V\) of the dataset, which corresponds most directly to a domain-incremental setting in which the same content is presented under a perturbation of the input distribution. The main-text experiments focus on the more widely used incremental-class-learning benchmark with split CIFAR-10. To complement them, we report here the corresponding domain-incremental experiment on rotated MNIST – the setup that most literally matches the rotated-dataset assumption underlying our theorems.

0.0.12.0.0.1 Setup.

Task 1 is standard MNIST. Task 2 is the same MNIST images rotated by \(30^\circ\), instantiating the rotated-dataset construction of Section 0.0.4.1 on real images. We train a pixel-space DDPM from scratch on Task 1 and continue training on Task 2. After Task 1 we assign every Task-1 image a scalar Hopfield energy \(E_\theta(x)\) via the log-sum-exp expression (1 ), using all Task-1 training images as stored memories. As in the main-text diffusion-model experiments, forgetting is measured by the per-sample reconstruction MSE \(\mathcal{F}_{t^\star}(x)\) defined in Eq. (12 ). For the replay analysis we use the same three policies as in Section 0.0.5.3 – no replay (baseline), energy-guided top-\(K\) (high-energy buffer), and energy-guided bottom-\(K\) (low-energy buffer) – with an identical buffer size \(K=5000\). Full hyperparameters are listed in Appendix 0.0.14.

0.0.12.0.0.2 Forgetting indicator (Fig. 10).

After Task 2 fine-tuning, per-sample reconstruction MSE is a monotone-increasing function of the Task-1 Hopfield energy, reproducing the same qualitative relationship observed for split CIFAR-10 in Fig. 3 (b) and Fig. 3 (c). Higher-energy Task-1 images suffer larger reconstruction forgetting, exactly as predicted by Theorem 1.

0.0.12.0.0.3 Energy-guided replay (Fig. 11).

The mean–range tradeoff predicted by Eq. (14 ) – \(\Delta_{r\to k}\sim\rho_{r\mid k}\,\Delta E_k^{\mathrm{fp}}\) – is reproduced on rotated MNIST. The high-energy buffer (Fig. [11][a](#fig:rmnist-replay)) yields a sharply peaked bin-mean reduction concentrated in the high-energy tail (Fig. [11][c](#fig:rmnist-replay)), whereas the low-energy buffer (Fig. [11][b](#fig:rmnist-replay)) gives a smaller per-sample reduction spread over a wider low-energy range. The bin-summed reduction (Fig. [11][d](#fig:rmnist-replay)) is dominated by the high-energy tail under the high-energy buffer, exactly as in the CIFAR-10 results (Fig. 5, 6). Together with the main-text experiments, this confirms that the energy-guided replay rule predicted by Theorem 2 is robust across both incremental-class and domain-incremental continual learning of diffusion models.

0.0.13 Reconstruction metric for diffusion-model forgetting↩︎

The theory in Section 0.0.4 defines forgetting as an intrinsic change in the Hopfield energy, \[\Delta E(x) = E(x\mid\mathcal{X}_{\mathrm{new}}) - E(x\mid\mathcal{X}_{\mathrm{old}}).\] For an explicit modern Hopfield network this quantity can be evaluated directly. For a trained diffusion model, however, the corresponding implicit energy is not available as a normalized scalar function for each sample and each training stage. Although the score determines the local reverse dynamics, measuring the exact sample-wise energy increase before and after Task-2 training would require reconstructing or integrating the implicit energy landscape itself. We therefore use a reconstruction-based quantity as an operational proxy for intrinsic forgetting.

For each Task-1 image \(x\), we first corrupt it to a fixed diffusion timestep \(t^\star\) and then run the reverse process of the model obtained after Task-2 training. Let \(\hat{x}_{0,t^\star}\) denote the resulting reconstruction. We define the reconstruction forgetting score as \[\mathcal{F}_{t^\star}(x) = \|x-\hat{x}_{0,t^\star}\|_2^2 .\] In Stable Diffusion experiments, the image is first encoded into the VAE latent space, noise is added to the latent, the UNet denoises the latent trajectory, and the final latent is decoded back to pixel space. We report the pixel-space reconstruction MSE in the main figures. In the pixel-space diffusion experiments, the same procedure is performed directly in pixel space. When multiple stochastic reconstructions are used, we average the MSE over generations.

This metric should be interpreted as a proxy rather than an exact estimator of \(\Delta E(x)\). If Task-2 training raises or deforms the implicit basin supporting a Task-1 sample, then a noisy version of that sample is less likely to be mapped back to the original point by the post-Task-2 reverse dynamics. This leads to a larger reconstruction error. Conversely, samples that remain well supported by the post-Task-2 landscape should remain easier to reconstruct. Thus, while \(\mathcal{F}_{t^\star}(x)\) is not itself an energy difference, it probes the same local loss of support that intrinsic forgetting is intended to capture.

0.0.14 Hyperparameters↩︎

0.0.14.0.0.1 Synthetic MHN experiment.

\(N=12\) cluster memories, \(d=50\), cosine \(c=0.35\), inverse temperature \(\beta=8\), rotation size \(\varepsilon=0.02\), \(n_{\text{rot}}=2000\) random antisymmetric \(\Omega\), fixed-point tolerance \(10^{-13}\) (scripts/mhn_equal_angler_replay.py).

0.0.14.0.0.2 Synthetic MHN-BM experiment.

\(N=12\) cluster memories, \(d=50\), \(c=0.35\), \(\beta=16\) (consistent in loss, update, and energy), data scale \(2.0\), rotation \(\varepsilon_{\mathrm{rot}}=0.02\), \(n_{\mathrm{hidden}}=N+1\), Adam with learning rate \(0.02\), batch size \(64\).

0.0.14.0.0.3 Stable Diffusion split CIFAR-10.

Backbone: runwayml/stable-diffusion-v1-5; only the UNet is fine-tuned, VAE frozen. Task 1 classes: bird, cat, deer, dog, horse. Task 2 classes: airplane, automobile, frog, ship, truck. 25,000 samples per task, 32\(\times\)​32 images. Replay buffer \(K=5000\); replay selection \(\in\{\text{no replay, random, high\_energy}\}\). Forgetting metric: reconstruction MSE at \(t=0.8\).

0.0.14.0.0.4 Pixel-space diffusion split CIFAR-10.

PixelCNN-UNet trained from scratch in pixel space (\(32\times 32\) RGB), “standard” channels \((128,256,512)\) (\(\sim\)​45M params). Same living/non-living split as the Stable Diffusion setup, \(25{,}000\) samples per task. AdamW (lr \(2\times 10^{-4}\), weight decay \(10^{-4}\), \(500\) warmup steps), fp16, batch \(512\), \(2000\) epochs/task, EMA decay \(0.9999\). Linear noise schedule (\(T=1000\), \(\beta\in[10^{-4},2\!\times\!10^{-2}]\)), \(50\) inference steps. Replay buffer \(K=5000\); replay selection \(\in\{\text{no replay, random, high\_energy, low\_energy}\}\). Forgetting metric: reconstruction MSE at \(t=400\).

0.0.14.0.0.5 Pixel-space diffusion rotated MNIST.

Same PixelCNN-UNet adapted to \(28\times 28\) grayscale. Two tasks defined by rotation_angles \(=[0^\circ,30^\circ]\). Optimizer, EMA, and noise schedule identical to the pixel-space CIFAR-10 setup; batch \(256\) for \(200\) epochs/task, no augmentation. Replay buffer \(K=5000\); replay selection \(\in\{\text{no replay, random, high\_energy, low\_energy}\}\). Forgetting metric: reconstruction MSE at \(t=400\).