1 Introduction↩︎

Visual diffusion models represent a significant advancement in the field of generative modeling, particularly for image synthesis tasks. These models leverage the concept of diffusion, a process inspired by statistical physics, to generate images from random noise [1], [2]. Compared to traditional Generative Adversarial Networks (GAN) [3] and Variational Autoencoders (VAE) [4], visual diffusion models excel in producing high-quality, diverse, and stable images. Famous visual diffusion models include OpenAI’s DALL-E [5]–[7], Stability AI’s Stable Diffusion [8]–[10], Google’s Imagen [11], and Baidu’s ERNIE-ViLG [12], [13], drawing widespread attention from researchers, practitioners, and enthusiasts.

Visual diffusion models have a broad range of real-world applications across various industries. In the entertainment sector, these models are utilized for creating highly realistic visual effects [14], animations [15], and virtual environments in movies and video games [16], significantly reducing production costs and time. In the field of design and fashion, they aid in generating new styles, patterns, and prototypes, fostering innovation and creativity [17]–[19]. Marketing and advertising benefit from these models through the creation of visually appealing and customized content that enhances consumer engagement [20]. Additionally, in healthcare, visual diffusion models assist in medical imaging by enhancing the quality of diagnostic images [21], [22] and creating synthetic data for research and training purposes [23], [24]. The image-generating AI market is estimated to be valued at around \(349.6\) million in 2023 and is expected to grow to approximately \(1,081.2\) million by 2030 [25].

To achieve such outstanding performance and broad applications, visual diffusion models highly rely on extensive web data, such as LAION-5B [26], for training. However, this data encompasses several significant issues: First, at the concept level, the training data often contains biased gender [27] and culture [28], racial representations [29], and Not Safe For Work (NSFW) materials [30]. Second, at the content level, web data includes a substantial amount of copyrighted images [31], medical images containing patient information [32], and photos of politicians or celebrities [33]. Third, at the style level, data may include works characterized by unique stylistic elements from contemporary artists [34], [35]. These issues lead to some generated images exhibiting unfair outcomes, inappropriate content, ethical risks, and copyright infringement [36], thereby negatively impacting the widespread applications of visual diffusion models.

Fundamentally, as shown in Fig. 1, this problem comes from an inevitable and important phenomenon in current visual diffusion models, i.e., during training, these models memorize the training images and replicate their concepts, content, or styles during the inference stage. Currently, an increasing amount of research is being conducted to discuss this replication phenomenon. However, there is a lack of surveys that specifically focus on replication in visual diffusion models. In this survey, we provide the first comprehensive review of replication in visual diffusion models, which not only systematically investigates this research topic but also potentially benefits the improvement of model safety and ethical standards in the real world.

Our survey systematically introduces the concept of replication in visual diffusion models from three perspectives: unveiling, understanding, and mitigation. sec:Unveiling involves identifying and exposing replication through techniques such as similarity retrieval [34], [37], membership inference [38], [39], and prompting [40], [41]. sec:Understanding explores the mechanisms behind replication, including factors like data duplication [31], [42] and inappropriate training methods [43], [44]. sec:Mitigation discusses strategies to minimize replication, such as differential privacy [45], [46],[47], data deduplication [48], [49], machine unlearning [50], [51],[52], [53], and inference-time anti-replication guidance [54]. Lastly, we explore the influence of replication in the real world, including regulation [55], [56], art [57], [58], society [29], [59], and healthcare [60], [61]. An overview of this survey is available at Fig. [Fig:32overview].

This survey makes the following contributions:

1. This is the first survey that systematically reviews the concept of replication in visual diffusion models. We innovatively discuss this phenomenon from the perspectives of unveiling, understanding, mitigation, and its influence in the real-world.

2. We provide a brief overview of visual diffusion models, including their categorization, theoretical foundations, and functionalities. We then formally introduce the term replication within this context, providing a concise definition and understanding of its meaning.

3. By pointing out the inadequacies of current methods and the challenges existing in replication, we provide a roadmap for future research, such as developing more accurate and efficient unveiling methods and creating more robust mitigation strategies.

=[draw=none, rounded corners,minimum height=1.2em, edge=black!10, text opacity=1, align=center, fill opacity=.3, text=black,font=, inner xsep=2pt, inner ysep=3.6pt, ] =[draw=none, rounded corners,minimum height=1.2em, edge=black!10, text opacity=1, align=center, fill opacity=.5, text=black,font=, inner xsep=2pt, inner ysep=3.6pt, ] =[draw=none, rounded corners,minimum height=1.2em, edge=black!10, text opacity=1, align=center, fill opacity=.8, text=black,font=, inner xsep=2pt, inner ysep=3.8pt, ] =[draw=none, rounded corners,minimum height=1.2em, edge=black!10, text opacity=1, align=center, fill opacity=1, text=black,font=, inner xsep=2pt, inner ysep=3.8pt, ]

=[draw=none, rounded corners,minimum height=1.2em, edge=black!10, text opacity=1, align=center, fill opacity=1, text=black,font=, inner xsep=2pt, inner ysep=3.8pt, ]

The remainder of this survey is organized as follows: In Section 2, we highlight the differences between our survey and existing ones. In Section 3, we briefly introduce visual diffusion models and define the phenomenon of replication. Subsequently, we summarize unveiling, understanding, and mitigation in Sections 4, 5, and 6, respectively. Additionally, in Section 14, we review papers that focus on the influence of replication in the real world. Finally, we present the current challenges and future directions in Section [sec:Challenges] and conclude the survey in Section 9.

2 Related Works↩︎

Diffusion models in vision. Existing surveys on diffusion models, such as [62]–[65], provide comprehensive overviews of various diffusion modeling techniques and their applications in computer vision. These surveys categorize diffusion models, discuss their theoretical foundations, and highlight their performance in tasks like image synthesis and data augmentation. In contrast, our survey uniquely focuses on the critical issue of replication within diffusion models. We systematically explore this phenomenon through the lenses of unveiling, understanding, and mitigation, thereby filling a gap between general diffusion model overviews and the specific challenge of replication.

Safety of diffusion models. Existing surveys on the safety of diffusion models often address issues such as bias, misinformation, privacy concerns, and copyright protection. For instance, [66] emphasizes the critical need to identify AI-generated content to prevent its misuse and potential societal disruptions. [36] explores privacy risks associated with generative AI and highlights the importance of robust detection and authentication. Additionally, [67] and [68] investigate the broader ethical implications and technical challenges of ensuring the integrity and trustworthiness of AI-generated content, including the use of privacy-preserving techniques and blockchain for content verification. Furthermore, [69] addresses the legal and technical challenges of protecting intellectual property rights in the context of AI-generated works, emphasizing the need to identify and verify copyrighted content.

In contrast, while our survey also falls under the safety of diffusion models, we specifically target the replication phenomenon within visual diffusion models. This focus is unique compared to existing surveys: while these surveys emphasize detection and mitigation of AI-generated content to prevent misuse and ensure ethical deployment, our survey goes deeper into the intrinsic properties of diffusion models related to replication. This distinction not only complements existing studies but also provides a more granular understanding of the safety concerns associated with visual diffusion models.

Replication in large language models. The replication phenomenon in large language models (LLMs) have been extensively studied in recent literature. Works such as [70] explore the implications of memorization for privacy, security, and copyright. Similarly, the survey [71] provides a comprehensive overview of methods for extracting training data from LLMs and discusses the inherent challenges in mitigating these risks. Our survey differentiates itself by focusing specifically on visual diffusion models, filling this gap in the current literature.

Comparison with related surveys. To further clarify the position and contributions of our survey, Table 1 provides a structured comparison with representative related works. Our survey is the only one that (i) concentrates exclusively on the replication phenomenon in visual diffusion models, (ii) covers all three semantic levels (concept, content, style), (iii) provides a formal threshold-based definition, and (iv) includes a dedicated section on real-world influence spanning regulation, art, society, and healthcare.

3.8pt

3 Background↩︎

In this section, we provide an overview of visual diffusion models and formally define the replication phenomenon.

3.1 Visual Diffusion Models↩︎

Categorization and theoretical foundations. Diffusion models are typically categorized into three main types: denoising diffusion probabilistic models (DDPMs) [2], noise-conditioned score networks (NCSNs) [72], and stochastic differential equations (SDEs) [73].

Denoising Diffusion Probabilistic Models (DDPMs): DDPMs add Gaussian noise to the data in a forward process and learn to reverse this process to denoise the data. The forward process is defined as: \[q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{\alpha_t} \mathbf{x}_{t-1}, (1 - \alpha_t) \mathbf{I}),\] where \(\alpha_t\) is a noise schedule parameter. The reverse process is: \[p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \mathbf{\mu}_\theta(\mathbf{x}_t, t), \sigma_t^2 \mathbf{I}),\] with \(\mathbf{\mu}_\theta\) being predicted by a neural network.

Noise-Conditioned Score Networks (NCSNs): NCSNs estimate the score function, the gradient of the log density of the data, to denoise the data. The forward process introduces noise, and the model learns to predict the score: \[\mathbf{s}_\theta(\mathbf{x}_t, t) \approx \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t).\] The reverse process uses Langevin dynamics to generate new samples: \[\mathbf{x}_{t+1} = \mathbf{x}_t + \frac{\epsilon^2}{2} \mathbf{s}_\theta(\mathbf{x}_t, t) + \epsilon \mathbf{z}, \quad \mathbf{z} \sim \mathcal{N}(0, \mathbf{I}),\] where \(\epsilon\) is a step size parameter.

Stochastic Differential Equations (SDEs): SDEs generalize the diffusion process using continuous-time dynamics. The forward process can be described by an SDE: \[d\mathbf{x}_t = \mathbf{f}(\mathbf{x}_t, t) dt + g(t) d\mathbf{w}_t,\] where \(\mathbf{w}_t\) is a standard Wiener process. The reverse-time SDE is used to generate samples: \[d\mathbf{x}_t = [\mathbf{f}(\mathbf{x}_t, t) - g(t)^2 \nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t)] dt + g(t) d\mathbf{\hat{w}}_t,\] where \(d\mathbf{\hat{w}}_t\) is the reverse-time Wiener process.

Functionalities. Visual diffusion models exhibit a broad range of functionalities, including storytelling [74]–[76], virtual try-on [77]–[79], drag edit [80]–[82], diffusion inversion [83]–[85], text-guided editing [86]–[88], T2Iaugmentation [89]–[91], spatial control [92]–[94], image translation [95]–[97], inpainting [98]–[100], layout generation [101], [102], super resolution [103], [104], video generation [105], [106], and video editing [107], [108], showing their versatility and applicability across diverse domains. However, at the same time, visual diffusion models also pose potential threats to this wide range of functionalities through the replication of their training data. This underscores the necessity of our survey, which provides a comprehensive review of this phenomenon, aiming to enhance model safety and ethical standards.

3.2 Replication↩︎

Definition. Let \(\mathcal{T}=\left\{x_1, x_2, \ldots, x_n\right\}\) denote a training set of \(n\) samples. A diffusion model trained on this set is denoted as \(f_{\mathcal{T}}\). During the inference phase, the model generates a set of \(m\) data points denoted as \(\mathcal{G}=\left\{\hat{x}_1, \ldots, \hat{x}_m\right\}\). Given a feature extractor \(\phi\) and a distance metric \(d\) defined in the induced feature space, we say that a trained diffusion model \(f_\mathcal{T}\) replicates a training sample \(x_i \in \mathcal{T}\) via a generated output \(\hat{x}_j \in \mathcal{G}\) if and only if \[d\!\left(\phi(\hat{x}_j),\, \phi(x_i)\right) < \tau,\] where \(\tau > 0\) is a task-specific threshold calibrated on a labeled replication dataset. We say that \(f_\mathcal{T}\) replicates its training set if such a pair \((x_i, \hat{x}_j)\) exists.

Remark on the threshold \(\tau\). Because replication occurs at multiple semantic levels, \(\phi\) and \(\tau\) are defined separately for each level:

• Content level: \(\phi\) is a copy-detection backbone (e.g., SSCD [109]); \(d\) is cosine distance; \(\tau_\text{content}\) is calibrated on labeled near-duplicate pairs to maximize the F1 score.

• Style level: \(\phi\) extracts Gram-matrix or style-space features [34], [35]; \(\tau_\text{style}\) is calibrated on artist-attributed style pairs to capture stylistic near-replication without requiring identical content.

• Concept level: \(\phi\) is a semantic encoder (e.g., CLIP [110]); \(\tau_\text{concept}\) is calibrated on concept-level annotations to capture high-level semantic similarity, including biased or inappropriate conceptual replications.

The distance metric \(d\) is a function defined on the feature space \(\mathcal{F}\): \(\mathcal{F} \times \mathcal{F} \rightarrow \mathbb{R}_{\geq 0}\), satisfying for all \(u, v, w \in \mathcal{F}\):

• \(d(u, v)=0 \iff u=v\) (identity of indiscernibles);

• \(d(u, v)=d(v, u)\) (symmetry);

• \(d(u, w)\leq d(u, v)+d(v, w)\) (triangle inequality).

4 sec:Unveiling↩︎

In this section, we focus on the first aspect of our survey on replication in visual diffusion models, which is unveiling. Unveiling [111]–[114] refers to the process of uncovering the phenomenon of replication, either manually or through the use of specially-designed machine learning models. As shown in Fig. [Fig:32overview], we organize the unveiling of replication into 6 categories, i.e., prompting, membership inference, similarity retrieval, proactive replication, watermarking, and novel perspectives. Fig. 2 illustrates these categories.

4.1 Prompting↩︎

These articles investigate how prompting can reveal replication in visual diffusion models. As shown in Fig. 2 (a): by using specific prompts, i.e., prompts deliberately designed to elicit memorized content by including verbatim or near-verbatim training captions, researchers can generate images that closely resemble the model’s training data; beyond that, some papers show visual diffusion models may replicate learned copyrighted images implicitly, i.e., through prompts that indirectly trigger replication by describing concepts strongly associated with specific training images.

Specific. Specific prompts are carefully chosen phrases or descriptions from researchers to test whether visual diffusion models can replicate. For instance, [37], [115]–[117] employ specific prompts that are known to correspond to particular training images to see if the generated images closely resemble these originals. By injecting maliciously crafted data into the training set, researchers [118] can use specific prompts to trigger the model to produce near-identical copies of copyrighted images. The articles [119]–[121] demonstrate that by using prompts that are likely to invoke sensitive or controversial topics, the diffusion model can be coaxed into generating unsafe or offensive images. By using prompts that include the names of famous artists [40] or refer to different social stereotypes [122], the researchers show that the model can produce images that closely mimic the unique features of their styles or reflect biased society representations.

Implicit. Replication can also occur when user prompts are related to certain concepts or topics implicitly or unintentionally. For instance, these studies [41], [117], [123] highlight how diffusion models can replicate copyrighted content with such prompts. They further utilize techniques such as keyword extraction and gradient-based prompt optimization to analyze the attention mechanisms within these models.

4.2 Membership Inference↩︎

Membership inference attacks (MIAs) aim to determine whether specific data samples are part of the model’s training set. These attacks exploit patterns in the model’s behavior, such as how it processes and reconstructs data, to infer the presence of training data. If visual diffusion models have not been trained on a data sample, they will never replicate it. Therefore, MIAs have a strong relationship with replication, and we review MIAs in the context of visual diffusion models in this section. Based on the level of access attackers have, MIAs can be categorized into white-box and black-box attacks, as shown in Fig. 2 (b).

White-box. White-box membership inference attacks diffusion models by leveraging their internal parameters and gradients. Key methods include loss-based attacks [124], [125], likelihood-based attacks [124], [125], gradient-based attacks [126],[127], quantile regression [128], proximal initialization [39]. These methods highlight significant privacy risks for diffusion models when accessing their internal weights, especially handling sensitive data.

Black-box. Black-box MIAs focus on exploiting the generated images rather than visual diffusion models’ internal parameters. Key studies have shown that these attacks can effectively differentiate members based on image quality and semantic fidelity [125], [129]. Existing techniques include leveraging probabilistic fluctuation [130], using data watermarking [131], and analyzing statistical properties of generated distributions [132]. Some methods also highlight significant privacy risks in fine-tuned [133],[134] and large-scale [135] diffusion models. Novel techniques such as leveraging initial noise residuals [136], and frequency-calibrated attacks on medical images [137] further expand the attack surface. Dual-model defenses [138] have also been proposed to safeguard diffusion models against MIAs while maintaining utility. Some work [128], [135] extends MIA evaluation to realistic open-world settings, demonstrating that reported near-100% AUC values are significantly attenuated when membership labels are noisy or the member/non-member distributions overlap.

To provide a systematic comparison, Table 5 compiles published MIA results from representative methods across both controlled (DDPM trained from scratch) and realistic (pre-trained Stable Diffusion on LAION) evaluation settings. The most striking finding is the evaluation gap: black-box methods that achieve AUC 0.88–0.99 on DDPMs trained from scratch on small datasets [38], [39], [130] drop to near-random AUC (\(\approx\)​0.51–0.54) when evaluated against pre-trained Stable Diffusion on its actual LAION training data [128]. Only white-box gradient attacks (GSA) [126] remain effective across all settings (AUC = 1.0), but these require full model parameter access, which is rarely met in practice. Our own experiments on SD v1.5 (50 members, 50 non-members) confirm the moderate performance (AUC \(\approx\) 0.61) of both loss-based and gradient-norm attacks under realistic conditions, consistent with the critical reassessments by [128], [135]. Furthermore, [139] demonstrates that diffusion language models’ multiple maskable configurations exponentially increase attack opportunities, achieving 30% relative AUC improvement via subset-aggregated attacks.

3.8pt

4.3 Similarity Retrieval↩︎

Similarity retrieval is a method that closely aligns with human common sense for uncovering replication. This approach involves searching for and identifying items in a dataset that are similar to a given query item. In the context of diffusion models, similarity retrieval allows for comparing generated outputs against the training data. When a generated image/video closely matches an image/video from the training set, it raises concerns about the model replicating specific data points rather than generalizing from the training data. As shown in Fig. 2 (c), the primary retrieval methods for unveiling replication are through content similarity, while style similarity is also used to help identify artworks mimicry.

Content similarity. Content similarity focuses on comparing the actual content or subject matter of the generated images or videos to the training data. The first step of comparison involves feature extraction with pre-trained vision(-language) models [110], [140]–[143] or specialized image copy detection methods [109], [144]–[146]. After that, these extracted features are used to compute similarity scores between generated content and training samples through various metrics such as cosine similarity, Euclidean distance, or more complex learned metrics [37], [147]–[151].

Style similarity. Style similarity involves comparing the artistic style or aesthetic elements of generated images or videos to those in the training data. This approach is crucial for identifying instances where a diffusion model replicates the distinctive style of contemporary artworks or artists. For instance, [152] explores how well diffusion models can replicate the styles of human artists. Additionally, [34] discusses a framework for understanding and extracting style descriptors from images generated by diffusion models. Furthermore, [35] generalizes the pattern retrieval algorithm for image copy detection to measure style similarity.

We empirically validate style-level replication by generating images with Stable Diffusion v1.5 [8] using artist-named prompts versus generic painting prompts. Table 2 reports both VGG-19 Gram-matrix similarity [34] and CLIP ViT-B/32 similarity [110]. Artist-named prompts consistently produce 15–18 percentage points higher intra-set similarity than generic prompts across both metrics, confirming style-level memorization. The cross-artist similarity block further shows that stylistically related artists (e.g., Van Gogh–Monet Gram = 0.588) share higher cross-set similarity than unrelated pairs (e.g., Monet–Picasso Gram = 0.418), indicating that the model captures fine-grained style relationships learned during pre-training. These findings are consistent with the style descriptors proposed by Somepalli et al. [34] and the pattern retrieval framework of Wang et al. [35].

4.4 Watermarking↩︎

By embedding imperceptible watermarks into the data, one can detect the presence of these watermarks in the generated images if a visual diffusion model uses the data during training or fine-tuning processes. In this way, unveiling possible replication is simplified to detecting and verifying the occurrence of watermarks, as shown in Fig. 2 (d). Unlike comparing similarities, which aligns with common sense but is difficult to use as legal evidence, watermarking techniques provide concrete evidence of copyright infringement and protect the intellectual property of rights holders. Several methods have been proposed to embed such watermarks into images. For instance, DIAGNOSIS [153] detects unauthorized data usage in text-to-image diffusion models by injecting unique behaviors into models via modified datasets; DiffusionShield [154] embeds invisible watermarks containing copyright information into images; and FT-SHIELD [155] uses imperceptible watermarks embedded in data to verify if it has been misused in the training or fine-tuning of text-to-image diffusion models. Beyond watermarking general images, [156] embeds robust, invisible watermarks into artworks to trace art theft. More recently, VideoShield [157] extends watermarking to diffusion-based video generation by embedding watermarks directly during the denoising process, enabling both ownership verification and temporal–spatial tamper localization. Comprehensive surveys [158] and adoption studies [159] further analyze the current landscape, finding that only a minority of AI image generators implement adequate watermarking practices. PhaseMark [160] achieves state-of-the-art watermark resilience via single-shot phase modulation in the VAE latent frequency domain, while TrajPrint [161] extracts unique manifold fingerprints from deterministic denoising trajectories for training-free, lossless copyright verification.

4.5 Proactive Replication↩︎

Recently, some personalized visual diffusion models [85], [162]–[170] have been successfully designed to fine-tune on specific subjects or styles using minimal input data. Remarkably, some models [171], [172] can even learn from this minimal input data in a training-free manner. This enables users to generate images that highly preserve the original visual characteristics and essence of the subjects or styles at a very low cost, as shown in Fig. 2 (e).

We refer to this as proactive replication, i.e., the intentional, user-driven use of fine-tuning or inference-time mechanisms to replicate a specific subject or style, unlike the aforementioned reviewed methods, which inevitably and unintentionally replicate. Proactive replication in visual diffusion models represents a double-edged sword: while it offers opportunities for the creative industry with enhanced personalized content [162], it also poses significant ethical and practical challenges [173]. One of the most pressing concerns is the potential for these models to replicate and commercialize the artistic styles of living artists without consent [174]. This capability to reproduce artists’ styles at low cost undermines the years of effort artists invest in their unique visual signatures.

4.6 Novel Perspectives↩︎

In addition to these categorizations of unveiling replication, several novel perspectives have emerged that offer unique insights and techniques. As shown in Fig. 2 (f), these perspectives [175]–[181] leverage different aspects of model behavior and training data characteristics to uncover replication in visual diffusion models:

1. Magnitude of noise. This research [175] presents a method for detecting replication by examining the magnitude of text-conditional noise predictions. By analyzing these magnitudes, the study unveils how specific text tokens contribute to replication, allowing users to adjust their prompts.

2. Bright ending attention. The work [182] identifies a novel attention anomaly in diffusion models prone to memorization: memorized image patches exhibit significantly greater attention to the final text token. This “bright ending” pattern is the first method to detect localized memorization regions using only a single inference pass, without requiring access to training data.

3. Anisotropic detection. The study [183] shows that existing norm-based memorization detection metrics only work under isotropic log-probability; integrating anisotropic alignment with the isotropic norm yields a detection metric computable from two forward passes on pure noise, running approximately 5\(\times\) faster than prior methods.

4. Training data attribution. The papers [176], [177] emphasize the role of training data in guiding diffusion models by tracing back generated outputs to their original training data. This approach aids in identifying instances where the model excessively relies on specific training samples. More recently, influence functions [184], diffusion attribution scores [185], and nonparametric methods [186] have been proposed to scale data attribution to large diffusion models.

5. Cross attention. This work [178] investigates the role of cross attention mechanisms in text-to-image diffusion models. Examining cross-attention mechanisms helps identify a model’s replication because models tend to focus on certain text-image pairs.

6. Fine-tune to leak. This research [179] highlights the risks associated with fine-tuning diffusion models, which can amplify replication issues. To determine if a visual diffusion model has serious replication issues, it is feasible to check whether the model has undergone fine-tuning.

7. Overfitted Masked Autoencoder (MAE). The paper [180] proposes using overfitted MAEs to detect generative parroting in diffusion models. By identifying overfitting patterns, the study spots when a model is replicating training data instead of generating novel content.

8. Property inference. This work [181] explores how property existence inference can be used to detect replication in generative models. By inferring whether certain properties exist in the training data, the method helps in identifying instances of replication and implementing measures to reduce such occurrences.

To consolidate the above discussion, Table 6 compiles published memorization extraction and detection results across representative diffusion models and datasets, offering the first cross-study quantitative comparison. Several observations stand out: (i) content-level verbatim extraction rates are low in absolute terms but non-negligible for large-scale deployment, e.g., Carlini et al. [115] extract 109 training images from Stable Diffusion v1.4 out of 175 million candidates; (ii) the extraction rate decreases substantially from SD v1 to SD v2 due to dataset deduplication [116]; (iii) detection-oriented methods [175] achieve near-perfect AUC (0.999) for identifying which prompts trigger memorization, suggesting that proactive detection is more practical than post-hoc extraction; (iv) our own controlled experiment confirms that cross-seed same-prompt similarity (0.906) far exceeds cross-prompt similarity (0.574), indicating prompt-driven replication dominates stochastic variation.

5 sec:Understanding↩︎

After unveiling the phenomenon of replication in visual diffusion models, understanding its mechanism is crucial for developing effective mitigation strategies and improving the safety and ethical standards of these models. As outlined in Fig. [Fig:32overview], this section explores the underlying mechanisms that contribute to replication from three different perspectives: data, methods, and theory. The demonstration of these aspects is shown in Fig. 3.

5.1 Data↩︎

Data plays a crucial role in the replication phenomenon observed in visual diffusion models. The quality, duplication, and bias present in the training data directly impact the model’s behavior. As shown in Fig. 3 (a), this section explores how insufficient training data, image duplication, misleading captions, and data types contribute to replication.

Insufficient training data. When the training dataset is too small, the model is not exposed to enough variety and tends to overfit the limited examples it has seen. This overfitting means that the model memorizes specific details of the training data, which it then replicates during the generation phase. The concept of Effective Model Memorization (EMM) [187] is introduced to represent the maximum size of a training dataset where the model approximates the theoretical optimum in terms of memorization. Empirically, researchers [187] also show that as the size of the training dataset increases, the replication ratio decreases.

Image duplication. When training data contains multiple copies or near-duplicates of the same images, the model is more likely to replicate these images during inference [188]. This issue is particularly prevalent in large-scale datasets scraped from the web, where duplicates are common due to the lack of rigorous data cleaning processes. Experiments by [37] and [31] on datasets such as Oxford Flowers [189], Celeb-A [190], ImageNet [191], and LAION [26] demonstrate that the degree of content replication varies with the image duplication rates in these datasets.

Misleading captions. When captions are duplicated, specific, or inaccurate, they can misguide the model during the training phase, leading to the replication of specific image-caption pairs. For instance, while it is commonly believed that image duplication alone causes replication, research [31], [37], [42], [188] indicates that the similarity of captions in the training data can also influence replication behavior. Additionally, experiments [42] reveal that specific keywords, such as “Van Gogh”, in the training data can lead to clusters of nearly identical images. Surprisingly, [187] discovers that the replication issue in diffusion models can be significantly exacerbated when training data is conditioned on inaccurate captions. This may be because such captions do not provide meaningful guidance for the model during training, leading to overfitting on specific training examples.

Data types. Beyond these common understandings in data, [192] finds that inliers (data points that are representative of the general distribution of the training data) are memorized earlier in the training process, while outliers (data points that are atypical or rare within the training set) tend to be memorized later. This indicates that the visual diffusion model focuses on learning the core characteristics of the dataset before handling more unusual data.

5.2 Methods↩︎

To complement insights from a data perspective, this section demonstrates how training methods can influence replication in visual diffusion models. It specifically examines the roles of a deterministic sampler and model capacity. To deepen the analysis of model behavior, we additionally review new metrics developed specifically for assessing replication. The illustrations of these are shown in Fig. 3 (b).

Deterministic sampler. Deterministic samplers are mechanisms used in visual diffusion models to generate data in a repeatable and predictable manner. The researchers [43] find that deterministic samplers lead to generated samples that are highly correlated with the training set. This high correlation indicates that the model is replicating patterns seen during training rather than generating truly novel data. Further, [44] demonstrates that while deterministic samplers can lead to replication, they can also support generalization under appropriate training conditions. More recently, [193] shows that classifier-free guidance (CFG) within the attraction basin of training samples can directly cause memorization, providing a mechanistic explanation for why high guidance scales amplify replication.

Model capacity. Model capacity refers to a machine learning model’s ability to fit a wide variety of functions, which is determined by the model’s complexity. Complexity factors include the number of parameters, the depth of the model, and the model’s structure. In visual diffusion models, replacing the commonly-used U-Net backbone [194] with a transformer [195] – referred to as Diffusion Transformers (DiTs) [196] – results in a higher model capacity. Although models with greater capacity often achieve lower Frechet Inception Distance (FID) and better visual fidelity, they are also more prone to replicating training data. For instance, [31] demonstrates that large models with substantial capacity can retain detailed information from the training data, which may lead them to replicate these details during inference. Furthermore, [188] finds that high-capacity models, due to their complexity, are more likely to replicate training data, particularly under conditions of insufficient data diversity or small dataset size.

New metrics. Beyond understanding replication from the perspective of training methods, [197]–[199] underscore the importance of developing more comprehensive evaluation frameworks. Traditional evaluation metrics, like FID for visual diffusion models, are useful but insufficient for addressing issues such as overfitting and generalization beyond the training set. Therefore, new metrics, such as Feature Likelihood Divergence (FLD), have been proposed to specifically account for:

• ensuring that generated samples differ from the training samples;

• assessing the quality of the generated samples;

• promoting a wide variety of generated samples.

Empirical evaluations show that FLD effectively reveals overfitting cases where other metrics fail across various datasets and model classes. More recently, InvMM [200] proposes an inversion-based measure that quantifies memorization through sensitive latent noise distributions, providing a reliable and complete assessment across multiple datasets.

We complement the above discussion with a controlled ablation on classifier-free guidance (CFG) scale, a key inference-time parameter known to amplify memorization [193]. Table 3 reports intra-set duplication rates and CLIP text-image alignment scores across CFG values from 1.0 to 15.0 on Stable Diffusion v1.5 [8]. We observe a clear monotonic trend: as CFG increases from 1.0 to 15.0, the duplication rate at \(\tau=0.9\) rises from 2.2% to 85.6%, while CLIP text-image score improves from 0.274 to 0.327. This trade-off between fidelity and replication risk has direct implications for safe deployment.

3.8pt

5.3 Theory↩︎

Beyond the straightforward understanding of the replication phenomenon from the data and methods perspectives, some researchers [43], [201]–[205] offer formal and theoretical explanations using various mathematical theories, such as probability and information theory. In this section, we illustrate these theories in Fig. 3 (c) and provide a brief review as detailed below:

1. Near access-freeness. The authors [201] introduce the concept of “near access-freeness” (NAF) and provide bounds on the probability that a model will generate protected content.

2. Dichotomy. This study [202] examines the generalization capabilities of diffusion probabilistic models, introducing the “memorization-generalization dichotomy”. The key finding is that these models generalize well when they fail to memorize their training data.

3. Geometry-adaptive. This paper [203] explores how the generalization properties of diffusion models can be attributed to their use of geometry-adaptive harmonic representations and argue that these representations allow the models to adapt to the underlying geometric structures of the data.

4. Data-(in)dependent. The authors [204] introduce a framework to estimate the Kullback-Leibler (KL) divergence between the learned and target distributions, providing both data-independent and data-dependent bounds.

5. Mutual information. This paper [43] defines generalization in terms of mutual information between the generated data and the training set, suggesting that models generating data with less correlation to the training set exhibit better generalization.

6. Creativity. Theoretically, the authors [205] discuss various dimensions of creativity, including originality, flexibility, and elaboration, and analyze how current AI technologies perform in these areas.

7. Implicit dynamical regularization. The NeurIPS 2025 Best Paper [206] identifies two distinct timescales: an early time \(\tau_\text{gen}\) at which models begin generating high-quality samples, and a later time \(\tau_\text{mem}\) beyond which memorization emerges. Crucially, \(\tau_\text{mem}\) increases linearly with training set size \(n\) while \(\tau_\text{gen}\) remains constant, creating a growing window where models generalize.

8. Overestimation dynamics. The work [207] shows that memorization is driven by the overestimation of training samples during early denoising, which reduces diversity, collapses denoising trajectories, and accelerates convergence toward the memorized image.

9. Balanced representation. The study [208] proves that memorization corresponds to the model storing raw training samples in learned weights (yielding localized spiky representations), whereas generalization arises when the model captures local data statistics (producing balanced representations).

10. Reproduction quantification. The work [209] quantifies the ease of reproducing training data by measuring the volume growth rate in the ODE that projects images to latent space, enabling detection and modification of easily memorized samples at low computational cost.

11. Ambient diffusion for creativity. The study [210] shows that experimentally observed creativity in diffusion modeling happens when models fail to perfectly minimize their training loss, and proposes ambient diffusion as a principled approach to creative generation.

12. Score smoothing. The work [211] shows that memorization stems from sharp softmax weights in empirical score functions that let individual training samples dominate, and proposes noise unconditioning and temperature smoothing techniques that provably promote generalization.

13. Memorization in diffusion language models. The study [212] provides a unified probabilistic framework proving that increasing sampling resolution strictly increases training data extraction probability, while showing that diffusion language models substantially reduce PII leakage compared to autoregressive models.

6 sec:Mitigation↩︎

After we unveil and understand the replication phenomenon in visual diffusion models, the final and most crucial step is to design strategies to mitigate these issues. Mitigation means avoiding the (un)intentional leakage of training data through model outputs. To effectively finish that, it is essential to employ a multifaceted approach that encompasses both data management techniques and algorithmic innovations. Specifically, as shown in Fig. [Fig:32overview], in this section, we explore mitigation strategies through training data optimization, machine unlearning, and prompt disturbing. Beyond that, we also review some novel perspectives towards addressing this issue. The illustration of these aspects is shown in Fig. 4.

6.1 Training Data Optimization↩︎

Since data is the direct cause of replicating biased concepts, copyrighted and private content, and artwork styles, optimizing training datasets becomes crucial for mitigating replication in visual diffusion models. As shown in Fig. 4 (a), based on current key interests, we categorize training data optimization, i.e., strategies that modify the training data itself to reduce replication, into four main areas: deduplication, protection, purification, i.e., the removal of copyrighted or privacy-sensitive samples from training sets, and corruption, i.e., intentionally adding noise or distortion to training data so that models learn general patterns rather than memorizing specific samples.

Deduplication. Deduplication involves identifying and removing duplicate or near-duplicate data entries within training datasets. This process is essential to ensure a diverse training dataset and prevent models from overfitting to repetitive patterns. Techniques such as hashing, semantic analysis, and clustering are typically used to detect duplicates based on exact matches or semantic similarities. For visual diffusion models, deduplication can be particularly challenging due to the scale and complexity of training data. Approaches like [48], [49], [213] leverage embeddings from pre-trained models like CLIP [110] to perform semantic deduplication, which not only identifies exact duplicates but also uncovers semantically similar image entries, thereby refining the dataset more effectively. Furthermore, [31], [188] focus on the deduplication of captions, highlighting how unique texts can influence the diversity of generated images. Beyond these, the paper [214] proposes a novel dual fusion enhancement method to simultaneously deal with image and captions. Initially, it introduces a generality score to measure caption generality and employs a large language model to generalize training captions. The method then leverages these generalized captions along with a new dataset to enhance image and text diversity and randomness, effectively reducing potential duplication in the fine-tuning dataset.

Protection. This involves protective measures for images or videos to prevent misuse or unauthorized imitation by visual diffusion models. A typical protection involves adversarial examples, which ensures that visual diffusion models cannot accurately learn or reproduce training data. For instance, the authors [215] discuss advanced strategies for generating adversarial attacks that disrupt the latent diffusion model’s ability to generate accurate outputs. The concept of unlearnable examples [216] is also proposed to add specially crafted noise to data to make it unlearnable by diffusion models. [217] is proposed to use score distillation sampling in conjunction with projected gradient descent to perturb images, thereby protecting them from unauthorized use. By embedding watermarks and crafting adversarial perturbations, this study [218] not only prevents unauthorized replication but also ensures that any reproduced images visibly indicate their protected status.

Although these adversarial example techniques are useful for general protection purposes, they are not specifically designed for personalized or customized visual diffusion models, which may brings suboptimal protection performance in this area. With the increasing prevalence of these models, such as DreamBooth [162], and the ethnic concerns they bring, there is a growing number of papers focusing on developing adversarial techniques to combat unauthorized customization of visual diffusion models. These adversarial methods are crucial for ensuring that personal and copyrighted images are not replicated by these powerful AI frameworks. The utilized techniques include:

• subtle imaging perturbations [219]–[223], which involve making minor adjustments to an image that are imperceptible to the human eye but disrupt the AI’s ability to learn from these images effectively;

• adversarial watermarking [224], [225], which embed specific patterns into images that can degrade output quality when the watermarked images are used to train a model.

One of the most controversial applications of personalization technology is its ability to mimic artworks created by contemporary artists. This unethical practice undermines the significant time and effort that artists invest in developing their unique styles. Consequently, several measures have been proposed specifically to prevent the mimicking of artworks. For instance, researchers have developed methods like PAG [226], Glaze [174], MAMC [227], and soft restriction strategy [228], which apply nearly imperceptible distortions to images before they are shared online, misleading AI models that attempt to mimic the artist’s style. Additionally, the study [229] introduces adversarial examples as a way to protect paintings. By generating adversarial examples that are visually similar to the original paintings but are designed to mislead diffusion models, this method effectively prevents visual diffusion models from replicating the artwork’s style.

Beyond these protective methods, differential privacy [230] also helps reduce the risk of visual diffusion models replicating training data. Differential privacy is a technique to enhance the privacy of a dataset by adding noise to the data, which prevents the exact inference of individual information from released data. [45] and [46] were among the first to introduce the concept of differential privacy into visual diffusion models. Recently, Normalizing Flows [231] are used to model and analyze data while implementing differential privacy to enhance data protection; MPCPA [232] explores a multi-center privacy computing framework; DP-RDM [233] adapts diffusion models to private domains without fine-tuning; and DP-LoRA [47] integrates low-rank adaptation modules under differential privacy constraints, achieving competitive FID scores with strict privacy budgets (\(\varepsilon \leq 10\)). Two-stage approaches [234] further optimize the privacy budget by decoupling training into privacy-encoding and non-privacy diffusion stages.

Although these protective measures show effectiveness in their respective settings, they have limitations. Research [235] indicates that while adversarial perturbations can protect data, advanced methods like destruction-restoration can remove these perturbations, allowing the diffusion models to function normally with protected data. Similarly, [236] reveals that although protective perturbations can safeguard images, their effectiveness can be compromised by advanced diffusion models which can adapt and mitigate these protections. The study [237] exposes the vulnerabilities in probabilistic copyright protection, demonstrating how repeated interactions can significantly amplify the probability of generating infringing content. Furthermore, [238] highlights that existing methods like GLAZE [174], which introduce imperceptible perturbations, can be detected and neutralized by sophisticated AI models, rendering these protections ineffective over time. Most recently, LightShed [239] demonstrates that a trained DNN can detect and remove adversarial perturbations from tools like Glaze and Nightshade with 99.98% accuracy, fundamentally challenging the viability of perturbation-based protection. On the defensive side, CopyrightShield [240] proposes a framework to detect poisoned training samples using spatial masking and data attribution, while CopyJudge [241] leverages large vision-language models to simulate court processes for automated copyright infringement identification. Therefore, future efforts should focus on creating more adaptive, robust, and multi-layered protection mechanisms that can withstand the increasing capabilities of modern AI tools.

Table 7 compiles published results from representative data protection and differential privacy methods. For adversarial protection, Glaze [174] achieves \(>\)​92% artist-rated protection success, but LightShed [239] demonstrates 99.98% accuracy in detecting and removing such perturbations. Anti-DreamBooth [220] effectively disrupts identity replication (FDFR 0.63–0.76) but at the cost of visible perturbation artifacts. For differential privacy, DP-LoRA [47] achieves competitive FID at strict privacy budgets (\(\varepsilon \leq 10\)), representing \(>\)​35% improvement over prior SOTA [45]. The table highlights a fundamental tension: stronger protection degrades either visual quality or generation utility.

Purification. Purification involves the removal of undesirable samples from training datasets, particularly those containing copyrighted or privacy-sensitive content. This process is essential to ensure that even if visual diffusion models replicate data, they do not pose security or legal risks. While this method effectively addresses the issue from its roots, its adoption remains limited due to the complexity and time-consuming nature of the process. The CommonCanvas [242] initiative tackles this challenge by assembling a dataset of Creative Commons (CC)-licensed images along with corresponding high-quality synthetic captions. The models trained on the CommonCanvas dataset achieve performance comparable to Stable Diffusion 2 [8] in human evaluations while avoiding the typical copyright issues. In the artistic creation area, the article [243] introduces innovative methods for creating new artistic styles using models trained solely on natural images, thereby avoiding any claims of copying existing human art styles.

Corruption. Corruption refers to data samples that have been altered, typically due to noise or other forms of distortion, making them different from their true, clean distribution. Leveraging visual diffusion models to learn from corrupted data can be beneficial for reducing data replication and enhancing privacy. This is because these models are able to learn general data patterns in the absence of specific individual samples. To learn from these corrupted data, the methodologies involve introducing additional distortions [244] or using sophisticated statistical formulas [245].

6.2 Machine Unlearning↩︎

Machine unlearning [246] is a process designed to remove specific data or concepts from a machine learning model, effectively making the model “forget” particular information without needing to retrain from scratch. As shown in Fig. 4 (b), in the context of visual diffusion models, machine unlearning plays a vital role in mitigating the issues of replication of specific concept, content, and style [51], [247], [248]. Specifically, the studies [50], [178], [249]–[252] emphasizes the significance of choosing cross-attention-related parameters to fine-tune for effective erasure. Focusing on gradient, SalUn [253] utilizes gradient-based weight saliency to improve the limitations of traditional machine unlearning methods, aiming to enhance accuracy, stability, and cross-domain applicability of the unlearning process. Utilizing continual learning, Selective Amnesia [254] explores how to selectively forget concepts in deep generative models.

There are also some works focusing on specialized aspects or applications. The paper [255] discusses the application of machine unlearning techniques in image-to-image generative models. Regarding defending against unexpected user inputs: Espresso [256] is the first method to robustly remove unacceptable concepts; Task Vectors [257] have been shown to be more robust compared to input-dependent erasure methods; and [258] proposes the use of pruning techniques to enhance the model’s robustness. [259] and [260] are specifically designed to use machine unlearning to mitigate unsafe content generation and enhance copyright protection, respectively.

Beyond traditional machine unlearning methods that focus on erasing single concept at a time, recent advancements move towards more comprehensive approaches that aim to modify, erase, or refine multiple concepts simultaneously within diffusion models. For instance, UCE [261] can handle multiple concept editing tasks simultaneously, such as debiasing, style erasure, and content moderation. SDD [262] effectively reduces the proportion of harmful content generated by aligning the conditional noise estimate with an unconditional one and allows for the removal of multiple concepts simultaneously. SepME [263] flexibly erases or recovers multiple concepts while preserving overall model performance. MACE [264] and EMCID [265] scale up to handle the erasure of 100 and 1,000 concepts, respectively, while maintaining the integrity of other non-edited concepts.

More recent advances address fundamental limitations. Data unlearning [52] proposes the SISS loss family with theoretical guarantees, successfully mitigating memorization on nearly 90% of tested prompts. Meta-unlearning [266] prevents relearning of erased concepts through a meta-learning objective. T2VUnlearning [267] extends concept erasure to text-to-video models, while [268] pursues irreversible unlearning that cannot be undone by further fine-tuning. Differential Vector Erasure [269] introduces training-free concept erasure for flow matching models. ScaPre [270] achieves scalable concept unlearning by identifying concept-relevant parameters via spectral trace regularization, forgetting up to 5\(\times\) more concepts than prior methods. Prompt-free instance unlearning [271] enables removing specific outputs without requiring text-prompt specification. RAPTA [272] reduces memorization during training through region-aware prompt augmentation combined with multimodal copy detection.

Although these unlearning methods are effective to some degree, some evaluations question their reliability and indicate that they are susceptible to jailbreaking. For instance, UnlearnCanvas [273] includes high-resolution, stylized images that allow researchers to effectively test and quantify the unlearning of artistic painting styles and associated image objects. The paper highlights shortcomings in existing machine unlearning evaluation methods, such as a lack of diverse unlearning targets, lack of evaluation precision, and a lack of systematic study on retainability. Additionally, the study [274] shows that special learned word embeddings can retrieve supposedly erased concepts from sanitized models without needing to alter the models’ weights. Furthermore, some prompts are also designed for testing the reliability of deployed safety mechanisms:

• UnlearnDiff [275] leverages the inherent classification capabilities of visual diffusion models to simplify the generation of adversarial prompts;

• Ring-A-Bell [276] first performs concept extraction to gain comprehensive representations of sensitive and inappropriate concepts, then uses these concepts to automatically select problematic prompts;

• Researchers in [277] combine multiple prompts to reconstruct the vector responsible for target concept generation, even when direct computation of this vector is infeasible.

In addition, [53] systematically demonstrates the unstable nature of machine unlearning in diffusion models, showing that erased concepts can re-emerge under slight perturbations. JailbreakDiffBench [278] and the Holistic Unlearning Benchmark (HUB) [279] provide comprehensive evaluation frameworks covering diverse concepts and attack vectors. The study [280] reveals that erasing a concept does not necessarily erase child concepts, and that superclass erasure can be circumvented through subclasses.

6.3 Prompt Disturbing↩︎

As illustrated in Fig. 4 (c), the term “prompt disturbing” refers to the intentional modification of user inputs to prevent the model from merely replicating memorized patterns or details from its training data. These modifications include direct change to the original user prompts. For instance, [175] alters specific terms or removing elements from prompts to decrease direct ties to memorized data and promote a broader range of creative outputs. Negative prompts [30] are introduced to guide a visual diffusion model to avoid producing certain elements, thereby encouraging more original creations that are not simply replication of its training data.

Beyond that, some methods further disturb prompt-related components in the visual diffusion models to avoid replication. For instance, ProtoRe [281] incorporates language-contrastive knowledge to identify prototypes of negative concepts, which are then used to extract and eliminate undesirable features from outputs. Degeneration Tuning [282] is proposed to disrupt the correlation between undesired textual concepts and their corresponding image domains. [283] works by optimizing text embeddings during inference time to better control the image content generated from textual descriptions.

3.8pt

6.4 Novel Perspectives↩︎

Beyond these common mitigation methods for the replication phenomenon, some novel perspectives can be explored to further reduce these issues within visual diffusion models. As shown in Fig. 4 (d), these novel perspectives [188], [284]–[286] aim to tackle the underlying causes of replication by diversifying the training approaches and incorporating principles from other domains of machine learning and data security:

1. Composition. This research [284] allows different diffusion models to be trained on separate data sources and arbitrarily composed at inference time. Each model only contains information about the subset of the data it was exposed to during training, which effectively prevents the leakage of training data.

2. Model immunizing. The article [285] discusses how to mitigate replication by improving learning algorithms to reduce the risk of malicious adaptation. Malicious adaptation refers to the behavior of fine-tuning visual diffusion models to produce harmful or unauthorized content. The article proposes an approach called IMMA, which mainly modifies the parameters of the pre-trained model using a bi-level optimization strategy.

3. Low-rank adaptation. This paper [286] discusses how to apply Low-Rank Adaptation (LoRA) in diffusion models while reducing the risk of Membership Inference Attacks (MIA). These attacks can identify whether specific data belongs to the training dataset, leading to severe privacy leaks. To address this challenge, researchers introduced a new method called PrivateLoRA, which uses a min-max optimization strategy to balance the model’s adaptation loss and the MIA gain of a proxy attack model.

4. Despecification guidance, i.e., reducing the specificity of text prompts during inference to prevent overly similar outputs to training data. This approach [188] attempts to diminish the specificity of text prompts in guiding the inference process, thus decreasing the model’s dependency on specific inputs and preventing overly similar outputs to training data. Specifically, the method starts with a noised image or latent-space representation and predicts noise at a given time to infer the original image. It then calculates the similarity between the inferred image and the closest neighbor in the training set. This similarity is used to adjust the scaling of epsilon predictions to reduce alignment with the prompt-conditioned prediction.

5. Replication-aware architecture. LoyalDiffusion [54] identifies that skip connections in the U-Net enhance image quality but also reinforce memorization, and proposes a Replication-Aware U-Net (RAU-Net) that incorporates information transfer blocks into skip connections selectively at critical timesteps, achieving a 48.63% reduction in replication while maintaining image quality.

6. Anti-memorization guidance. Anti-Memorization Guidance (AMG) [287] modifies the sampling process of diffusion models at inference time to discourage memorization.

7. Privacy-utility enhancement. [288] proposes prompt re-anchoring and semantic prompt search to enhance privacy-utility trade-offs, mitigating memorization while preserving text alignment in generated images.

Table 4 consolidates representative quantitative results from the mitigation literature, enabling a direct comparison of strategies on common dimensions. Several observations emerge. First, deduplication is the most computationally efficient strategy and reliably reduces content-level replication, but it cannot address concept- or style-level replication that arises even without near-duplicate images. Second, differential privacy (DP) provides the only formal per-sample privacy guarantee, but the utility-privacy tradeoff is steep: the FID penalty grows rapidly as the privacy budget \(\varepsilon\) decreases. Third, machine unlearning is highly targeted and preserves overall model quality, but it provides only a soft guarantee and is susceptible to re-extraction attacks. These complementary tradeoffs suggest that a defense-in-depth approach, i.e., combining deduplication at the data level, DP at the training level, and unlearning at the deployment level is more robust than any single strategy.

7 Influence↩︎

The discussion of replication’s influence in the real world (regulation, art, society, healthcare, and emerging applications) is in the Supplementary Material, Sec. 14.

8 Challenges and Future Directions↩︎

The discussion of current challenges and future directions in this field is in the Supplementary Material, Sec. [sec:Challenges].

9 sec:Conclusion↩︎

This paper presents a comprehensive and methodical examination of the replication phenomenon within visual diffusion models. We begin by concisely defining replication, establishing a clear understanding of the concept. Subsequently, we review papers focusing on this phenomenon from the perspectives of unveiling (methods to detect and reveal occurrences), understanding (analyzing the underlying causes), and mitigation (strategies to address and resolve issues). Furthermore, we discuss the influences of replication in the real world. We conclude by highlighting persistent challenges in this domain and proposing potential directions for future investigation. We view this survey as an initial step towards advancing academic research on replication in visual diffusion models and enhancing AI security efforts.

10 Replication in Video and 3D Diffusion Models↩︎

While the majority of existing work targets image diffusion models, the replication phenomenon is equally concerning in emerging video and 3D diffusion models. We briefly outline the unique challenges and early findings in these domains.

Video diffusion models. Video diffusion models [289]–[294] extend image generation to the temporal domain and have attracted enormous research interest in 2024–2025. The training data for these models typically consists of video clips sourced from the web, where temporal redundancy is far greater than spatial redundancy in images: the same scene or person often appears across thousands of frames or even multiple clips. This redundancy amplifies replication risk along two dimensions. First, clip-level replication, i.e., the reproduction of an entire short video sequence rather than a single frame, is harder to detect than single-frame replication because standard copy-detection pipelines operate frame-independently. [148] takes an early step in this direction by adapting frame-level similarity retrieval to the video domain, and [295] confirms that memorization is widespread across all tested video diffusion models, but a principled clip-level replication metric remains an open problem. Second, subject identity replication, i.e., the reproduction of a specific person’s face or body across generated frames, poses heightened privacy risks, since videos carry richer biometric information than still images.

3D diffusion models. 3D diffusion models [296], [297] trained on geometric datasets (e.g., ShapeNet [298]) face a distinct form of replication: not only can surface textures be replicated (analogous to content-level image replication), but the underlying geometry, such as mesh topology, volume structure, can also be memorized and reproduced. This compounds the risk: a generated 3D asset may replicate copyrighted geometry even when its texture appears novel. Existing replication metrics defined for 2D images (e.g., SSCD [109]) do not directly apply to 3D representations, necessitating new distance metrics for point clouds, meshes, and implicit neural fields. Recent work [299] provides the first empirical study of memorization in 3D shape generation, finding that memorization depends on data modality, increases with data diversity and finer-grained conditioning, and can be mitigated by rotation augmentation.

11 Detailed Benchmark Tables↩︎

11.1 Membership Inference Attacks Benchmark↩︎

3.8pt

11.2 Memorization Extraction and Detection↩︎

3.8pt

11.3 Data Protection and Differential Privacy↩︎

3.8pt

11.4 Machine Unlearning Benchmark↩︎

3.8pt

12 Cross-Cutting Conclusions on Data Properties↩︎

From the above findings, we draw several cross-cutting conclusions about how dataset properties influence the degree of replication.

1. Scale matters, but non-linearly. [187] shows that the replication ratio decreases as dataset size grows, but the relationship is sublinear: halving the dataset more than doubles the per-image replication probability. In the extreme small-data regime (as in medical imaging [60], [300]), replication becomes near-certain.

2. Duplication rate is a stronger predictor than raw size. [31] demonstrates that a large dataset with high duplication (common in web-scraped corpora) can exhibit higher replication rates than a smaller but carefully deduplicated dataset. This suggests that data cleaning, not merely data collection, is the primary lever for controlling replication.

3. Caption specificity interacts with duplication. [42], [187] find that unique or inaccurate captions exacerbate replication when paired with duplicated images, because the model cannot distribute its capacity across diverse captions. Conversely, diverse captions partially compensate for moderate image duplication.

4. Data type determines which semantic level is replicated. Web images with repeated subjects (e.g., celebrity photos) predominantly cause content-level replication; stylistically homogeneous artistic datasets (e.g., a single artist’s portfolio) cause style-level replication; and datasets with strong demographic skew cause concept-level replication via biased associations.

These insights collectively motivate the mitigation strategies reviewed in Section [sec:mitigate]: deduplication targets factor (ii), caption diversification targets factor (iii), and differential privacy targets factor (i) by bounding the per-sample influence.

13 Limitations and Deployment Challenges of Machine Unlearning↩︎

Limitations and deployment challenges. Although the machine unlearning methods reviewed above show promise, several fundamental challenges remain before they can be reliably deployed in practice.

1. Catastrophic forgetting. Aggressive unlearning of a target concept often degrades the model’s ability to generate semantically related but benign content. For example, erasing a specific artist’s style may impair generation of broader artistic styles. UnlearnCanvas [273] empirically demonstrates that many methods fail to balance retainability with erasure.

2. Incomplete unlearning. Memorized content may persist in model weights even after targeted unlearning: [274] shows that learned word embeddings can re-elicit erased concepts without modifying the model’s weights, and [277] demonstrates that combining multiple prompts can reconstruct erased concept vectors. This implies that current unlearning methods offer only a soft guarantee.

3. Scalability constraints. Most methods require access to the original training data or per-sample gradients. In API-only deployment settings, neither is available. Furthermore, MACE [264] and EMCID [265] show that even scaling to 1,000 concepts introduces measurable degradation in unrelated generation quality.

4. Verification difficulty. There is no reliable protocol to certify that unlearning is complete. Evaluation typically relies on held-out prompts, which can be circumvented by adversarially designed queries [275], [276]. A standardized auditing framework, analogous to security penetration testing, is needed but absent.

5. Regulatory uncertainty. It remains legally unclear whether successful machine unlearning satisfies “right to be forgotten” requirements under GDPR [301] or analogous legislation, since no court has yet adjudicated whether computational forgetting is equivalent to data deletion.

To quantify these trade-offs, Table 8 compiles published results from the UnlearnCanvas benchmark [273] and the Holistic Unlearning Benchmark (HUB) [279], which evaluate representative methods across unlearning accuracy (UA), in-domain retainability (IRA), cross-domain retainability (CRA), and generation quality (FID). Key observations: (i) ESD [50] and UCE [261] achieve the highest UA (\(>\)​98%) but suffer significant IRA/CRA degradation; (ii) SalUn [253] offers the best retainability (CRA up to 99.6%) at the cost of moderate UA; (iii) CA [51] preserves generation quality (FID 54.2) but has the weakest erasure effectiveness; (iv) no single method dominates across all metrics, underscoring the need for task-specific selection.

14 Influence↩︎

After sequentially discussing the unveiling, understanding, and mitigation of replication in visual diffusion models, as outlined in Fig. [Fig:32overview], this section focuses on its influence in the real world. Specifically, as illustrated in Fig. [Fig:32influence], we focus on regulation, art, society, and healthcare. This involves opinions from legal scholars, artists, sociologists, and doctors.

14.1 Regulation↩︎

As shown in Fig. [Fig:32influence] (a), training and generating processes in some visual diffusion models raise significant law issues due to the replication of copyrighted materials. As these models become more powerful and prevalent, an increasing number of legal scholars are focusing on this area. They primarily investigate how these models manage and utilize copyrighted materials during the creation process, along with the challenges and implications for the existing copyright law framework. For instance, they [55], [56], [302]–[307] question whether using copyrighted works as training data for AI constitutes copyright infringement, whether AI-generated outputs are derivative works infringing on the original copyrights, and who owns the copyright for AI-generated works. Furthermore, [305], [308] discuss the intricate infringement challenges that arise when generative AI models, particularly visual diffusion models, are trained using copyrighted materials without proper authorization. Additionally, [309] aims to define and clarify what constitutes replication from the perspective of copyright infringement; [310] thoroughly explores the intersection of copyright law and economic principles in the context of rapid technological advancements; and the core idea of [311] is to evaluate whether privacy protection measures can align with and support copyright law.

3.8pt

Unveiling through litigation. Several landmark lawsuits have acted as “whistleblowers” that expose how diffusion models replicate training data at scale. In Andersen v.Stability AI [313], a class of visual artists alleged that Stable Diffusion [8] was trained on their copyrighted works without consent, and that its outputs constitute infringing derivative works. Similarly, Getty Images v.Stability AI [314] focuses on the near-verbatim replication of Getty’s watermarked photographs, with watermark fragments visible in generated outputs serving as forensic evidence of content-level replication. These cases operationalize replication detection as a legal evidentiary tool. In November 2025, the UK High Court ruled in Getty Images v.Stability AI [315] that although the model is exposed to copyrighted works during training, it does not store the training data itself, rejecting the copyright infringement claim, a landmark decision with significant implications for the field. Meanwhile, the U.S. Copyright Office concluded [316] that AI developers using copyrighted works to train models generating “expressive content that competes with” original works exceed the scope of fair use.

Understanding applicable legal doctrines. The central doctrinal question varies by jurisdiction. [305], [307] analyze whether AI training constitutes fair use under US copyright law: while the transformative-use factor may favor developers, the commercial nature of outputs and market substitution weigh against it. [302] empirically demonstrates that foundation models can reproduce verbatim training data under adversarial prompting, strengthening the plaintiffs’ case. In the EU, the Copyright in the Digital Single Market Directive [317] introduced a text-and-data mining (TDM) exception (Article 4), but rights holders can opt out; the legal validity of mass opt-out mechanisms for AI training remains unsettled [310].

Mitigation through regulatory compliance. Effective compliance strategies are emerging in response to this regulatory landscape:

1. Opt-out and licensing mechanisms. Platforms such as DeviantArt (NoAI tag) and Adobe Stock (opt-in only) have begun implementing opt-out metadata standards. [302] proposes a structured approach to auditing whether such opt-outs are honored at training time.

2. Training data disclosure. The EU AI Act [318] requires providers of general-purpose AI models to publish summaries of training data, enabling rights holders to identify unauthorized use [319], [320]. As of August 2025, every GPAI provider must maintain documentation of training data provenance and prove compliance with EU copyright rules. This creates an institutional incentive for replication auditing analogous to the methods surveyed in Section 4.

3. GDPR “right to be forgotten.” Personal data replicated by diffusion models falls under GDPR Article 17. Machine unlearning is the primary technical mechanism proposed to implement this right, though the legal sufficiency of computational forgetting remains untested in court.

International comparative analysis. The regulatory landscape is highly heterogeneous across jurisdictions, as summarized in Table [tab:regulation]. The key divergences concern (i) whether training on copyrighted data is prima facie infringing, (ii) whether opt-out mechanisms are legally recognized, and (iii) whether AI-generated outputs can themselves be copyrighted.

Beyond the copyright issues, there are also privacy concerns and corresponding data protection regulations [321], [322]. The replication of data by visual diffusion models can pose significant privacy risks, especially when the models inadvertently replicate sensitive or personal data. This contravenes data protection regulations such as the General Data Protection Regulation (GDPR) [301] in Europe, which mandates the protection of personal data with appropriate technical measures. Regulatory frameworks ensure that AI systems, particularly those trained on vast amounts of potentially sensitive data, comply with privacy regulations and do not retain or reproduce personal data without consent.

The replication of biases in training data by AI models is another regulatory concern [321], [323], [324]. Ensuring that diffusion models do not perpetuate or amplify biases present in the data they are trained on is crucial. Regulations enforce fairness, accountability, and transparency in AI systems to mitigate these issues. This could involve mandatory bias audits, transparency in data usage, and clear documentation of the data and methodologies used in training AI models.

14.2 Art↩︎

The influence of generative AI in art worlds presents both opportunities and challenges. These models have transformed the art market, personalizing the buying experience and enhancing the efficiency of curators in identifying trends and managing collections [325]. Despite these advances, as shown in Fig. [Fig:32influence] (b), many artists fear that AI may threaten their jobs and dilute the authenticity of art by replicating styles and producing art without human involvement [57], [326]. This fuels the ongoing debate about whether art can exist without an artist [327], [328]. Additionally, some researchers [329] are investigating artistic copyright infringements, underscoring the complex challenges in protecting intellectual property because artistic style itself is not copyrightable [58].

Unveiling style replication in art. Several real-world controversies have emerged that concretely illustrate how diffusion models replicate artistic style. (i) Hollie Mengert. The illustrator Hollie Mengert became one of the first documented cases in which a diffusion model (NovelAI, which built on Stable Diffusion [8]) was used to generate images in a style closely matching her distinctive flat, graphic work. This case exemplifies style-level replication: no individual image was directly copied, but the statistical signature of her style, as measurable by style-similarity metrics [34], [35], was reproduced at will. (ii) DeviantArt / NovelAI controversy. In 2022, NovelAI released an image-generation model trained on ArtStation and DeviantArt images without artist consent. Artists organized mass protests by flooding ArtStation with “No AI Art” posts. This incident prompted ArtStation and DeviantArt to introduce opt-out mechanisms and “AI-generated” content labels. (iii) Empirical style-replication measurement. [34] provides the first large-scale quantitative study of style replication, measuring cosine similarity in a style-aware feature space between LAION-trained model outputs and training images. They find that style replication is prevalent even when content is not directly copied, and that replication rate correlates with training dataset duplication frequency.

Understanding artistic style vulnerability. Several factors make style-level replication particularly pervasive in artistic domains: (i) Style is not copyrightable, creating a legal gap [58]: even if a model demonstrably replicates an artist’s style, current copyright law in most jurisdictions provides no remedy. (ii) Stylistically homogeneous training sets. When a model is fine-tuned on a single artist’s portfolio (e.g., via DreamBooth [162]), the resulting model can generate near-perfect style replications on demand. [329] analyzes this phenomenon and questions whether style fine-tuning constitutes a form of creative appropriation that current legal frameworks fail to address. (iii) CLIP-based training amplifies style coherence. Because CLIP encodes stylistic associations in its embedding space, models trained with CLIP guidance tend to replicate style clusters present in the training distribution more faithfully than unconditional models.

Mitigation through style protection. Mitigation strategies for artistic style replication mirror those for content replication but require style-aware tools: (i) Style-adversarial perturbations (Glaze [174]). Glaze adds imperceptible perturbations to an artist’s images before public release, shifting their style-space representation to confuse fine-tuning pipelines while remaining visually faithful to the artist’s intent. (ii) Opt-out metadata and content filtering. Platforms now support opt-out flags (e.g., DeviantArt’s NoAI tag), and some model providers filter style-specific fine-tuning requests. (iii) Regulatory advocacy. Artists have lobbied for amendments to copyright law to recognize style as a protectable element under certain conditions, particularly when a style is uniquely associated with a specific living artist [326]. (iv) Arms race. However, LightShed [239] demonstrates that perturbation-based protections like Glaze can be defeated with 99.98% accuracy by a trained DNN, suggesting that the current generation of adversarial protection tools may not provide permanent solutions and that the protection–attack arms race will continue to escalate.

14.3 Society↩︎

From a societal perspective, as shown in Fig. [Fig:32influence] (c), the phenomenon of replication in visual diffusion models manifests in the duplication of human values, ideals, and even biases within generated images or videos. Much of the research in this area focuses on biases, as these can reinforce and amplify societal inequalities and discrimination. Different from the biases discussed in the Regulation subsection, we review papers from a societal perspective here.

Unveiling societal bias replication. Several studies have systematically measured the extent to which diffusion models replicate societal biases. [29] introduces a method for assessing social biases by analyzing how varying input prompts related to gender, ethnicity, and professions influence the diversity of generated images. [330] demonstrates at large scale that ordinary prompts describing traits, occupations, and objects produce amplified demographic stereotypes in text-to-image models, with stereotypes persisting even after user-level counter-stereotype prompting. [331] probes gender and skin-tone biases across professions, revealing that models learn specific demographic associations from web image-text training pairs. [332], [333] specifically audit gender representation in text-to-image models, emphasizing how they reinforce gender stereotypes. More recently, [334] proposes the first open-set bias detection pipeline using an LLM to hypothesize biases and a VQA model to verify them in generated images, discovering novel biases across Stable Diffusion 1.5, 2, and XL. [335] further demonstrates that popular Stable Diffusion models respond to harmful prompts by generating content with troubling biases such as the disproportionate portrayal of certain racial groups in violent contexts.

Understanding bias origins. The root causes of bias replication can be traced to multiple factors. [59] highlights that diffusion models exacerbate biases present in their training data, with the effect depending on dataset size and composition. [336] introduces sociologically grounded stereotype scores and internal probing tools, showing that despite improved fidelity, newer models retain strong stereotypical predispositions; stereotypes are worse for nationalities with lower Internet footprints, tracing the root cause to training data distribution imbalances. [337] investigates feedback loops where generated images train future models; surprisingly, substituting real images with diffusion outputs does not uniformly amplify bias and can sometimes mitigate it, attributing this to generation artifacts that disrupt learned associations. These findings collectively suggest that bias replication originates from the interaction between skewed training data distributions, CLIP-based text encoders that encode societal associations, and the high-fidelity generation capability that faithfully reproduces these patterns.

Mitigation through debiasing. Several strategies have been proposed to reduce bias replication in diffusion models. [338] proposes a distributional alignment loss with adjusted direct finetuning, markedly reducing gender, racial, and intersectional biases. [339] introduces TIME (Text-to-Image Model Editing), which corrects biased implicit assumptions (e.g., “a CEO” always generating a white male) by editing only \(\sim\)​2% of model parameters in under one second, without any retraining. [340] proposes learning latent representations that promote fairness without requiring predefined sensitive attributes. Most recently, [341] identifies internal “bias features” within diffusion model architectures via mechanistic interpretability and directly manipulates them to achieve granular control over bias levels, bridging both understanding and mitigation within a unified framework.

14.4 Healthcare↩︎

Visual diffusion models have significantly impacted the field of healthcare by enhancing the generation and analysis of medical images, which are critical tools in diagnosis, treatment planning, and research.

A primary way diffusion models assist in medical imaging is through the generation of synthetic images [342]–[344]. These models can create realistic medical images, such as MRI scans or X-rays, from a dataset of existing images. This capability is particularly useful for training medical professionals, as it allows for the creation of diverse scenarios and conditions that might not be readily available in educational settings due to rarity or ethical concerns. Additionally, synthetic images can augment datasets used to train other machine learning models, improving their ability to recognize and diagnose conditions from real patient data.

Furthermore, diffusion models can enhance image quality and detail [21], [345], [346], which is vital in medical diagnostics where the clarity of an image can influence the accuracy of assessments made by radiologists. For example, diffusion models can refine images, improving resolution and contrast, or even reconstruct incomplete scans. This enhances the interpretability of medical images and assists in more accurate diagnosis and patient monitoring.

Moreover, visual diffusion models support the development of automated diagnostic tools [347]–[349]. By generating high-quality, detailed images, these models aid in training algorithms that can detect anomalies such as tumors, fractures, or degenerative conditions. This speeds up the diagnostic process and helps in reducing human error by providing a consistent, objective analysis that can be used as a second opinion or to verify human-made diagnoses.

As shown in Fig. [Fig:32influence] (d), while visual diffusion models offer significant benefits in medical imaging, such as enhancing image quality and generating scarce datasets, these models also pose substantial risks due to their potential for replication. The replication phenomenon could lead to generated images being overly similar to real patient data, thus risking personal health information disclosure. In the following, we review papers that unveil, understand, and mitigate the replication phenomenon in the context of medical imaging.

Unveiling replication in medical imaging. Several studies have served as “whistleblowers” in highlighting these issues. For instance, the research [60] reveals that 3D latent diffusion models are more prone to replicate original training images, affecting the model’s generalizability. Another study [350] compares diffusion models and GANs in synthesizing medical images, finding that diffusion models, compared to GANs, are more likely to replicate training images when generating 2D slices from 3D volumes, increasing the risk of patient re-identification. Further research in [351] confirms the tendency of latent diffusion models to replicate data in an unconditional generation setting, suggesting that diffusion models might fail to prevent the disclosure of training data details even when not targeted for specific tasks.

Understanding data and training factors. The occurrence of replication in medical imaging, can be attributed mainly to two factors: the size of the original dataset and the number of training epochs. Firstly, when diffusion models are trained on small datasets, there is a higher risk of replication, as the model has fewer examples from which to learn and generalize. Studies such as [60] and [300] have highlighted that models trained on small datasets, such as those containing detailed scans for brain tumors, tend to produce synthetic images that too closely replicate the training images, reducing their utility and increasing privacy risks. Secondly, [352] reveals that over-training a model – running too many epochs – can lead to a situation where the diffusion models begin to precisely replicate the training patient data rather than generating diverse synthetic images. This occurs because excessive training on the same dataset reinforces the model’s exposure to and retention of specific data characteristics, thereby increasing the likelihood of producing identical or nearly identical images to those seen during training.

Mitigation through privacy-preserving techniques. There are effective solutions that can mitigate these risks and thus enhance the privacy and utility of synthetic data. Firstly, the approach of privacy distillation, as discussed in [61] offers a robust method for safeguarding patient information. This technique involves training a diffusion model on real data to generate a synthetic dataset, which is then filtered to remove any potentially identifiable information. A second model is then trained exclusively on this sanitized dataset. Secondly, data augmentation is another approach that can enhance the diversity of training datasets and reduce overfitting. By artificially expanding the dataset through transformations and variations of the original patient images, models are less likely to replicate [60], [352]. Thirdly, differentially private 3D synthesis [353] introduces controllable latent diffusion models under strict DP constraints for volumetric medical images, while comprehensive privacy assessments [354] reveal that leakage risks extend throughout the entire deep learning pipeline, from data sharing to model deployment, necessitating holistic privacy-preserving strategies.

14.5 Emerging Applications↩︎

Beyond healthcare, the replication phenomenon poses unique risks in several rapidly growing application domains. We discuss three representative cases below.

Metaverse and virtual environments. Diffusion models are increasingly used to generate 3D scenes, avatars, and virtual assets for metaverse platforms and game engines [16]. This introduces novel replication risks: (i) 3D asset replication, i.e., the reproduction of copyrighted geometry or textures in generated 3D models. As noted in Section 10, generated 3D assets may replicate the geometry or texture of copyrighted assets in training sets, enabling large-scale plagiarism of virtual goods whose commercial value can be substantial. (ii) Real-space replication, i.e., the reproduction of identifiable physical locations in generated virtual environments. Models trained on street-view or indoor imagery can replicate identifiable real-world locations (homes, offices, medical facilities) within virtual environments, raising both privacy (exposure of private spaces) and security concerns (reconnaissance). (iii) Avatar identity replication, i.e., reproducing a real person’s biometric identity in avatar form. Diffusion models conditioned on facial images can replicate the biometric identity of real individuals in avatar form, enabling identity theft within virtual worlds. This is a concept-and-content dual replication: both the person’s identity (concept) and their facial features (content) are replicated.

Autonomous driving. Diffusion models are increasingly used to synthesize rare driving scenarios for training and testing autonomous vehicle (AV) systems [355], [356]. The replication phenomenon manifests here in safety-critical ways: (i) Pedestrian identity replication, i.e., reproducing recognizable individuals from training data in synthetic traffic scenes. Synthetic traffic datasets may replicate the appearance of real individuals photographed in training data, raising GDPR concerns and potentially causing discriminatory behavior if the model replicates demographic biases in pedestrian recognition. (ii) Scenario memorization, i.e., reproducing the exact spatiotemporal dynamics of real driving incidents rather than generating novel scenarios. When a model is trained on a limited set of accident recordings, it may replicate the exact spatiotemporal dynamics of real incidents rather than generalizing to novel scenarios [187]. This makes the synthetic dataset less diverse than intended and may give a false sense of safety coverage. (iii) Geographic location replication, i.e., reproducing identifiable roads, intersections, or landmarks that reveal proprietary training routes. Models trained on geolocation-tagged datasets may replicate identifiable road segments, intersections, or landmarks, enabling inference of the geographic distribution of the AV company’s proprietary training routes.

Education and scientific research. Diffusion models are increasingly used to generate illustrative figures, data visualizations, and synthetic experimental images in educational materials and scientific publications. The replication phenomenon raises concerns in this domain: (i) Scientific image fabrication. Generated images intended as novel illustrations may inadvertently replicate figures from published papers in the training data, leading to unintentional plagiarism or fabrication allegations. (ii) Training data leakage in synthetic datasets. When diffusion models generate synthetic data for educational or research purposes (e.g., augmenting small datasets for student projects), replicated training samples may introduce bias or violate the privacy of original data subjects. (iii) Erosion of academic integrity. The ease of generating realistic images undermines the evidentiary value of visual data in scientific publications, as reviewers and readers cannot distinguish genuine experimental results from AI-generated replications of prior work.

Mitigating replication in these emerging domains requires domain-specific adaptations of the general strategies, particularly data minimization, differential privacy, and machine unlearning, as well as the development of domain-specific benchmarks. The VideoShield framework [157] provides a first step toward regulation in the video domain through watermark-based provenance tracking, and [267] extends concept erasure to text-to-video models.

15 Challenges and Future Directions↩︎

After reviewing the replication issues in visual diffusion models, including unveiling, understanding, mitigation, and its influence in the real world, this section will discuss the current challenges and future directions in this field.

15.1 Specialized Visual Copy Detection↩︎

Unlike generic image retrieval, specialized visual copy detection refers to detection models specifically designed to identify diffusion-generated replications across content, style, and concept levels. Currently, many research efforts [37], [147]–[151] focus on the analysis of replicated content because this level of replication aligns well with human perceptions of similarity. However, these methods predominantly rely on existing feature extraction models, such as SSCD [109] and CLIP [110], which are not specifically designed for diffusion-based replication. SSCD [109], for instance, only learns invariance against image transformations, while CLIP [110] is developed primarily for natural images. Consequently, these models often fail to detect some forms of replicated content generated by diffusion models. As a result, the analysis of replicated content by visual diffusion models tends to be both inaccurate and biased.

Future efforts could focus on creating a new dataset that includes images and videos featuring various types of replicated content generated by visual diffusion models. This dataset would then be used to train specialized visual copy detection models. By employing these models, subsequent analysis is expected to become both more accurate and fairer.

15.2 In-context Similarity Retrieval↩︎

In-context similarity retrieval refers to adapting a single foundational retrieval model to detect different types of replication (content, style, concept) by providing contextual examples at inference time, without retraining. Replication in visual diffusion models manifest across various dimensions, including gender [27], culture [28], racial aspects [29], NSFW content [30], copyrighted images [31], patient information [32], photos of politicians or celebrities [33], and artistic styles [35]. Current approaches to unveiling this phenomenon typically utilize a spectrum of feature extraction models or purpose-specific models. Although these methods can be effective, they come with significant disadvantages: (1) Selecting suitable models for practical deployment is time-consuming; (2) labeling new datasets and training specialized models are costly; and (3) the models, once trained, often lack generalizability to other contexts, thereby increasing the overall costs of practical applications.

In light of these challenges, introducing in-context learning to this area presents a promising direction for future development. In-context learning, a paradigm where a single foundational model adapts to a variety of tasks based on the context provided during inference, eliminates the need for multiple specialized models. Specifically, for in-context similarity retrieval, this approach could enable the foundational model to dynamically adjust the feature extraction process based on specific concerns, such as gender biases, racial characteristics, or copyrighted content. This is achieved by simply presenting relevant contextual examples, which allows adjusting without the need for retraining. This methodology eliminates the need for selecting/training specialized models and significantly increases generalizability.

15.3 Robust Mitigation↩︎

The current mitigation methods often fail to successfully solve the replication problems. For instance,

• Even after deduplicating images and captions in the dataset, visual diffusion models can still generate samples similar to data points from the training set;

• Malicious researchers can easily develop AI methods to bypass training data protection strategies, as demonstrated by LightShed [239] defeating Glaze with 99.98% accuracy;

• Visual diffusion models equipped with machine unlearning methods can still generate concepts that have already been erased, with recent work [53] showing that erased concepts re-emerge under slight perturbations;

• Prompt perturbation methods cannot always generate images that align well with the prompt while simultaneously avoiding the creation of copyrighted content.

Therefore, in the future, researchers can (1) focus on enabling visual diffusion models to learn only the semantic content from any training sample, rather than its specific details; (2) develop irremovable protection mechanisms for images and videos; (3) explore strategies that combine deduplication, differential privacy, and unlearning at different stages of the pipeline; and (4) enhance prompt engineering techniques to ensure that the generated content not only avoids legal pitfalls but also more accurately reflects users’ intent.

15.4 Unified Benchmarks↩︎

In the context of computer vision, benchmarks serve as crucial tools for measuring and comparing the performance of various algorithms across standardized tasks and datasets. Although research on replication is flourishing, researchers often work in isolation, leading to inconsistencies in evaluation. Future research may focus on building unified benchmarks for comparing algorithms in unveiling, understanding, and mitigating replication.

Unveiling. The benchmarks for unveiling may evaluate the accuracy of current methods. For instance, many membership inference methods claim that they can nearly judge with 100% accuracy whether a visual diffusion model was trained on one specific image or video. However, as shown in Table 5, this may not be the case in the real world setting, where black-box AUC drops to near-random levels. Therefore, building a benchmark to compare which membership inference attack is most effective is essential.

Understanding. Currently, all the understanding of replication phenomenon seems to be reasonable and correct. However, each understanding on replication may only captures a part of the entire complexity. Building a unified benchmark for understanding is challenging, yet it provides a valuable measure of the applicability of different interpretations.

Mitigation. The benchmark for mitigation may include assessing how successful the data optimization strategies and unlearning methods are. Specifically, researchers may evaluate what proportion of the training data is replicated using different protection or unlearning strategies.

15.5 New Regulation↩︎

As AI’s capability to mimic human characteristics in creative outputs grows, there is an increasing need for transparency in AI’s role in content creation. This transparency is crucial for addressing copyright claims and enhancing public understanding. Current initiatives focus primarily on the necessity of disclosing AI involvement in registered works. Furthermore, the development and training of AI systems often involve using large volumes of data, some of which includes copyrighted material. This practice has sparked concerns over potential copyright infringement, an issue that remains legally ambiguous. As discussed in Section 14 and Table [tab:regulation], the regulatory landscape varies significantly across jurisdictions, with the EU mandating opt-out mechanisms while the US relies on case-by-case fair use analysis. Consequently, there is a pressing need for updated regulations or clarifications on existing copyright exceptions to accommodate the complexities introduced by AI. Jurisdictions worldwide are beginning to consider such amendments, but the global landscape is still uneven and undergoing transition. Emerging domains such as metaverse, autonomous driving, and education (Section 14.5) further complicate the regulatory picture, as domain-specific replication risks require tailored legal frameworks.

References↩︎