Introduction↩︎

Advancements in computer vision have expanded its applications across diverse domains such as healthcare, manufacturing, surveillance, autonomous driving, and transportation, significantly increasing its societal impact. Given these circumstances, sophisticated deep learning models centered on image classification generally require large amounts of training data. However, in real-world scenarios, constraints such as cost, data privacy, ethical concerns, availability, and annotation burdens make it difficult to collect sufficient images to train models [1]–[3]. Therefore, in recent years, efforts to utilize generated data for training have become widespread as a means to compensate for the shortage of real image data. Generation technologies based on GANs [4], diffusion models [5], and variational autoencoders [6] have enabled the creation of high-quality images that are difficult for human perception to distinguish from real ones.

However, the use of synthetic data is not necessarily safe. For example, research by [7] reported that while models trained on synthetic images showed comparable accuracy to those trained on real images, significant differences emerged in metrics reflecting class imbalance and misclassification tendencies, such as F1 scores by validating the CIFAKE dataset [8], [9]. CIFAKE contains 60,000 synthetic images generated by Stable-Diffusion-v1-4 [10] and 60,000 real images, which were collected from the CIFAR-10 dataset [9] with ten classes. This finding suggests that, contrary to the goal of improving performance through synthetic data, classification performance may degrade when real-world images are input after model deployment, potentially compromising model reliability and safety. A survey for the overall picture of synthetic data research also concluded that synthetic data is convenient, but there are some risks if its quality is not checked [11]. Therefore, to utilize synthetic data in a manner suitable for real-world deployment, it is crucial to quantitatively understand how generative data differs from real data. This requires a multifaceted analysis of how these differences impact the training process and the correlation of image quality differences and performance.

Previous research on the differences between real and synthetic data using CIFAKE has primarily been developed with the goal of deepfake detection [12]–[19]. These studies typically formulated the problem as a binary classification task (real or fake) and employed approaches where ML models are trained to recognize patterns that distinguish between the two. However, since the primary focus was on improving detection accuracy, there was often insufficient in-depth analysis of what specifically caused these differences in datasets. While this is crucial for addressing societal issues arising from misuse, its objectives differ from those of evaluation and analysis frameworks designed to safely use of synthetic data as a trainable resource. The objective of this research is not to detect generated images, but to establish a framework for safely using generative images through multifaceted evaluation methods. To that end, we analyze differences between generated and real data based on information within the generated data, including cases where the generation model’s quality is unknown and is not systematically accounted for in model training.

Specifically, using three heterogeneous characteristics datasets, the CIFAR-10, CIFAKE dataset alongside images generated by fine-tuning with Elucidating the Design Space of Diffusion-Based Generative Models (EDM) [20] which is a state-of-the-art model, we analyze three main perspectives: (1) High-dimensional feature space, (2) low-level statistics measures, and (3) model structures in the learning process. This study proposes a prior validation framework for safely utilizing synthetic data in training, based on multifaceted analysis results. This framework identifies metrics that quantify the distribution gap between generative and real data and estimates the required proportion of real data to blend to achieve the target performance, based on the metrics. This clarifies the previously unquantified acceptable usage range for synthetic data, contributing to improved AI safety in real-world operations.

Related Work↩︎

This section explores research focusing on the differences between real and synthetic images. Prior research can be categorized into four main areas.

Model-centric analysis↩︎

These studies focus on how differences between real and synthetic data affect model behavior. For example, in a distillation learning setting, [21] analyzed which layers of student models trained on synthetic data were most affected, clarifying factors behind performance gaps compared to real data. Diffusion curriculum [22] was designed using image guidance strength, improving synthetic data learning efficiency by controlling the generation process. While these studies provide valuable insights into model behavior, systematically analyzing the distribution differences in the data itself is limited. Therefore, this manuscript focuses on a multifaceted analysis of the data itself and examines how it correlates with the model’s learning process.

Learning process strategy↩︎

The following group of studies verifies the effectiveness of learning methods using synthetic data. [23] systematically compared multiple mixing strategies from the perspective of how synthetic and real data should be mixed. Through empirical outcomes, they presented practical implications for mixed learning. However, they did not provide quantitative operational metrics, such as how much real data must be mixed to enter a safe zone. [24] conducted multifaceted robustness evaluations on unsupervised, self-supervised, multi-modal models trained solely on synthetic data, examining aspects such as shape bias, background bias, and noise resilience. [25] experimentally verified the effectiveness of synthetic data in zero-shot, few-shot settings, and pre-training. They also delved into the question of which parts of the model structure are influenced by learning with synthetic data. In [26], performance was evaluated across three settings: synthetic only, real + synthetic augmentation, and synthetic scaling law. However, both studies by [25] and [26] provided limited analysis of data quality at the data level and of its direct links to performance degradation factors.

Data-centric analysis↩︎

An analysis by [26] centered on distribution consistency in high-dimensional representation spaces based on the feature space of the CLIP image encoder. However, they did not directly analyze pixel-level statistical properties or differences in low-level visual elements. [27] highlights that generative models can output unnatural quirks or artificial noise imperceptible to humans, and inaccurately represent important visual details. They demonstrate that these issues can stem from models learning unnecessary features or failing to sufficiently learn essential features. However, the specific data distribution discrepancies causing these differences are not always systematically analyzed. [28] aimed to explain differences between datasets. They summarized differences in essential features across tasks using feature importance vectors and defined their own metrics. However, they did not delve into quantitative evaluations of effect sizes, such as how much individual feature misalignments impact classification performance. pyMDMA [29] is a tool that systematizes quality evaluation metrics for real and synthetic data and implements them as an audit framework. However, verification of how these quality metrics actually impact the learning process and performance remains limited at this stage.

Data Analysis using CIFAKE for Deepfake Detection↩︎

The following three studies all primarily aim to detect deepfakes, but they also analyze data characteristics. The studies by [30] and [31] analyzed distributions in RGB and YCrCb using Kullback-Leibler divergence, introducing a preprocessing step that utilizes these differences for feature extraction. It focuses on the unnatural color often seen in generated images, skewed color difference distributions in the relationship between luminance and color components. Chrominance statistics can effectively be used for authenticity discrimination tasks, such as real vs. fake. However, in semantic classification tasks, deviations in hue component statistics are not necessarily intrinsic features. Additionally, hosencifake? applied LIME to deepfake detection models, analyzing them through the explainability of prediction results. However, this primarily visualizes the model’s reasoning basis and differs in nature from approaches that systematically and quantitatively analyze the image data distribution itself.

Overall, while prior studies have explored differences between real and synthetic data from model-centric, learning process strategy, and data-centric perspectives, a comprehensive framework that systematically connects multi-level distribution discrepancies, ranging from high-dimensional feature spaces to low-level statistics, with downstream classification performance remains underdeveloped. This gap motivates this study.

Research Questions↩︎

Given the above introduction and gaps in related research, this study sets the following research questions to clarify the differences between generated data with distinct characteristics. CIFAKE1 denotes generated data that fails to train models capable of correctly classifying real images, whereas CIFAKE2 denotes generated data that successfully enables such models.

• RQ1: What differences arise in the high-dimensional feature space pre-trained on general images between CIFAKE1 and CIFAKE2?

• RQ2: What differences can be observed in the low-level statistical feature space between CIFAKE1 and CIFAKE2?

• RQ3: What differences emerge in the learning process between CIFAKE1 and CIFAKE2?

• RQ4: How can CIFAKE1 be effectively utilized in the learning process?

Contributions↩︎

By responding to the research questions above, this study comprehensively analyzes differences in characteristics between the two types of synthetic datasets: CIFAKE1 and CIFAKE2. As a result, it highlights the potential risks of generative data that have not been sufficiently discussed to date, while also providing guidelines for its safe utilization. Moreover, by clarifying appropriate usage methods based on the characteristics of generative data, it contributes to enhancing the safety of learning using synthetic data.

Methodology and Experimental Settings↩︎

Figure 1 presents the data quality analysis pipeline, where three heterogeneous datasets are analyzed using our proposed framework, and the impact of the synthesized data is examined under realistic conditions. The details are explained in each subsection.

Datasets↩︎

The datasets used in this study are the following three types in Table 1. 10,000 records from the REAL dataset are used as test data, while CIFAKE1 and CIFAKE2 are used solely for training. The number of images per class is set equally across all 10 classes.

Classification Models↩︎

Three types were adopted: the CNN-based ResNet50 [32], the MobileNetV2 [33] designed for mobile environments, and the transformer-based Vision Transformer [34]. All models were pre-trained on ImageNet [35], and this study focuses on fine-tuning these models. This is because, in many image classification applications, it is common practice to utilize pre-trained backbones rather than training models from scratch. Training was conducted for up to 50 epochs, with the initial learning rate set to 0.1 and gradually reduced to achieve convergence. For each experiment, the model yielding the highest F1 score is selected for analysis. Data augmentation is limited to shuffling, as image color-space analysis is conducted.

Differences in High-Dimensional Feature Space↩︎

To evaluate differences in image representations within high-dimensional feature spaces, this study employs three feature extraction models: Contrastive Language-Image Pre-training (CLIP) [36], DETRwith Improved deNoising anchOr boxes v2 (DINOv2) [37], and DINOv3 [38]. To visualize differences in the distribution of feature representations obtained by these models, plots are generated using Uniform Manifold Approximation and Projection (UMAP) [39] and t-Distributed Stochastic Neighbor Embedding (t-SNE) [40] as dimensionality-reduction techniques. Total six patterns are analyzed.

Differences in Low-Level Statistics↩︎

In analyses of statistical differences in low-level features, illuminance components, the V channel in the HSV color space, the Y channel in the YCrCb color space, and the L* channel in the CIELab color space are targeted. While color components are expected to differ between classes, illuminance components are class-independent and are ideally assumed to be constant. From the first to fourth order statistics and entropy are measured to compare the REAL dataset.

Differences in Model Learning Process↩︎

This section analyzes the following two points: (1) Compares differences in internal representations within the model during the learning process between REAL, CIFAKE1, and CIFAKE2 based on cosine similarity and L2 difference per layer. (2) Systematically evaluates the effects of mixing generated images based on realistic usage scenarios.

1. One class replacement: For instance, in cases where data for specific disease labels is extremely scarce, such as skin lesion images, methods that supplement only the minority class with generative images are commonly employed. Therefore, this experiment replaced only the class showing the most significant performance degradation with REAL images to analyze its impact.

2. Randomized mixture: This setting anticipates situations where sufficient real images are unable to be collected due to rare events, and generates images mixed across all classes. In this verification, the mixing ratio of REAL images is gradually varied evenly from 10% to 90% across each class to analyze the impact of the mixing ratio on model performance.

3. Additional finetuning with small real data: This configuration assumes a scenario where a small amount of data is available locally and models have already been trained on generated images. It investigates the differences between models trained by fine-tuning the best F1 score model trained on CIFAKE1 with an additional 10% of REAL images, for periods ranging from 10 epochs up to a maximum of 100 epochs.

Results↩︎

Generative Image vs Real Image↩︎

As shown in Figure 2, CIFAKE1 performs worse than CIFAKE2 and REAL across many classes and models across all evaluation metrics. This result suggests that the CIFAKE1 dataset possesses different characteristics from the other two. On the other hand, CIFAKE2 shows a trend largely similar to REAL, confirming that it has properties close to REAL.

Furthermore, examining the classification details of CIFAKE1 in the confusion matrix (Figure 3 (a)) reveals that all models tend to misclassify frog as bird and cat. Cats and deer are prone to misclassification as birds, horses, or dogs, while CNN-based models showed a tendency for dogs to be misclassified as birds, cats, or horses. On the other hand, trucks and ships generally exhibited relatively fewer misclassifications overall. As shown in Figure 3 (b), CIFAKE2 showed a similar trend but, due to its higher performance, did not exhibit the same level of disturbance as CIFAKE1.

Differences in High-Dimensional Feature Space↩︎

Visualization results in Figure 4 obtained by projecting the high-dimensional feature space by DINOv3 embeddings onto two dimensions using UMAP reveal that CIFAKE1 tends to cluster relatively far from the centroids of the real data in the UMAP space. This separation is particularly pronounced for frog, bird, and cat, where classification performance degradation was significant, showing large distances between the CIFAKE1 and REAL clusters of centroid distances. In contrast, the centroids of CIFAKE2 are closer to those of REAL, and its data points are generally distributed within the same areas. Furthermore, the size of the circles representing the spread of the distribution is similar, suggesting that the feature distribution in CIFAKE2 is closer to REAL.

In the Table 2, the top table shows results when dimensionality reduction was performed using UMAP, while the bottom row shows results when it was performed using TSNE. Each value represents the distance between the centroids of real and generative data. Notably, large distance differences were observed in the frog class, so the corresponding values are shown in bold.

In both DINOv2 and DINOv3, the CIFAKE2 classes consistently approach the real centroids in both UMAP and t-SNE, suggesting that CIFAKE2 is less in overall distribution misalignment. In contrast, CLIP showed no uniform shortening of distances for CIFAKE2, with some classes exhibiting increases and others decreases. That said, frog exhibited significantly larger distances than the other classes in both methods. It was confirmed that classes showing performance degradation also exhibited greater divergence in the feature space.

Furthermore, the correlation between distance and performance is shown in Figure 5. In CIFAKE1, a tendency toward performance degradation was observed as the distance from the real data centroid increased. Meanwhile, even in CIFAKE2, where performance degradation was not pronounced, a tendency toward performance decline with increasing distance was confirmed. However, the impact was relatively minor in CIFAKE2. These results suggest that deviation from the real distribution in the feature space may be associated with performance degradation.

Differences in Low-Level Statistics↩︎

As shown in Figure 6, analysis of the relationship between distribution differences in real data and performance degradation, focusing on the illuminance channel, revealed that for CIFAKE1, larger distribution differences tended to lead to greater performance degradation across statistical metrics and entropy. However, krutosis was not sufficiently captured. While the linear relationships were relatively weak for the first- and second-order statistics, a clearer linear correlation was observed for skewness and entropy. Furthermore, since CIFAKE2 itself has a small distribution difference from the real data, its impact on performance was also limited.

Furthermore, Table 3 quantitatively evaluated the correlation using Pearson’s correlation coefficient, based on the linear trend observed in Figure 6. Entropy (indicated in bold) showed a moderate to strong positive correlation with the F1 score and Recall across all models and color spaces when compared to the Illuminance channel. Notably, MobileNet exhibited a strong positive correlation exceeding 0.75 with the F1 score, reaching the highest value of 0.87 with the L channel. While ViT16 showed slightly weaker correlations compared to CNN-based models, it still exhibited positive correlations with the L channel (0.65 for F1 score and 0.75 for Recall) and a negative correlation with Precision (-0.45). These results suggest that changes in the entropy distribution within the L channel significantly impact performance.

Differences in Model Learning Process↩︎

Differences in Model Structure↩︎

The box plots of cosine similarity for each layer, shown in Figure 7, demonstrated that CIFAKE1 already diverges from the REAL model starting from the feature layer immediately after the input layer. Significant differences are particularly evident in the early layers sensitive to low-level features such as color and shape, and this divergence persists through the 17th layer. Although similarity slightly increases in the final layer, it overall showed a declining trend, with wide variation. In contrast, CIFAKE2 shows no substantial change in cosine similarity relative to the REAL model until the early layers (approximately 3th layer). Even in the intermediate layers, the range of divergence in feature representations remains about half that of CIFAKE1. While divergence tends to increase as the model progresses to later layers, the rate of decline is less pronounced than in CIFAKE1. These trends were consistently observed across all models.

As shown in Figure 8, the L2 distance also increases as the layer depth increases, becoming farther compared to the REAL model. However, the gap between the CIFAKE2 model and the REAL model is not as large as that between the REAL model and CIFAKE1.

Realistic Scenarios↩︎

One class replacement: Table 4 shows the change in performance metrics when only the frog class is replaced with the REAL dataset. For all models, Recall for the frog class improved significantly, while Precision decreased substantially. As a result, although the F1 score showed slight improvement for some models, no significant overall improvement was observed on average.

Randomize Mixture: As shown in Figure 9, the F1 score for frogs, which was significantly low when using only CIFAKE1, improved substantially with just a 10% mixture of REAL data. Furthermore, across all models, when the mixture ratio is around 30–40%, no significant difference is observed compared to using REAL data alone.

It was confirmed that the statistical illuminance entropy value also approximates the REAL value as the number of REAL images increases. Similar to the performance improvement, even adding a small amount of REAL data significantly improved the statistical information.

Additional Finetuning with Small Real Data: Table 5 shows the F1 score performance when training for an additional 150 epochs using 10% REAL data. The score was evaluated every 10 epochs, and, as shown in Table 5, after a sharp increase at 10 epochs, it was only a slight increase. Even after training for up to 150 epochs, the results hardly improved and never exceeded those obtained from training on real data. This same result was observed for both CNN and transformer-based models.

Discussion↩︎

In the discussion, we address each research question based on the experimental results from the previous section. RQ1: The difference between CIFAKE1 and CIFAKE2 in the high-dimensional feature space was confirmed to lie in the distribution of feature clusters and could be quantitatively captured as the distance between the clusters. It became apparent that a greater distance from the centroid within the REAL cluster negatively impacts classification performance. Comparing feature vector extraction using CLIP versus the DINO family, the DINO-based approach detected image-appropriate feature differences more clearly. While CLIP emphasizes semantic information through learning from text-image correspondence, DINO, which performs self-supervised learning using only images, captures visual features such as shape and texture more accurately.

RQ2: The distribution differences between CIFAKE1 and 2 in low-level statistical information (in illuminance channels) were observed significantly in standard deviation, skewness, and entropy. The correlations between F1 scores and illuminance channel entropies in all color spaces were strong. These results indicate that classes with small differences in illuminance distribution compared to REAL images can be classified accurately, but as the differences increase, false negatives become more likely.

RQ3: Analysis of layers in the model training process also revealed that CIFAKE1 exhibited significant differences from REAL already during the initial layers in both cosine similarity and L2-distance, in other words, the stage of learning low-level statistical features. CIFAKE2 also gradually shows differences from REAL, and, while consistent with the argument that significant accuracy degradation occurs in the final layer [21], depending on data quality, performance degradation can appear as early as the initial layers due to differences in color-space-level distributions, as seen in CIFAKE1. This indicates that the primary cause of performance degradation cannot be definitively attributed to the "final layer."

RQ4: Under realistic synthetic data scenarios, when images with differing brightness distributions, already low-level in CFAKE1, were used during initial fine-tuning, approximately 30% to 40% of REAL images could be secured. This achieved a distribution comparable to REAL training, with performance remaining nearly equivalent. Conversely, with only 10% of the data, the distribution differences lingered, and even after approximately three times the fine-tuning, performance remained far below that of REAL. Furthermore, replacing only the poorly performing classes with REAL data disrupted the overall learning balance and did not improve performance.

These findings demonstrate that when applying synthetic data for fine-tuning, it is critically important to comprehensively measure the anticipated differences in properties compared to REAL data beforehand. Depending on these differences, training solely on synthetic data carries the risk of inducing data shifts. To mitigate these risks, a strategy of blending data to make the overall training dataset’s characteristics approach those of REAL data could be adopted.

Limitations↩︎

This study conducted experiments using publicly available general-purpose natural image datasets. Therefore, the findings from these results may not be directly applicable to specialized domains that heavily rely on pigment distribution or shape information, such as medical images. Particularly in application areas where diagnostic accuracy and safety are strongly demanded, further domain-specific verification is necessary.

Future Work↩︎

Based on the findings from this research, we are planning to propose a method for stable learning in distributed learning environments prone to instability due to heterogeneous data distributions by monitoring the quality of learning data.

Conclusion↩︎

To meaningfully utilize synthetic images in realistic scenarios, it is crucial to pre-assessment of the quality of the data. This study demonstrated that the degree of performance degradation can be predicted by quantifying differences in high-dimensional feature space as feature vector distances using DINOv3, and further employing entropy differences in low-level statistical information regarding illuminance. Furthermore, results from data blending experiments simulating real-world deployment confirmed that incorporating approximately 30% REAL data per class during training mitigates data distribution bias and maintains classification performance comparable to training solely on real images. These findings provide practical guidelines for pre-evaluating synthetic data of unknown quality and safely and effectively incorporating it into image classification, thereby contributing to enhanced reliability and safety when utilizing synthetic data.

Acknowledgments↩︎

The authors would like to thank the EPSRC National Edge AI Hub at the University of Hull for its support throughout this research. The authors also acknowledge COST Action European Materials Informatics Network (EuMINe CA22143) for supporting a Short-Term Scientific Mission, which contributed to the completion of this work.

References↩︎