1 Introduction↩︎

Biometric identifiers such as fingerprints, facial geometry, and vein patterns constitute sensitive personal data. Their collection and processing fall under strict legal scrutiny, including the EU’s General Data Protection Regulation (GDPR) [1] and the Illinois Biometric Information Privacy Act (BIPA) [2]. Beyond legal compliance, ethical obligations further constrain the biometric data life cycle. Data acquisition typically requires explicit informed consent and clearance from an institutional ethics committee [3]. These constraints limit both development and reproducibility, because many biometric datasets cannot be redistributed once privacy or consent requirements change.

Synthetic data offers a more ethical and privacy-friendly alternative [4]–[6] to real biometrics, since no sensitive information is collected from real subjects and regulatory exposure is correspondingly reduced. Advances in generative models, from Generative Adversarial Networks (GANs) [7] to Diffusion Models (DMs) [8], have made synthetic training data increasingly competitive. Models trained entirely on artificial faces now achieve recognition rates comparable to those of models trained on real photographs [4]. Yet this is only half a solution. While the training side of the pipeline can now operate without real faces, evaluation still cannot. Even studies that propose new synthetic datasets evaluate them on real benchmarks. A proper assessment of whether synthetic data suffices for evaluation is, to the best of our knowledge, absent from the literature. Without this evidence, researchers remain forced to include real data in their pipelines, if only for benchmarking, and the privacy problem persists.

Research into synthetic benchmark datasets has been limited [9]–[11] and has not produced clear guidance on their practical applicability. In this work we provide a systematic answer to the question: can synthetic benchmarks reliably reflect model performance on real data? This question is not merely academic. With limited control over the generation process [12], synthetic data can exhibit poor identity separability, insufficient intra-class diversity, and demographic biases. A distributional mismatch between synthetic and real images therefore cannot be excluded without thorough empirical evaluation.

The main contributions of this work are as follows:

• We conduct the first comprehensive evaluation of 12 synthetic face recognition datasets as benchmarks across 24 face recognition models, comparing them with 7 commonly used real benchmark datasets.

• We provide a detailed breakdown of each dataset’s benchmarking performance across recognition accuracy and a set of standard biometric evaluation metrics.

• We show that MorphFace and Vec2Face achieve the strongest alignment with real benchmarks and can serve as reliable substitutes.

• We investigate the factors that influence benchmarking utility, including domain gap, identity separability, and intra-class diversity.

• To promote reproducible research, our code is publicly available².

2 Related Work↩︎

Face recognition benchmarking has historically relied on real-world data (2.1). The growing body of synthetic datasets (2.2) has recently opened the door to synthetic evaluation (2.3).

2.1 Real Benchmark Datasets↩︎

Labeled Faces in the Wild (LFW) [13], with 13,233 images of 5,749 identities, was long the standard unconstrained benchmark for face recognition. As model performance on LFW has saturated [14], however, with most recent systems approaching 100% accuracy, its ability to discriminate among models has diminished.

Several more challenging alternatives have since been introduced. Cross-Pose LFW (CPLFW) [15] and Cross-Age LFW (CALFW) [16] modify the original LFW pairing strategy. CPLFW increases pose variability, while CALFW narrows the age gap between mated and non-mated pairs so that age alone is less predictive. On LFW, the mean age difference is 4.94 years for mated pairs and 14.85 years for non-mated pairs; CALFW brings these to 16.61 and 16.14, respectively. AgeDB-30 [17] targets temporal variation more directly by selecting 16,488 images from 568 identities, each pair separated by approximately 30 years. Celebrities in Frontal-Profile (CFP-FP) [18] instead focuses on viewpoint, pairing frontal and profile images across 7,000 images of 500 identities.

Representativeness is an equally important concern. IJB-B [14] addresses this with 1,845 human-labeled identities captured under diverse acquisition conditions; its 21,798 still images and 55,026 video frames carry, among others, annotations for occlusion, skin tone, facial landmarks, and localization. IJB-C [19] extends IJB-B to 31,334 images of 3,531 identities, broadening occupational and racial diversity and introducing stronger occlusion variation.

2.2 Synthetic Training Datasets↩︎

Retractions of real face datasets [3] and potential legal consequences [20] have motivated the search for alternatives. Synthetic data emerged as a response to the mass-scraping practices behind many widely used datasets. The development of Generative Adversarial Networks [7] and Diffusion Models [8] made high-fidelity image generation feasible, and research on synthetic face datasets has accelerated accordingly. State-of-the-art generators [21], [22] can be conditioned [3] to preserve a specific identity across multiple images.

The central difficulty [4] in constructing a useful synthetic dataset is achieving high intra-class diversity, the ability to produce semi-hard [23] samples that challenge the model, without sacrificing identity separability, meaning that distinct identities must not overlap unnaturally in the embedding space. Early datasets such as Syn-1M [24] and SynMulti-PIE [10] struggled with this balance and yielded lower performance when used for training. More recent datasets have closed the gap considerably. MorphFace [25], CemiFace [23], Vec2Face [26], and VariFace [27] now support training pipelines whose recognition performance matches, and in some cases exceeds [27], that of models trained on real data.

2.3 Synthetic Benchmark Datasets↩︎

Only a handful of datasets have been proposed specifically for benchmarking. SynMulti-PIE [10] was the first to address this goal, emphasizing the balance [4] between identity separability (minimal overlap between the similarity distributions of genuine and impostor pairs) and intra-class diversity (sufficient variation to make evaluation challenging). The authors argue that generated samples preserve identity without revealing the real identities used to train the generator. However, their evaluation covers only 64 identities, which limits the robustness of the conclusions, particularly with respect to universality.

ControlFace10k [11] contains 10,008 images of 3,336 synthetic identities generated through the Synthetic Identity Generation (SIG) pipeline. Although it provides three images per identity (frontal, left profile, and right profile), the intra-class variation was reported to be insufficient [4]. Identity separability is also weak, making ControlFace10k a questionable choice for either training or benchmarking.

SDFD [9] consists of 1,000 high-quality diffusion-generated samples with one image per identity. It is useful for demographic attribute prediction but cannot support pairwise verification benchmarking since it lacks multiple images per identity.

No synthetic dataset, whether designed for training or evaluation, has been directly validated as a face recognition benchmark. Because synthetic images may differ from real images in distributional properties and generative biases, their suitability as evaluation tools cannot be assumed. Establishing this suitability through rigorous empirical comparison is the gap that our study fills.

3 Methodology↩︎

Our testbed comprises 7 real and 12 synthetic face recognition datasets evaluated across 24 pre-trained models.

3.1 Datasets↩︎

We selected 7 real datasets for evaluation: LFW, CPLFW, CALFW, AgeDB-30, CFP-FP, IJB-B, and IJB-C. Beyond being the most commonly used benchmarks in the literature, these datasets collectively cover a broad range of challenging scenarios including pose variation and age gaps.

For the synthetic side, we selected 12 datasets generated by methods ranging from pure GANs and pure diffusion-based approaches to hybrid pipelines combining DMs with 3D morphable models (see Table ¿tbl:tab:datasets?). Together, these 12 datasets constitute the majority of publicly available synthetic face data. Access to IDiff-Face and SFace2 was obtained through online request forms; access to MorphFace was obtained by contacting the authors directly.

3.2 Models↩︎

We selected 24 pre-trained models spanning 9 vision transformers (ViTs) [35]–[37] and 15 convolutional neural networks (CNNs) [38]–[40], trained on diverse datasets [41]–[44]. Because this paper focuses on synthetic data, we specifically included 7 CNNs trained entirely on synthetic data [26], [30], [32], [33], [45], all using the ResNet50 backbone, which is the standard architecture for benchmarking synthetic face recognition. Table 1 lists all models used.

3.3 Evaluation Metrics↩︎

Face recognition systems typically report recognition accuracy, the percentage of correctly classified pairs, as a primary metric. Accuracy alone is insufficient for real-world biometric systems, however, because the costs of false positives and false negatives differ substantially [46]. Following ISO/IEC 19795-1:2021 [47], security-critical systems prioritize keeping the False Match Rate (FMR), also called False Acceptance Rate (FAR), below a fixed threshold, and then minimize the False Non-Match Rate (FNMR), the proportion of genuine samples incorrectly rejected.

The acceptable FMR is application-dependent. Systems protecting sensitive resources demand near-zero false positives, while consumer devices tolerate a less strict decision threshold. This distinction is captured by FMR100 and FMR1000, which report the FNMR when the decision threshold is set so that only 1 in 100 or 1 in 1,000 impostor comparisons results in a false match. ZeroFMR is the FNMR at FMR\({}={}\)​0. The Equal Error Rate (EER) is the operating point where FMR equals FNMR, balancing security against usability. Finally, TAR@FAR reports the percentage of genuine samples accepted at a given FAR threshold, complementing the FMR-based metrics.

3.4 Dataset Correlation↩︎

A synthetic dataset can serve as a benchmark only if the scores it produces correlate with those obtained on real data. Given \(n\) paired observations (\(x_i\), \(y_i\)) representing the same models evaluated on two different datasets, we measure linear agreement with Pearson’s coefficient: \[r = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \cdot \sum_{i=1}^{n}(y_i - \bar{y})^2}}\]

where \(\bar{x}\) and \(\bar{y}\) are the sample means. We complement Pearson’s \(r\) with the Mean Absolute Error (MAE), which quantifies the average absolute difference between the scores produced by two benchmarks: \[\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - x_i \right|\]

Even if a synthetic benchmark is not expected to replicate absolute accuracy scores, it must at minimum preserve the relative ordering of models: a model that ranks higher on real data should also rank higher on synthetic data. We measure this property with Spearman’s rank correlation, computed from the rank differences \(d_i\) (difference between the positions, based on a sorted evaluation metric, assigned to the same model by two benchmarks) across \(n\) models: \[\rho = 1 - \frac{6 \sum_{i=1}^{n} d_i^2}{n(n^2 - 1)}\]

Both \(r\) and \(\rho\) range from \(-1\) (perfect negative alignment) to \(+1\) (perfect positive alignment). Following established guidelines [48], we interpret intermediate values as weak (0–0.5), moderate (0.5–0.75), semi-strong (0.75–0.85), or strong (0.85–1), while noting that these thresholds are not universally fixed.

We also examine the distributional properties that may explain benchmarking performance, following prior analyses [4], [30]. To quantify the domain gap (statistical difference) between synthetic and real images, we use the Fréchet Inception Distance (FID) [49], formulated as: \[\text{FID} = \|\mu_r - \mu_g\|_2^2 + \text{Tr}\!\left(\Sigma_r + \Sigma_g - 2\left(\Sigma_r \Sigma_g\right)^{1/2}\right)\]

where the subscripts \(r\) and \(g\) denote the real and generated distributions, respectively. The mean term measures distributions center shift, while the trace term evaluates mismatches in the shape and orientation of their covariance structures. To establish statistical significance, we report 95% confidence intervals obtained by bootstrapping (10,000 resamples with replacement), taking the 2.5th and 97.5th percentiles as interval lower and upper bounds.

3.5 Evaluation Protocols↩︎

We followed the evaluation protocol defined for each real dataset to preserve methodological consistency. LFW, CPLFW, CALFW, and AgeDB-30 all use the Unrestricted with Labeled Outside Data procedure, in which 3,000 mated pairs and 3,000 non-mated pairs are divided into 10 folds. In each fold, an optimal threshold is computed and then applied to compute accuracy; final results are the mean accuracy \(\pm\) standard deviation across folds. CFP-FP follows the same procedure but increases the sample size to 3,500 mated and 3,500 non-mated pairs.

IJB-B and IJB-C use a template-based protocol in which multiple images or frames are aggregated per identity to approximate real-world conditions. We use the 1:1 Baseline Verification protocol with 8,010,270 comparisons to compute TAR at FARs of 1% and 0.01%.

For the synthetic datasets, we adopted the Unrestricted with Labeled Outside Data protocol on a randomly selected set of 3,000 mated and 3,000 non-mated pairs, matching the majority of the real dataset protocols. The image-pair lists are included in our replication package.

In addition to the protocol-specified metrics, we computed EER, FMR100, FMR1000, and ZeroFMR for every model-dataset combination. This provides a more complete picture of how synthetic benchmarks behave across different security and usability requirements.

4 Experiments↩︎

For every model-dataset pair, we first compute cosine similarities and the resulting recognition accuracies. We then evaluate biometric performance metrics (EER, FMR100, FMR1000, and ZeroFMR) and measure Pearson’s \(r\) and Spearman’s \(\rho\) between datasets. IJB-B and IJB-C were excluded from the accuracy computation as the template-based protocol is incompatible with the pair-based synthetic setup. To shed light on the factors behind benchmarking quality, we also examine similarity score distributions and FID scores. We close with a focused analysis of the two strongest synthetic candidates.

4.1 Pearson’s Correlation↩︎

Recognition accuracies assessed with Pearson’s \(r\) (Figure 1) reveal that most dataset pairs exhibit high correlations, with values rarely falling below 0.9. The one clear outlier is ControlFace10k, whose low \(r\) indicates that it does not satisfy the criteria for a reliable benchmark. Analysis of 95% confidence intervals, however, exposes additional weaknesses that the point estimates conceal. Real-real pairs maintain consistently high lower bounds (0.77–0.97), whereas several real-synthetic pairs show substantially larger drops, with Langevin-DisCo and SynMulti-PIE reaching negative lower bounds. DCFace, Digi2Real, HyperFace, MorphFace, and Vec2Face, by contrast, maintain confidence bounds comparable to those of real-real pairs.

2pt

A finer-grained view by biometric metric (Table 2) shows that synthetic datasets achieve their strongest correlations under EER and their weakest under ZeroFMR. The ZeroFMR gap points to an unnatural distribution of outliers, specifically highly similar non-mated pairs as documented by Borsukiewicz et al. [4], even though overall separability remains fairly consistent. In line with the accuracy results, MorphFace and Vec2Face rank as the best benchmarks, both averaging \(r = 0.90\). The two are interchangeable for EER and FMR1000; MorphFace leads on FMR100 (0.93 vs.) while Vec2Face leads on ZeroFMR (0.81 vs.). The datasets designed explicitly for evaluation, ControlFace10k and SynMulti-PIE, are among the worst performers, with ControlFace10k being the only dataset that consistently shows negative correlations.

4.2 Spearman’s Rank Correlation↩︎

Pearson’s \(r\) assumes a linear relationship, which may not hold. Spearman’s \(\rho\) (Figure 2) relaxes this assumption and directly measures how well model rankings are preserved across datasets. The results draw a clear line between synthetic datasets that can rank models and those that cannot. MorphFace and Vec2Face show the strongest alignment with real datasets (\(\rho > 0.85\)), only slightly below the agreement among real-real pairs, confirming them as the most suitable synthetic alternatives. The synthetic datasets that perform well do so consistently across all seven real benchmarks, including those focused on extreme poses or large age gaps, while poor synthetic benchmarks underperform just as uniformly.

Spearman’s \(\rho\) for biometric metrics (Table 3) is broadly consistent with the Pearson results. MorphFace and Vec2Face again lead with a mean \(\rho\) of 0.87, while ControlFace10k (\(-0.52\)) and SynMulti-PIE (0.03) fall far behind. Most remaining datasets produce \(\rho\) values between 0.5 and 0.8, indicating a moderate-to-strong relationship that supports the feasibility of synthetic benchmarking for face recognition.

2pt

4.3 Fréchet Inception Distance↩︎

To quantify the domain gap, we computed FID (Table 4) using an IResNet18 model trained for face recognition, rather than the commonly used Inception-v3 [50]. Among real datasets, the highest alignment appears between LFW, CPLFW, and CALFW (FID at most 24.0), which share a substantial portion of their images. AgeDB-30 is the most distinct from the other real datasets, likely because of its high age variance, with FID scores in the range 52.1–73.2.

Comparing synthetic to real data reveals that a low FID is necessary but not sufficient for good benchmarking performance. Datasets with extremely high FID scores (ControlFace10k, SynFace, SynMulti-PIE, SFace2) consistently show poor correlation with real benchmarks, confirming that some baseline similarity to real data is required. Among the top-performing benchmarks, however, FID does not rank them correctly. MorphFace (FID 31.6–64.2) and Vec2Face (FID 51.9–92.0) both outperform CemiFace (FID 26.1–57.9) as benchmarks despite having higher FID scores, which indicates that distributional similarity alone does not determine benchmark reliability.

2pt

4.4 Similarity Distributions↩︎

To understand why some synthetic datasets succeed and others fail, we examined the cosine similarity distributions for mated and non-mated pairs (Figure 3). On real datasets, SwinFace and IResNet18 produce similar performance levels while FaceNet_CASIA clearly underperforms; Vec2Face and MorphFace reproduce this ordering, but SynMulti-PIE and ControlFace10k do not.

The shape of the distributions is also informative. Vec2Face and MorphFace resemble AgeDB-30 and CFP-FP, suggesting that they introduce more variability than LFW and therefore pose a greater challenge to models. This property makes them well suited for evaluating robustness to pose and age variation. SynMulti-PIE, by contrast, shows nearly identical overlap between mated and non-mated distributions regardless of model quality, indicating that some of its pairs are too diverse for any reasonable model to match, while others lack the diversity needed to challenge recognizers. ControlFace10k suffers from the opposite problem. High similarity scores among non-mated pairs signal low between-class variability and poor identity separability.

4.5 Analysis of MorphFace and Vec2Face↩︎

Given the results above, we focus on MorphFace and Vec2Face to assess their suitability as benchmarks more closely.

A per-dataset Spearman’s \(\rho\) comparison (Table 5) shows that MorphFace and Vec2Face are often more aligned with real benchmarks than the real benchmarks are with one another. This is most visible for ZeroFMR, where MorphFace (\(\rho = 0.79\)) exceeds all real-real pairs (0.58–0.78) and Vec2Face (0.77) is close behind. The generally lower \(\rho\) values for ZeroFMR across the board confirm that measuring performance at zero false positives is inherently more variable, even among real benchmarks. The minimum 95% CI bounds show that alignment between individual dataset pairs can drop lower, but this is expected because certain real benchmarks serve specialized roles such as evaluating robustness to extreme pose or age differences. Overall, the evaluated metrics show no systematic gap between real and synthetic data for benchmarking purposes.

Pairwise comparisons (Figure 4) confirm that alignment levels are broadly consistent across metrics. MorphFace and Vec2Face position themselves as the most reliable proxies of LFW for recognition accuracy and FMR100, of CFP-FP for FMR1000, and of CALFW for EER. The only divergence appears for ZeroFMR, where Vec2Face tracks AgeDB-30 most closely while MorphFace corresponds best to CFP-FP. Comparing the two datasets head to head, Vec2Face has a slight edge for accuracy and FMR1000, while MorphFace is marginally better for FMR100. Differences for ZeroFMR and EER are inconsistent.

1pt

MAE estimates against real benchmarks (Table 6) show that MorphFace and Vec2Face align best on accuracy and EER and diverge more on FMR-based metrics. This pattern suggests that the two synthetic datasets serve well as direct accuracy proxies for the first two metrics and as ranking tools for the rest. The lowest error values appear most frequently against CALFW and AgeDB-30, indicating that MorphFace and Vec2Face are particularly well suited for evaluating models on age-invariant identity features. The two datasets are nearly indistinguishable from each other in MAE, confirming that either can serve as a benchmark.

3pt

5 Discussion↩︎

Our experiments distinguish reliable from unreliable synthetic face datasets for the purpose of model benchmarking. Building on these results, we discuss the factors that determine how well synthetic data aligns with real samples and propose criteria for validating future synthetic datasets.

5.1 Factors Affecting Benchmarking Quality↩︎

The benchmarking quality of a synthetic dataset appears to be governed primarily by data diversity. The similarity distributions and FID scores support this interpretation. MorphFace and Vec2Face, the two strongest synthetic benchmarks, exhibit well-separated mated and non-mated distributions and relatively low FID scores, and they achieve the highest agreement with real benchmarks. Weaker candidates such as Langevin-DisCo, SFace2, and especially ControlFace10k show limited separation [4], [12], [27], larger distributional distance from real data, and correspondingly poor benchmarking performance. ControlFace10k represents the extreme case of domain gap, combining near-absent identity separability and a large distribution gap, as exemplified by FID score, with the weakest correlations observed in this study.

A related observation connects benchmarking quality to training utility. The synthetic datasets that perform best as benchmarks also tend to perform best for training [4], [25], [26]. MorphFace and Vec2Face rank among the strongest candidates in both settings, while Langevin-DisCo and SFace2 remain among the weakest. This overlap suggests that the dataset properties supporting effective training, namely high intra-class diversity and clear identity separability, also contribute to reliable benchmarking. The relationship is not strictly one-to-one, however. SynFace, for instance, is relatively weak for training [27] but comparatively stronger for benchmarking, indicating that the connection between training utility and benchmarking utility deserves further investigation.

5.2 Criteria for Validating Future Synthetic Face Recognition Datasets↩︎

To confirm future synthetic dataset applicability for benchmarking, in line with our methodology, one should empirically confirm that it aligns with real benchmarks. Importantly, synthetic datasets should:

• not exceed the natural disagreement level observed between real datasets

• maintain agreement at operating points such as EER/FMR100/FMR1000

Our results across 24 models show that, when measuring model accuracy, we should strive for both Pearson’s \(r\) and Spearman’s \(\rho\) to exceed 0.9, with lower bounds of the 95% CIs \(>\) 0.75. However, specific values depend on the number and choice of the models to be evaluated.

6 Conclusions↩︎

We evaluated whether synthetic face datasets can serve as reliable benchmarks for face recognition. The results reveal substantial variability across candidates. Datasets designed explicitly for benchmarking do not necessarily produce the strongest agreement with real benchmarks; several synthetic training datasets prove far more effective in that role. Among all candidates, MorphFace and Vec2Face stand out, consistently achieving Pearson’s \(r\) and Spearman’s \(\rho\) values whose 95% confidence intervals overlap with the disagreement observed among real benchmarks themselves.

Beyond identifying the strongest candidates, our results show a consistent separation between high-fidelity and low-fidelity synthetic benchmarks across all biometric metrics, a separation that is also reflected in the underlying similarity distributions. Well-constructed synthetic datasets can therefore support not only training but also reliable comparative evaluation for face recognition, reducing reliance on privacy-sensitive real facial data.

A promising direction for future work is the construction of synthetic evaluation pairs under explicit diversity constraints spanning both subject attributes and acquisition conditions. Such a strategy could yield stronger alignment with specialized benchmarks and enable targeted evaluation under specific covariates such as pose, age, or occlusion.

Acknowledgments↩︎

• This research was funded by the Luxembourg Army.

• AI-based tools were used for fixing grammatical and typographical errors.

References↩︎