1 Introduction↩︎

The past several years have seen an extraordinary acceleration in the quality and accessibility of synthetic media. What once required specialized expertise and hand-tuned generative adversarial networks can now be accomplished by an ordinary user with a consumer-facing web application, a short text prompt, or a single uploaded photograph. Diffusion-based generators produce photorealistic faces and scenes frequently indistinguishable from genuine photographs under casual inspection, while face-swapping and avatar tools allow a person’s likeness to be transplanted into arbitrary video or image content in minutes. This democratization has a well-documented dark side: deepfakes and AI-generated imagery have been implicated in large-scale financial fraud carried out through impersonated video calls [1], non-consensual intimate imagery [2], manufacture of child sexual abuse material [3], and coordinated disinformation [4]. As convincing fakes have become cheap and fast to produce, the burden of distinguishing authentic from synthetic content has shifted from specialized forensic laboratories to ordinary platforms, institutions, and individuals with neither the expertise nor the tooling to perform that analysis themselves.

Three fundamentally different kinds of tool now compete to fill this gap, and they are almost never compared to one another. Commercial detection APIs let an organization submit an image and receive a verdict without training or hosting a model, and are marketed with prominent accuracy figures that are rarely independently verified. General-purpose vision-language models, never trained specifically for forgery detection [5], [6], are increasingly prompted zero-shot as an ad hoc detector by practitioners who already have API access to them for other purposes. And a large, fragmented ecosystem of open-source detectors, spanning academic checkpoints [7], [8], Hugging Face community classifiers, and purpose-built research models, offers a free alternative to both, at the cost of requiring local GPU infrastructure and offering no vendor accountability. Each of these three paradigms is evaluated, when it is evaluated rigorously at all, in isolation: commercial vendors publish internal benchmarks, vision-LLM providers report general-purpose capability rather than forgery-specific accuracy, and academic detectors are benchmarked against each other on shared research datasets that rarely resemble the messy, adversarial content circulating today [9], [10]. A practitioner deciding how to spend a detection budget, pay for a commercial API, prompt a model they already have access to, or self-host a free detector has no single source of evidence that measures all three options the same way. Figure 1 summarizes the paper’s motivation and workflow: the accuracy trap that motivates our metric choice (a naive always-fake predictor scores 79% accuracy yet 0.00 MCC), the benchmark pipeline itself (36 models, three paradigms, dual-metric evaluation), and a preview of the central finding, an MCC-versus-AUC landscape showing recurring trap, starved, and flat failure patterns.

This paper closes that gap. We construct VendorBench-100, a single, fixed, adversarial corpus of 100 images (79 fake, 21 real) assembled specifically to be hard rather than merely large: every image was hand-picked to probe a distinct failure mode, near-duplicate face swaps, letterboxed video-generation stills, on-device AI photo edits of otherwise-real scenes, and images of deliberately opaque provenance, so that a model’s score reflects performance on the difficult, modern middle ground rather than easy separability on saturated public datasets [11], [12]. We then evaluate 36 models spanning all three paradigms under one shared task definition, one common output schema, and one abstention policy, and rank them by the Matthews correlation coefficient, with ROC-AUC as a threshold-free tiebreak, because the corpus’s 79:21 class skew makes raw accuracy trivially gameable by a detector that simply predicts “fake” by default.

The result is not a simple ranking in which one paradigm or one model wins outright. Commercial APIs post the strongest typical performance, but the best open-source ranker outperforms every vision LLM in the study on pure ranking power, and the single highest-ranking-power model overall turns out to have one of the worst operating-point scores once its default decision threshold is taken into account. This divergence between a model’s ability to separate the classes (ROC-AUC) and its ability to make a well-calibrated decision at its shipped threshold (MCC) recurs across all three paradigms and is, we argue, a more important and more general finding than any single model’s leaderboard position. We report these results with an explicit accounting of the corpus’s small size and single-run design, and we release the full evaluation harness, per-image evidence, and aggregated results to support independent replication and extension. Our main contributions are

• We present VendorBench-100, a cross-paradigm benchmark evaluating 36 models such as 5 commercial APIs, 7 zero-shot vision LLMs, and 24 open-source detectors under one common normalized output schema and one abstention policy, so that results are directly comparable across three paradigms almost never measured on common ground.

• We construct and document a deliberately hard, edge-case-heavy 100-image corpus organized around eight distinct failure-mode families (live face-swap smearing, near-duplicate online face swaps, letterboxed text-to-video stills, avatar compositing seams, fully synthetic text-to-image output, on-device AI photo edits of real scenes, opaque-provenance manipulations, and compressed research-dataset frames), together with a per-image provenance registry and a filename-based anti-leakage protocol.

• We show a clear ordering by paradigm as commercial APIs lead, vision LLMs occupy the middle, and most open-source detectors trail, that is nonetheless paradigm-level rather than universal, since individual open-source detectors remain competitive with, or ahead of, the strongest vision LLMs on ranking power.

• We identify a recurring divergence between ranking power (ROC-AUC) and operating-point quality (MCC) that cuts across all three paradigms, showing that a model’s default decision threshold, not just its underlying discriminative ability, determines whether strong class separation translates into a trustworthy verdict.

2 Related Work↩︎

This section situates VendorBench-100 within the prior work its three tracks each draw on. We begin with the threat landscape that motivates detection in the first place, then survey the generative techniques and the academic detection methods built to counter them, including the datasets those methods are trained and benchmarked on. We then turn to the two paradigms outside the traditional academic-detector literature that this paper is the first to place alongside it: vision-language models used as zero-shot detectors, and the commercial detection services and provenance-based alternatives already deployed in practice. We close with the still-thin literature on evaluating detectors under real-world rather than laboratory conditions, and the evaluation methodology our own metric choices draw on.

2.1 Threat Landscape and Motivation↩︎

Deepfakes emerged as a societal concern with early landscape reports documenting rapid growth in synthetic media and its concentration in abusive applications [13], [14]. High-value financial fraud has since been carried out through real-time video impersonation, in one widely reported case costing a multinational firm tens of millions of dollars [1]; regulators have issued formal warnings to financial institutions about such schemes [15]. Beyond financial harm, cross-national survey work has documented the prevalence and psychological impact of non-consensual synthetic intimate imagery [2], and child-safety organizations have reported a troubling rise in AI-generated child sexual abuse material [3]. Broader assessments extend this threat picture further still, documenting risks to election integrity through synthetic disinformation [4], to financial-sector cybersecurity more generally [16], to minors specifically [17], and to the digital identity-verification systems that many online services rely on [18]. Taken together, this body of work establishes both why reliable detection matters and why the tools organizations actually deploy to perform that detection deserve careful, independent measurement rather than uncritical trust in vendor marketing.

2.2 Generative Models and Detection of Synthetic Imagery↩︎

The images a detector must classify are the product of a diverse, rapidly evolving generative ecosystem. Generative adversarial networks and style-based architectures established photorealistic face synthesis [19]–[21], and denoising diffusion probabilistic models together with latent diffusion have since come to dominate high-quality text-to-image generation [22], [23], producing a visually distinct artifact family from their GAN-era predecessors. Orthogonal to whole-image synthesis, convolutional face-swapping and neural-rendering methods enable targeted identity replacement within an otherwise real image or video, rather than generating a scene from scratch [24]–[27]. This three-way split including GAN-era synthesis, diffusion-era synthesis, and localized identity replacement is a central reason a detector that performs well on one manipulation family often degrades sharply on another: each leaves behind a different statistical fingerprint, and a model implicitly or explicitly tuned to one fingerprint has no guarantee of generalizing to the others.

A parallel literature targets fully synthetic, rather than face-manipulated, imagery, and much of it underlies the open-source detectors we benchmark in Section 4. Early work showed CNN-generated images carry detectable fingerprints within their generator family [28], while frequency-domain analyses showed the upsampling common to many generators leaves spectral artifacts [29], [30]. The central open problem is generalization across unseen generators: pre-trained vision-language feature spaces improve cross-generator robustness over narrowly trained detectors [31], [32], intermediate-encoder-block representations from large pre-trained backbones offer a further such feature space [8], reconstruction-based cues targeting diffusion models’ encoding process offer a complementary signal [33], [34], contrastive reconstruction-based training has been proposed specifically to improve cross-generator generalization for diffusion-generated images [7], and category-common-prompt injection into CLIP has been proposed as a further generalization strategy [35]. Benchmarks such as GenImage quantify exactly this generalization gap at scale, showing that detectors trained on one set of generators can lose substantial accuracy when evaluated on held-out ones [12]. This pattern shows strong in-distribution performance that degrades under distribution shift is precisely what our open-source track exposes at the level of individually deployable, publicly downloadable checkpoints: most of the 24 detectors we evaluate were not trained on the specific generator mix our adversarial corpus contains, and their scores reflect exactly this generalization failure rather than a defect in the corpus.

2.3 Deepfake Datasets and Detection Benchmarks↩︎

Empirical progress in face-manipulation detection has been driven largely by the availability of large, labeled corpora, whose evolution tracks how the threat has grown in scale and realism. FaceForensics++ standardized manipulated-face detection around a fixed set of manipulation methods and compression levels [11], [36]. The DeepFake Detection Challenge dataset substantially expanded both scale and actor diversity [37], [38], WildDeepfake introduced content collected directly from the internet rather than generated under laboratory conditions [39], and Celeb-DF specifically targeted the visual-quality gap between early, artifact-heavy deepfakes and the higher-fidelity forgeries that had begun to circulate in the wild [40]. More recent efforts have pushed further on both diversity and unification: ForgeryNet consolidates many distinct forgery types under a single benchmark [41], and DF40 explicitly targets the next generation of forgery techniques as generative models have continued to advance [42]. Consolidated frameworks such as DeepfakeBench standardize the training and evaluation protocol itself, showing how strongly reported accuracy depends on preprocessing and cross-dataset transfer choices [43]. Our own corpus draws on FaceForensics++ and DF40 style frames as two of more than twenty provenance sources, but where this prior work benchmarks trainable detector architectures on large, relatively homogeneous corpora, our object of study is a small, deliberately heterogeneous, adversarial set evaluated against closed commercial APIs, general-purpose vision LLMs, and independently distributed open-source checkpoints simultaneously.

2.4 Vision-Language Models as Zero-Shot Detectors↩︎

A newer and less-studied question is whether general-purpose vision-language models, never trained specifically for forgery detection, can be prompted zero-shot as usable detectors. Contemporary multimodal LLMs from several providers including OpenAI’s GPT family with vision input [5], Google’s Gemini [6], Anthropic’s Claude Opus [44], Alibaba’s Qwen vision-language models [45], Meta’s Llama 4 with multimodal variants including the Maverick configuration [46], NVIDIA’s Nemotron vision-language family [47], and Zhipu AI’s GLM family [48] expose strong general visual reasoning but no forgery-specific training signal, so their reliability as detectors is an open empirical question rather than an assumption we can import from their general-purpose benchmarks. We treat this question directly by including one representative model from each of these seven families as a dedicated evaluation track (Section 3), prompted with a shared forensic verdict schema rather than a bespoke prompt per model, and scored under exactly the same metrics as the commercial and open-source tracks.

2.5 Provenance, Watermarking, and Commercial Detection Services↩︎

Passive, after-the-fact detection is not the only proposed defense against synthetic media. Proactive provenance schemes instead try to establish authenticity at the moment of capture or generation. Content Credentials under the Coalition for Content Provenance and Authenticity’s (C2PA) specification bind cryptographically signed provenance metadata directly to a media file [49], [50], and internet-scale watermarking schemes such as SynthID embed an imperceptible signal directly into generator outputs at the moment of creation [51], [52]. These mechanisms are valuable but fundamentally proactive: they depend on voluntary adoption and provide no protection against the enormous volume of unmarked content already in circulation, or against generators that decline to participate. Passive detectors, of the kind evaluated across all three tracks in this paper, remain the only recourse for that unmarked content.

The commercial detection services in our first track sit within a broader, still-emerging industry landscape characterized by limited public documentation. Reality Defender [53], Hive AI [54], Sightengine [55], TruthScan [56], and Neural Defend [57] each expose a different native output schema: some return categorical tags, others per-class confidence scores, and others free-text verdicts. This heterogeneity, now multiplied across three entirely different paradigms rather than one, is precisely what necessitates the unified normalization schema described in Section 3. Other commercial and research detectors not evaluated here, including Sensity AI [58], Illuminarty [59], AI or Not [60], and Intel’s real-time FakeCatcher [61], illustrate that the commercial landscape this normalization problem applies to is broader still.

2.6 Evaluations of Detectors in the Wild and Evaluation Methodology↩︎

A growing body of work asks whether laboratory-reported accuracy survives contact with real-world data, and the answer is consistently discouraging. Empirical studies report substantial generalization gaps for image detectors when evaluated across unseen generators and under realistic degradations such as recompression and resizing [62], [63], and a broad survey specifically catalogs the reliability pitfalls that afflict deepfake detection as a field [10]. Some pitfalls are structural: dataset construction can leak shortcuts, with post-processing and JPEG artifacts, rather than synthesis cues, driving an apparently high accuracy [64], and “sanity check” analyses show ostensibly strong detectors can rely on spurious, dataset-specific signals [65]. Most directly related to our own work, several recent efforts have evaluated detection systems, including named commercial offerings, on content actually circulating in the wild rather than on curated laboratory datasets. Deepfake-Eval-2024 evaluates detectors, including commercial products, on recently circulated real-world content and reports accuracy figures markedly lower than vendor marketing claims [9]; related commercial and production-style evaluations include ARIA [66], “Organic or Diffused” [67], the Visual Counter Turing Test [68], AI-GenBench [69], and direct human-versus-AI comparisons of detection accuracy [70]. Robustness studies further show well-performing detectors can be defeated by adversarial perturbations [71], underscoring that accuracy on unperturbed content is only a partial picture of real-world reliability. Where these efforts each cover one or two paradigms, our contribution is a controlled, named, cross-paradigm comparison spanning all three on a single fixed corpus under identical normalization.

Finally, our metric choices draw on classical evaluation methodology well established in machine learning but under-applied to commercial and zero-shot deepfake-detection systems specifically. Systems that may decline to produce a verdict are properly analyzed through the reject-option and selective-classification framework [72]–[77], motivating reporting coverage alongside accuracy computed only over decided images. For imbalanced binary classification problems of the kind our real-versus-fake corpus presents, single scalar summaries such as raw accuracy can be actively misleading, and the Matthews correlation coefficient together with careful, separate reporting of class-conditional error rates is recommended in place of accuracy alone [78]–[80]. We adopt MCC as our primary ranking metric for this reason, and add threshold-free ROC-AUC as a tiebreak precisely because, as we show in Section 5, a model’s ranking power and its operating-point quality can diverge sharply.

3 Methodology↩︎

Evaluating three fundamentally different detection paradigms fairly requires getting three things right, in order: the corpus every model is measured against, the protocol that makes their heterogeneous outputs comparable at all, and the metrics used to interpret the result. We address each in turn, beginning with the corpus itself, since every downstream comparison inherits whatever strengths and limitations that corpus has. Figure 2 walks through the evaluation architecture: every model is normalized into a shared prediction record, scored by a common metrics engine reporting four core metrics plus the MCC-versus-AUC comparison, and ranked into a single final leaderboard, surfacing the leaderboard paradox where TruthScan posts the highest ROC-AUC (0.915) yet ranks 32nd of 36 by MCC (2̆2120.13).

3.1 Dataset↩︎

We evaluate all 36 models on a single fixed corpus of 100 still images: 79 labeled fake and 21 labeled real (a 79:21 split). Unlike a corpus optimized for scale, this one is optimized for difficulty: every image was hand-selected to probe a distinct, currently under-served failure mode, so that a model’s score reflects skill on hard, modern generator output rather than easy separability on saturated public benchmarks. The corpus deliberately spans more than twenty provenance sources so that no single generator, resolution, or lighting regime dominates and no detector can succeed merely by tuning to one artifact profile. Figure 3 summarizes the distribution of the 79 fake images across the eight edge-case families included in VendorBench-100. The corpus intentionally emphasizes difficult, contemporary manipulation scenarios rather than balancing category frequencies, ensuring that detector performance reflects robustness under realistic failure modes instead of performance on a homogeneous benchmark.

3.1.1 Edge-Case Taxonomy↩︎

The fake portion of the corpus is organized around eight distinct edge-case families, each defeating detectors for a structurally different reason. Live face-swap smearing (DeepFaceLive-style capture) entangles manipulation artifacts with ordinary webcam motion blur, so a detector cannot cleanly separate generation artifact from capture noise. Near-duplicate online face swaps (Picsi.Ai, Tuguoba, VidMage, Live3D) replace only the face region on an otherwise genuine selfie, leaving global image statistics looking authentic and punishing models that flip predictions on trivially similar inputs. Letterboxed text-to-video stills (Sora, Veo, Veo 3) are extracted frames carrying black letterbox bars and cinematic color grading that can read as ordinary photographic post-processing rather than a generation artifact. AI avatar compositing seams (HeyGen) exhibit a faint glowing outline around the subject that is a compositing edge, not a pixel-frequency signature, so frequency-domain detectors miss it. Fully synthetic text-to-image output (DALL-E, Gemini, and several additional generators) is the classical generated-image case, though several samples are high-resolution and photorealistic enough that only faint over-smoothness remains as a cue. On-device AI photo edits of otherwise-real scenes (Samsung Galaxy AI) are the single hardest family: because only a small, localized region of an overwhelmingly authentic photograph is synthetic, any whole-image fakeness score is diluted toward “real.” Opaque-provenance manipulations carry no independently identifiable generator at all, modeling the realistic “unknown tool” scenario a deployed detector actually faces. Finally, compressed research-dataset frames (DF40, FaceForensics++) contribute heavily compressed, low-resolution talking-head content with characteristic waxy skin blending. Compounding all eight families, the corpus skews toward small, heavily recompressed images: 78 of the 100 images are at or below 256 pixels on the long side, which erodes exactly the high-frequency evidence that many frequency-based detectors depend on.

3.1.2 Composition, Provenance, and Anti-Leakage Protocol↩︎

The 79 fake images are drawn from 21 distinct source groups, verified against public evidence wherever the group’s originating platform or dataset is independently identifiable: 15 groups are fully verified (e.g., OpenAI Sora, Google Veo, HeyGen, DF40, FaceForensics++), 4 are partially verified (the platform exists and is named, but the exact model or generation run is not pinned down), and 2 remain deliberately opaque, retained explicitly as unknown-tool cases rather than excluded. The 21 real images are authentic photographs including bare-faced selfies, portraits, and casual indoor and outdoor shots used as negative controls to measure false-positive behavior. A complete per-image provenance registry, including source, scenario, and verification status for every entry, is released alongside the corpus. Figure 4 provides an overview of the corpus provenance. Most source groups are fully verified and account for the majority of images, while a smaller subset consists of partially verified or intentionally opaque sources that emulate realistic unknown-provenance content encountered in deployment.

Because two of the three tracks in this study are systems capable of reading filenames or metadata (vision LLMs in particular), the corpus enforces a strict anti-leakage protocol: every model is served images under neutral, numeric filenames (001.jpg, 002.jpg, …) with no label-revealing information in the filename, path, or accompanying metadata. The descriptive, label-bearing identifiers used in the provenance registry (e.g.fake_SORA_003) are post-hoc bookkeeping keys only and are never presented to a model at evaluation time. Ground truth is held in a separate manifest file, so a correct prediction can only come from the image content itself.

We treat this corpus explicitly as a diagnostic stress test rather than a population-accuracy benchmark: with only 100 images spread across 21 fake source groups, per-group sample sizes are small, and the corpus is far too small to support training, fine-tuning, or a claim about calibrated field accuracy. Its purpose is to surface where and how each of the 36 models breaks under difficult, heterogeneous, adversarial conditions, and we return to this scope limitation explicitly in Section 6.

3.2 Three-Track Evaluation Architecture↩︎

Evaluating 36 models spanning three fundamentally different deployment paradigms requires three fundamentally different integration strategies, unified by a single canonical output schema. Every model, regardless of track, is normalized into an identical result record consisting of a hard label (FAKE or REAL), a confidence score interpreted uniformly as \(P(\text{fake}) \in [0,1]\), and a success flag distinguishing a genuine verdict from an abstention or failed call. This shared schema, not any single track’s native format, is what makes commercial, LLM, and open-source outputs directly comparable.

Commercial APIs (5 models). Each of the five commercial vendors such as Neural Defend, Reality Defender, Sightengine, TruthScan, and Hive AI are queried live over HTTP for every image in the corpus, and round-trip latency is measured directly. This is the only track for which latency is reported, since it is the only track measuring a genuine, comparable, hosted-service round trip.

Vision LLMs (7 models). Seven general-purpose vision-language models such as Gemini [6], Claude Opus 4.8 [44], Qwen [45], Llama 4 Maverick [46], a Nemotron vision-language variant [47], GPT [5], and GLM-5.2 [48] are prompted zero-shot with a shared forensic verdict schema requesting a structured judgment (status, whether the image appears AI-manipulated, and a probability estimate) rather than a bespoke prompt engineered per model. Results were collected out-of-band, some via vendor API and some via browser automation, and imported into the same canonical schema by per-model adapters; this track therefore evaluates each model’s zero-shot forensic reasoning ability rather than a tuned, purpose-built detection deployment.

Open-source detectors (24 of 27 surveyed). Twenty-four independently distributed open-source detectors are run locally on GPU against the full corpus, spanning standard Hugging Face transformers classifiers, custom Hugging Face loaders with bundled inference code, and academic checkpoints paired with their original repository’s inference script. Seven of the twenty-four correspond to published academic methods: DRCT [7], RINE [8], C2P-CLIP [35], DeCLIP [81], AIDE [82], GenD-DINOv3-L [83], and the Community Forensics ViT-384 checkpoint [84], [85]. The remaining seventeen are individually distributed Hugging Face Hub community checkpoints without an associated paper: ntire2026_deepfake [86], dima806_ai_vs_real [87], bombek1_siglip_dinov2 [88], organika_sdxl [89], aidfr_real_v2 [90], nahrawy_aiornot [91], ummmaybe_vit [92], yaya_source [93], ash_flux_vit [94], king1oo1_deepguard [95], haywoodsloan_deploy [96], date3k2_vit [97], jacob_distilled [98], opensight_commfor [99], ateeqq_siglip2 [100], sadra_sdxl_face [101], and wvolf_vit [102], the last of which originates from an MSc thesis at Solent University rather than a purely anonymous upload. For every community checkpoint we cite the exact Hugging Face repository path and uploader username as recorded in the project’s own model registry; we do not independently re-verify the uploader’s identity, training data, or claimed performance beyond what that registry and the checkpoint’s own model card state. Each detector’s native label convention is mapped onto the shared \(P(\text{fake})\) scale according to its own documented semantics; label mappings are never inverted post hoc to inflate a score, so a detector that genuinely disagrees with this corpus’s notion of “fake” surfaces a below-chance ROC-AUC rather than having that disagreement silently corrected away. Three additional, published detectors from the original 27-model survey like UniversalFakeDetect [103], FatFormer [104], and NPR [105] were not integrated in this study and are excluded from the reported results; we treat this as a limitation (Section 6) rather than omit it silently.

Fairness controls applied uniformly across all three tracks. All 36 models score the identical 100-image corpus under the identical neutral-filename protocol described in Section 3.1. Failed or refused responses are marked as unsuccessful, excluded entirely from Accuracy, F1, ROC-AUC, and MCC, and surfaced separately as coverage, so that a model is neither rewarded nor penalized for declining to answer, but its abstention rate remains visible. Latency is compared only within the commercial track, since open-source inference time depends on local hardware and batching, and the imported LLM results were not uniformly timed; comparing latency across these heterogeneous execution environments would not be a fair comparison, so we do not attempt it.

3.3 Metrics↩︎

Let \(TP\), \(FP\), \(TN\), and \(FN\) denote true positives, false positives, true negatives, and false negatives, where the positive class is fake. For a model that produces \(d\) successful verdicts out of the full corpus, all threshold-dependent metrics below are computed over those \(d\) decided images only, at the standard 0.5 decision threshold on \(P(\text{fake})\).

\[\text{Accuracy} = \frac{TP + TN}{d} \label{eq:accuracy}\tag{1}\] Accuracy is the fraction of decided images classified correctly; on this corpus’s 79:21 skew it is easy to game and, as we discuss below, should never be read in isolation.

\[\text{Precision} = \frac{TP}{TP + FP} \label{eq:precision}\tag{2}\] Precision measures, among images flagged fake, what fraction genuinely are fake.

\[\text{Recall} = \frac{TP}{TP + FN} \label{eq:recall}\tag{3}\] Recall (sensitivity, or true positive rate) measures the fraction of genuine fakes a model catches.

\[\text{Specificity} = \frac{TN}{TN + FP} \label{eq:specificity}\tag{4}\] Specificity (the true negative rate) measures the fraction of the 21 authentic images correctly left alone, and is the direct measure of false-alarm behavior on this corpus’s real-image controls.

\[F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \label{eq:f1}\tag{5}\] \(F_1\) is the harmonic mean of precision and recall; like accuracy, it can be dominated by performance on the larger (fake) class alone.

\[\text{MCC} = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} \label{eq:mcc}\tag{6}\] The Matthews correlation coefficient ranges from \(-1\) (total disagreement) to \(+1\) (perfect prediction), with \(0\) equivalent to chance once class prevalence is accounted for. Unlike accuracy or \(F_1\), MCC only rises when a model performs well on both classes simultaneously [78], which is why it is our primary ranking metric: a degenerate “always predict fake” classifier scores approximately \(0.79\) accuracy and a deceptively high \(F_1\) on this corpus, yet an MCC of approximately zero, correctly reflecting its complete lack of real discriminative skill.

ROC-AUC, our tiebreak metric, is computed from the ranked confidence scores rather than the thresholded hard labels, and is therefore threshold-free: it equals the probability that a randomly chosen fake image receives a higher \(P(\text{fake})\) score than a randomly chosen real image, with \(0.5\) corresponding to chance-level ranking and \(1.0\) to perfect separation. Because MCC and ROC-AUC measure genuinely different things, operating-point quality at a fixed threshold versus pure ranking ability independent of any threshold, so the two can diverge sharply for the same model, and Section 5 shows that this divergence, rather than either metric’s absolute value, is the paper’s central empirical finding. Coverage is the fraction of the corpus for which a model returned a successful verdict at all. We do not perform pairwise significance testing across all \(\binom{36}{2} = 630\) model comparisons in this revision; we discuss this omission explicitly in Section 6.

4 Results↩︎

All 36 models were scored on the identical 100-image corpus (79 fake, 21 real) under the protocol of Section 3, ranked by MCC with ROC-AUC as the tiebreak criterion. ROC-AUC is reported alongside every threshold-dependent metric so that both ranking quality and operating-point performance can be interpreted from the same row. Table 1 presents the complete unified leaderboard, combining all three evaluation tracks into a single ranking rather than separating commercial APIs, vision LLMs, and open-source detectors, so that the overall distribution of performance, where approaches cluster, and where they diverge, is visible directly rather than only the strongest performer within each category. Figure 5 visualizes the complete ranking presented in Table 1. It highlights not only the ordering by MCC but also the accompanying ROC-AUC values, making it immediately apparent that several models exhibit strong disagreement between operating-point performance and ranking ability.

3pt

4.1 Overview of the Unified Leaderboard↩︎

Table 1 makes several qualitative patterns visible before any per-track breakdown is needed. The top of the table is dominated by the two commercial APIs that decide every image confidently and consistently, and the gap between this leading pair and the rest of the field is the widest single gap anywhere in the ranking, suggesting these two systems are doing something structurally different from every other model rather than merely performing incrementally better. Immediately below them, the field becomes a genuine mixture of all three paradigms rather than a clean paradigm-by-paradigm block: strong vision LLMs, the best open-source rankers, and the remaining commercial APIs interleave closely, which is itself informative, since it means paradigm membership alone does not reliably predict where a given model lands once the leading pair is set aside.

The bottom of the table tells a different and equally informative story. Every model with a negative or near-zero score is an open-source detector, a vision LLM, or a commercial API whose default decision rule collapses toward predicting a single class almost regardless of the image it is shown; several of these models achieve superficially reasonable accuracy or F1 figures despite this collapse, which is exactly the pathology the ranking metric is designed to expose rather than reward. Reading the table column by column rather than row by row reinforces this: models that look strong on accuracy or F1 do not always look strong on specificity or on ranking power, and the ordering implied by any single column frequently disagrees with the ordering implied by another. No single column of this table should be read as the leaderboard on its own; the ranking metric is deliberately chosen to resist exactly this kind of single-column overinterpretation, and Section 5 unpacks the most consequential instances of that disagreement in detail.

4.2 Per-Track Summary↩︎

Aggregating to the track level confirms the ordering the full leaderboard already suggests: by typical operating-point quality, commercial APIs lead comfortably, the vision-LLM field occupies a middle tier, and the open-source track’s central tendency trails behind both, consistent with most of these detectors having been trained before, or without exposure to, the modern generator mix this corpus specifically targets. Ranking power complicates this picture rather than confirming it. The strongest vision LLM does not reach the ranking power of the single best open-source detector, even though the open-source track’s typical performance is by far the weakest of the three; a single well-designed academic method can out-rank an entire tier of expensive, general-purpose foundation models on pure class separation, even while the median detector in that same open-source track performs far worse than either alternative paradigm. The commercial track retains the single strongest ranking-power result overall, but, as Section 5 shows, that result does not translate into the corresponding operating-point result for the same model, which is the paper’s central finding rather than an incidental detail of this table. Table 2 aggregates the leaderboard into paradigm-level statistics, reporting the strongest and median performance within each evaluation track. This higher-level view highlights the overall performance differences between commercial APIs, vision LLMs, and open-source detectors while reducing the complexity of the full leaderboard.

Figure 6 summarizes the distribution of MCC values within each benchmark track. Commercial APIs exhibit the highest median operating-point quality, whereas the open-source ecosystem shows substantially larger variability, indicating that performance differences within that paradigm are considerably greater than between the commercial offerings.

4.3 Latency (Commercial APIs Only)↩︎

The five commercial providers span a response-time range wide enough to have direct deployment consequences independent of accuracy. The fastest provider responds quickly enough for genuinely interactive, real-time use, while the slowest is an order of magnitude too slow for any synchronous flow and is realistically suited only to asynchronous, batch-style pipelines where an extra delay per item is immaterial. This spread does not track accuracy or ranking power in any simple way: the provider with the strongest operating-point quality in the entire study is also comfortably the fastest, while the provider with the strongest ranking power in the study is among the slowest, so a procurement decision optimizing for either accuracy or ranking power alone would arrive at very different latency consequences depending on which of the two it optimized for. Table 3 summarizes the average inference latency of the five commercial APIs evaluated in this study. Since latency depends heavily on execution environment, comparisons are restricted to the commercial track, where all services were measured under a consistent evaluation setup.

Figure 7 complements Table 3 by illustrating both the latency distribution of the commercial APIs and its relationship with operating-point quality. The figure shows that low inference latency does not necessarily require sacrificing detection performance, as the fastest provider also achieves the highest MCC.

5 Discussion↩︎

Figure 8 illustrates the trade-off between sensitivity and specificity across all evaluated detectors. Several models cluster near perfect recall but very low specificity, revealing a strong tendency to classify nearly every image as fake, whereas the strongest detectors maintain a substantially better balance between detecting manipulations and avoiding false alarms. This distribution reinforces that high recall alone is insufficient for deployment, since practical forensic systems must identify synthetic media without overwhelming users with false alarms on genuine content.

The plane divides the field into three practically distinct populations rather than a single continuum. A small cluster in the upper-right, occupied by Neural Defend and Reality Defender, achieves both high recall and high specificity simultaneously; these are the only two models in the entire study that a deployment could plausibly trust without a compensating human-review step, since they rarely miss a fake and rarely flag a genuine photograph. A second, much larger cluster sits along the top edge at high specificity but with recall spread widely from near-zero to moderate; this is where most open-source detectors and several vision LLMs land, reflecting models that are conservative by default rather than genuinely discriminating, a pattern consistent with the recall-versus-specificity trade-off already visible in the per-model metrics of Table 1. A third population collapses toward the bottom-right corner, combining high recall with specificity near zero; dima806_ai_vs_real, truthscan, and zai_glm52 occupy this region, and each achieves its recall not through genuine discrimination but by defaulting toward “fake” on nearly every input, the same base-rate exploitation this paper returns to below. No model in the study occupies the bottom-left corner, meaning nothing evaluated here is simultaneously bad at catching fakes and bad at preserving real images; every detector fails, when it fails, in one of these two specific, interpretable directions rather than randomly.

5.0.0.1 A Clear Paradigm Ordering, With a Genuine Exception:

The headline ordering commercial APIs ahead of vision LLMs ahead of the open-source median is unsurprising on its face: commercial vendors have direct financial incentive to optimize detection accuracy specifically, vision LLMs are prompted zero-shot for a task they were never trained on, and most publicly distributed open-source detectors were trained against generator families that predate the modern, adversarial mix this corpus targets. What is not unsurprising is that this ordering is a paradigm-level statement about typical performance, not a universal one. DRCT, a single open-source ranker built on contrastive reconstruction training [7], reaches ROC-AUC 0.866, ahead of every vision LLM in the study including the strongest, Claude Opus 4.8, at 0.791. A practitioner who dismissed the entire open-source track on the basis of its poor median performance would have missed the one detector in that track genuinely competitive with, and by this metric better than, general-purpose multimodal LLMs costing far more per query to run.

5.0.0.2 MCC and ROC-AUC Diverge, and the Divergence Is the Finding:

The single most consequential pattern in these results cuts across all three tracks rather than belonging to any one of them: a model’s ranking power and its operating-point quality can point in opposite directions. TruthScan posts the highest ROC-AUC in the entire study (0.915) yet is the only commercial API with a negative MCC (\(-0.130\)), trailing all four of its commercial peers (0.876, 0.646, 0.290, 0.258), because its shipped default threshold flags essentially every image, including all 21 real controls, as fake: its underlying discriminative signal is excellent, but its calibration at the operating point a user actually receives is not. Neural Defend shows the mirror-image pattern, the best MCC by a wide margin (0.876) alongside a near-chance ROC-AUC (0.516), making excellent hard-label decisions from confidence scores that themselves barely separate the two classes. Read together, ROC-AUC and MCC answer different questions, whether a model’s scores rank fakes above reals in principle versus whether its default decision agrees with ground truth and a benchmark reporting only one will systematically mischaracterize one of these two real failure and success modes.

Figure 9 makes this concrete: reading it as four quadrants around AUC \(=0.5\) and MCC \(=0\), the upper-right quadrant holds the bulk of the field, where the two metrics agree and move together, the ordinary case this adversarial corpus is designed to make less common than on an easier benchmark. Neural Defend and TruthScan anchor the two diagonal exceptions that matter most for this paper’s argument, while the lower-left quadrant, where both metrics agree a model is weak, is populated almost entirely by open-source detectors.

Figure 10 shows why: models with superficially similar accuracy produce markedly different confusion compositions. Neural Defend and Reality Defender are dominated by true positives with only a thin false-positive sliver; dima806_ai_vs_real, TruthScan, and zai_glm52 show almost no true-negative bar at all, the base-rate exploitation this paper returns to below; and DRCT’s false-negative bar is nearly as large as a low-recall vision LLM’s, the visual cost of the same conservative default threshold that suppresses its MCC despite ranking fakes above reals better than any other open-source detector in the study. No single bar or metric predicts MCC in isolation; it is the balance across all four confusion cells that the ranking metric is built to capture.

5.0.0.3 The Base-Rate Trap Is Not Hypothetical:

Because the corpus is 79% fake by design, a model that predicts fake indiscriminately scores deceptively well on accuracy and F1 while carrying zero real discriminative skill. This is not a theoretical caveat: zai_glm52 is the concrete instance, posting 71.0% accuracy and F1 of 0.830 while its specificity is exactly zero (it flags all 21 real images as fake) and its MCC is negative (\(-0.152\)). TruthScan, discussed above, exhibits the identical pathology in the commercial track. Any evaluation of these systems that reported accuracy or F1 without also reporting specificity and MCC would have rated both models as reasonable performers; neither is.

5.0.0.4 Scope of These Claims:

Every result in this paper is a point measurement on one fixed, deliberately adversarial, 100-image corpus, evaluated in a single run per model. We do not compute confidence intervals or perform pairwise significance testing across the \(\binom{36}{2} = 630\) possible model comparisons in this revision, both because the corpus is too small to support many of the resulting per-comparison tests meaningfully and because a systematic multiple-comparisons correction across 630 tests was out of scope for this study; we return to this explicitly as future work in Section 6. The vision-LLM results in particular were collected out-of-band rather than through a uniformly controlled live harness, so that track’s numbers should be read as a generic zero-shot baseline rather than a tuned, purpose-built deployment. We interpret every number in this paper as a diagnostic signal on a hard, curated stress test, not as an estimate of population-level accuracy in deployment.

6 Limitations and Future Work↩︎

Several limitations bound the conclusions of this study, each paired with the extension it motivates. First, the corpus is small (100 images) and imbalanced (79:21), limiting confidence in individual metrics, while the smallest per-generator groups are too small for stable rate estimates; a larger, more balanced corpus would strengthen every reported estimate. Second, every result reflects a single run per model, without multiple seeds, bootstrap resampling, confidence intervals, or significance testing across the 630 possible pairwise comparisons; adding bootstrap confidence intervals, multi-seed runs, and a principled subset of statistical comparisons is a direct extension of this work. Third, the vision-LLM results were collected out-of-band, some via vendor APIs and others through browser automation, rather than through the uniformly controlled live harness used for the commercial track, making that track less reproducible. Fourth, provenance for 6 of the 21 fake source groups remains partial or unverified, and three open-source detectors from the original 27-model survey were not integrated in this study. Finally, the generator mix reflects a 2026 snapshot and will inevitably evolve as new generators and detectors emerge.

Future work includes expanding the corpus while preserving its adversarial, edge-case-heavy design; adding bootstrap confidence intervals and multi-seed runs for greater statistical rigor; completing the full 27-model open-source survey by resolving the three deferred detectors; producing per-generator-family analyses to identify where each paradigm fails; and investigating calibrated, per-model decision thresholds rather than shipped defaults, which Section 5 shows can substantially change operating-point quality without affecting underlying discriminative ability.

7 Conclusion↩︎

This paper set out to answer a question no single prior evaluation could: given a detection budget, does paying for a commercial API, prompting a general-purpose vision-language model already available, or self-hosting a free open-source detector actually buy meaningfully different capability against modern, adversarial synthetic media? We built VendorBench-100, a cross-paradigm benchmark evaluating 36 models including 5 commercial APIs, 7 zero-shot vision LLMs, and 24 open-source detectors on a single, deliberately hard 100-image corpus under one shared normalization schema, one abstention policy, and one ranking framework. The headline answer is that paradigm matters on average: commercial APIs achieve the strongest typical performance, vision LLMs occupy a middle tier despite never being trained for this task, and most open-source detectors trail on this adversarial generator mix. Yet this average conceals important exceptions, including DRCT, which outperforms every vision LLM on ranking ability, demonstrating that carefully designed academic detectors remain competitive despite the rapid evolution of proprietary detection systems.

The more consequential result is that ranking power and operating-point quality are fundamentally different properties of a detector. Models with excellent ROC-AUC can still produce unreliable default decisions, while models with modest ranking ability can achieve highly reliable operating-point performance. This divergence appears across all three paradigms and demonstrates that reporting only one metric can misrepresent real deployment behaviour. For a practitioner, the actionable consequence is concrete: any procurement or deployment decision should weigh both metrics side by side rather than trust a single leaderboard score, since either one in isolation can make a genuinely strong detector look weak, or a genuinely unreliable one look strong, depending on which axis is omitted. By releasing the complete evaluation harness, per-image evidence, and aggregated results, we provide a reproducible foundation for future benchmark development and more rigorous cross-paradigm comparisons. We believe this evaluation philosophy will encourage future benchmarks to prioritize fair, standardized, and practically meaningful comparisons over isolated leaderboard performance alone.

Beyond the specific 36 models evaluated here, VendorBench-100 represents a shift toward benchmarking paradigm choice itself rather than individual detectors in isolation. As commercial services evolve behind proprietary interfaces, vision-language models continue to expand beyond their original purpose, and open-source checkpoints proliferate rapidly, meaningful evaluation must remain unified, reproducible, and continuously updated. We hope the adversarial corpus design, shared normalization protocol, and complementary use of MCC and ROC-AUC become standard practice for future deepfake benchmarks, while larger corpora, repeated evaluations, statistical significance testing, and video-based extensions provide an increasingly reliable basis for selecting trustworthy detection systems.

References↩︎