Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks
July 14, 2026
Abstract
Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answering (VQA) benchmarks. We audit these claims and find them fundamentally compromised by data leakage at two hierarchical levels: patient-level leakage, where slides from the same case appear in both training and test folds, and institutional-level leakage, where different cases nonetheless share staining-batch and scanner signatures through a common Tissue Source Site (TSS). By tracing canonical slide, case, and TSS identifiers across major public resources, we document case-level train–test overlaps of 92.3–100% on TCGA-derived benchmarks, together with near-complete TSS overlap. We further demonstrate that both leakage levels are linearly decodable from foundation-model feature space, that they induce a measurable accuracy gap between leaked and audit-clean cases on a published checkpoint, and that across multiple published WSI VLMs, peak reported accuracies concentrate on the most heavily contaminated benchmarks. Therefore, the current WSI VQA evaluation cannot distinguish genuine multimodal reasoning from nearest-neighbor retrieval over memorized institutional and patient-specific artifacts. Finally, we outline concrete recommendations for contamination-free evaluation. By addressing benchmark construction, provenance disclosure, and automated overlap auditing, we aim to guide future research toward verifiable claims of progress.
1 Introduction↩︎
Large labeled datasets have been critical to the success of vision-language models (VLMs) in computational pathology, enabling rapid progress on tasks such as whole-slide image (WSI) captioning, visual question answering (VQA), and diagnostic subtyping. [1] Recent studies routinely report state-of-the-art performance on standard WSI benchmarks under nominally zero-shot evaluation protocols, and these numbers are taken at face value as evidence of genuine out-of-distribution generalization. [2], [3] However, the training datasets behind these models are constructed by aggressively scraping the same monolithic public repository: The Cancer Genome Atlas (TCGA), from which the evaluation benchmarks are independently derived. The possibility that training and test data substantially overlap has received little systematic attention.
This overlap constitutes a fundamental methodological flaw driven by two distinct levels of leakage. At the patient level, evaluating on previously seen slides collapses zero-shot generalization into mere memorized retrieval. At the institutional level, shared Tissue Source Sites (TSS) enable models to exploit laboratory-specific batch effects (e.g., staining and scanner artifacts) as predictive shortcuts rather than learning genuine pathology. Both forms of contamination guarantee silent generalization failure when these systems are deployed in real-world, out-of-distribution clinical settings.
We present the first systematic audit of train-test contamination across the public WSI vision-language ecosystem. By tracing canonical TCGA case and slide barcodes across instruction-tuning corpora and public evaluation benchmarks , we construct a cross-contamination matrix that quantifies overlap at the slide, case, and institutional levels. The results are striking: train-test overlap on TCGA-derived benchmarks reaches \(92.3\%\) to \(100\%\) at the case level, with near-complete institutional (Tissue Source Site) overlap as well. WSI-VQA-test, one of the most widely cited evaluation sets, shares \(100\%\) of its cases with WSI-Bench-train. Figure 1 illustrates the hierarchical structure that gives rise to this contamination and the three partitioning regimes we consider.
In summary, our contributions are:
A two-level taxonomy of data leakage specific to WSI vision-language modeling, separating patient-level from institutional (TSS-level) contamination, together with a reproducible audit pipeline operating on canonical TCGA case, slide, and TSS identifiers.
Empirical evidence that both forms of leakage are linearly encoded in the foundation-model feature space, that they produce a measurable in-benchmark accuracy gap on a published checkpoint, and that this pattern reproduces in a cross-paper meta-analysis spanning multiple published WSI VLMs.
Methodological guidelines, strict slide-, patient-, and TSS-level deduplication, mandatory disclosure of training-test samples overlap at these levels, and the adoption of independently curated multi-center holdout sets as preconditions for credible progress claims in WSI VQA.
2 Related Work↩︎
2.1 WSI Multimodal Benchmarks and Models↩︎
The pathology multimodal literature comprises a tightly coupled set of training corpora, benchmarks, and models. Two structural facts about this ecosystem motivate our audit.
First, slide-level supervision has grown by roughly two orders of magnitude within two years. Early efforts such as ARCH [4] mined \(7.6\)K image-caption groups from textbooks. Web-scale patch corpora including OpenPath [5], Quilt-1M [6], and CONCH [7] then emerged, followed by PathGen-1.6M [8] and slide-level instruction-tuning resources such as WSI-VQA [9], SlideInstruction [10], WSI-Bench [3], and TITAN [11], together with the report-generation corpora WsiCaption [12] and HistGen [13]. Across this trajectory, TCGA dominates the source pool, with BCNB and CPTAC the only widely used alternatives, since the joint requirement of diagnostic-grade WSIs, free redistribution, and paired reports admits no comparable source. The resulting structural convergence, in which many corpora draw from the same case pool and are then re-released as both training and evaluation splits, is the precondition for the contamination we quantify in Section 4.1. We exclude patch-level corpora such as OpenPath and Quilt-1M from the audit because their tiles carry no canonical patient or slide identifier and are not the substrate of slide-level VQA.
Second, the VLMs that consume these corpora are evaluated almost exclusively on the benchmarks those same corpora spawn. Branching from LLaVA-Med [14], pathology-specific systems span three design axes. Along the input-scale axis, patch-level systems such as PathAsst [15], Quilt-LLaVA [16], and PathChat [17] coexist with slide-level systems including SlideChat [10], WSI-LLaVA [3], and TITAN [11], the last pretrained natively over \(8{,}192 \times 8{,}192\) regions. Along the unification axis, CPath-Omni [18] consolidates patch- and slide-level tasks within a single \(15\)B-parameter LMM. Along the reasoning and agency axis, Patho-R1 [15] and PathReasoner-R1 [19] adopt RL-style post-training, while CPathAgent [20] and PathAgent [21] emulate the pathologist’s zoom-and-pan workflow. Every system reports headline state-of-the-art numbers on TCGA-derived test sets, and none report training-test overlap at the case or TSS level. As Section 4.3 shows, the most contaminated benchmarks host the strongest reported gains, leaving it unclear whether current accuracy reflects multimodal reasoning or retrieval over memorized reports.
2.2 Data Leakage in Medical Imaging and Digital Pathology↩︎
Leakage is a documented shortcut hazard across medical imaging. DeGrave et al. [22] showed that COVID-19 chest-radiograph models latched onto hospital-specific markers rather than pulmonary findings. Roberts et al. [23] reported that none of \(415\) reviewed studies were of sufficient quality for clinical use, citing patient duplication and inadequate external validation among the most common failure modes.
Within pathology, prior findings form a hierarchy of granularity that our audit directly extends. At the patch-within-slide level, Bussola et al. [24] demonstrated up to \(41\) pp inflation when tiles from a single patient appear in both training and validation folds. At the slide-within-patient level, Oner et al. [25] reanalyzed a prominent TCGA-LUAD model and showed that its apparent performance collapsed once patient-level splitting was enforced. At the patient-within-institution level, Howard et al. [26] established that TSS identity is recoverable from H&E features with AUROC up to \(0.998\) and proposed preserved-site cross-validation as a remedy. Subsequent work by de Jong et al. [27] showed that the same TSS signature persists in current pathology foundation models, indicating that the hazard has scaled with rather than been mitigated by modern encoders.
These prior audits share two limitations that our work addresses. First, all are confined to unimodal CNNs or self-supervised foundation models, and to classification or representation probing; none audit a multimodal VLM corpus. Second, each leakage level has been studied in isolation. We provide the first joint audit of patient- and institutional-level leakage across the multimodal WSI VLM ecosystem. In this setting the leakage surface is broader, since each contaminated case carries an entire instruction-tuning record rather than a single label, and the downstream consequence is more severe, since a contaminated benchmark cannot distinguish multimodal reasoning from textual retrieval of memorized reports.
3 Formalizing Data Leakage in Pathology VLMs↩︎
Unlike natural image benchmarks, where data leakage typically manifests as exact image duplication, whole-slide images (WSIs) admit a hierarchical form of contamination. This section first describes the architectural paradigm of WSI VLMs that motivates auditing at the encoder feature level, then formalizes the hierarchical contamination structure, defines two levels of leakage, and specifies the metrics used in our audit.
3.1 The Offline-Encoder Paradigm of WSI VLMs↩︎
A defining characteristic of the WSI VLM ecosystem is that visual representation learning and instruction tuning are decoupled into two strictly sequential stages, following the LLaVA-style recipe [14], [28]. In Stage 1, each WSI is processed once through a pretrained visual encoder to produce a fixed slide-level or patch-bag representation. In Stage 2, a lightweight projector and a large language model are trained on these cached features paired with instruction-following text. The visual encoder itself remains frozen throughout instruction tuning.
Within this universal paradigm, current WSI VLMs differ in how their frozen encoder is obtained, and we distinguish two categories. The first one adopts an off-the-shelf pathology foundation model without further modification. Representative examples include SlideChat [10], which composes the patch-level encoder CONCH [7] with a LongNet-based slide-level encoder [29], and WSI-LLaVA [3], which uses a dual-scale patch and slide encoder built on pretrained components. The second one pretrains a pathology-domain CLIP variant specifically for the VLM. Representative examples include Quilt-LLaVA [16], which uses Quilt-CLIP pretrained on Quilt-1M [6], and PathGen-LLaVA, which uses PathGen-CLIP-L pretrained on PathGen-1.6M [8]. The two categories differ in the provenance and supervision of their encoder, but they are identical in one respect that matters for our audit: in both cases, the encoder is fixed before instruction tuning begins and is not updated during it.
This shared design is not a stylistic choice but a computational necessity. A single diagnostic WSI typically contains tens of thousands of \(224 \times 224\) patches, so end-to-end backpropagation through the encoder during instruction tuning is intractable at scale. As a direct consequence, the visual representation that a WSI VLM operates on is fixed by its choice of encoder. Any patient or institutional signal that the encoder embeds into the feature space becomes a permanent input to the projector and the LLM. The downstream parameters can only reweight what the encoder has already encoded; they cannot remove a confounder that is already present. This observation motivates the auditing strategy adopted in Section 4.2, where we probe representative encoders from both categories. Probing the frozen encoder’s feature space is methodologically equivalent to probing the visual input of every VLM built on that encoder, while avoiding the confounds introduced by downstream projector and LLM hidden states.
3.2 Hierarchical Structure and Two Levels of Leakage↩︎
We begin with the WSI data hierarchy. Every diagnostic WSI in the TCGA repository is identified by a barcode encoding a three-level nesting [30]: \[\underbrace{\texttt{TCGA-BH}}_{\text{TSS}}\texttt{-} \underbrace{\texttt{A0BS}}_{\text{Case}}\texttt{-} \underbrace{\texttt{01Z-00-DX1}}_{\text{Slide}} \;\longrightarrow\; \{\,p_1, p_2, \ldots, p_N\,\}_{\text{Patches}}. \label{eq:barcode}\tag{1}\] First, the Tissue Source Site (TSS) identifies the contributing institution. Second, the case identifier maps one-to-one to a patient. Third, the slide suffix (DX1, DX2, \(\ldots\)) distinguishes tissue sections taken from the same case. We write \(\mathrm{case}(s)\) for the case to which slide \(s\) belongs and \(\mathrm{tss}(c)\) for the TSS to which case \(c\) belongs. Sibling slides from a single case share patient phenotype, staining batch, and scanner calibration, while cases from a single TSS share laboratory-level staining and scanning signatures.
Given a training dataset \(\mathcal{D}_{\mathrm{train}}\) and an evaluation benchmark \(\mathcal{D}_{\mathrm{test}}\), each consisting of a set of slides, we define two contamination conditions. First, a test slide \(s \in \mathcal{D}_{\mathrm{test}}\) is patient-leaked if there exists a training slide \(s' \in \mathcal{D}_{\mathrm{train}}\) drawn from the same case: \[\begin{align} \mathcal{L}_{\mathrm{patient}} & = \bigl\{\, s \in \mathcal{D}_{\mathrm{test}} \mid \exists\, s' \in \mathcal{D}_{\mathrm{train}} : \\ & \qquad \mathrm{case}(s) = \mathrm{case}(s') \,\bigr\}. \end{align} \label{eq:patient95leak}\tag{2}\] Second, a test slide is institution-leaked if it is not patient-leaked but shares a TSS with some training slide: \[\begin{align} \mathcal{L}_{\mathrm{TSS}} & = \bigl\{\, s \in \mathcal{D}_{\mathrm{test}} \setminus \mathcal{L}_{\mathrm{patient}} \mid \exists\, s' \in \mathcal{D}_{\mathrm{train}} : \\ & \qquad \mathrm{tss}\bigl(\mathrm{case}(s)\bigr) = \mathrm{tss}\bigl(\mathrm{case}(s')\bigr) \,\bigr\}. \end{align} \label{eq:tss95leak}\tag{3}\] A test slide belonging to neither set is audit-clean. The three sets partition the test benchmark exhaustively, \[\mathcal{D}_{\mathrm{test}} \;=\; \mathcal{L}_{\mathrm{patient}} \;\cup\; \mathcal{L}_{\mathrm{TSS}} \;\cup\; \mathcal{D}_{\mathrm{clean}}, \qquad \label{eq:partition}\tag{4}\] and a benchmark qualifies as a valid zero-shot evaluation only when \(\mathcal{D}_{\mathrm{clean}} = \mathcal{D}_{\mathrm{test}}\). To quantify contamination, we report the case-level and TSS-level overlap ratios, \[\rho_{\mathrm{case}} \;=\; \frac{|\,\mathcal{C}_{\mathrm{test}} \cap \mathcal{C}_{\mathrm{train}}\,|}{|\,\mathcal{C}_{\mathrm{test}}\,|} \;, \\ \rho_{\mathrm{TSS}} \;=\; \frac{|\,\mathcal{T}_{\mathrm{test}} \cap \mathcal{T}_{\mathrm{train}}\,|}{|\,\mathcal{T}_{\mathrm{test}}\,|}, \label{eq:overlap}\tag{5}\] where \(\mathcal{C}\) and \(\mathcal{T}\) denote the sets of cases and TSSs underlying the corresponding slide collections. A benchmark with \(\rho_{\mathrm{case}} = 1\) has been entirely subsumed at the patient level, whereas \(\rho_{\mathrm{TSS}} = 1\) indicates that no institutional generlizationablity remains.
Two scope clarifications are in order. First, we restrict this audit to the instruction-tuning stage of the VLM pipeline, where structured TCGA metadata renders \(\rho_{\mathrm{case}}\) and \(\rho_{\mathrm{TSS}}\) computable via Eq. 5 . Upstream leakage through the visual encoder’s pretraining corpus is acknowledged but falls outside our scope, since such corpora are typically not publicly available. Second, we exclude ROI-level and tile-level datasets such as OpenPath [5] and Quilt-1M [6], which lack canonical patient or slide identifiers and for which the functions \(\mathrm{case}(\cdot)\) and \(\mathrm{tss}(\cdot)\) are undefined. The structural reason such high overlap arises in WSI corpora is that TCGA is the only public repository satisfying the prerequisites for multimodal instruction tuning, which forces all major training dataset and benchmarks to draw from the same pool.
4 Experiments↩︎
This section presents four lines of evidence that the contamination formalized in Section 3 is pervasive and consequential. We first quantify train-test overlap across the public WSI VLM ecosystem (§4.1). We then verify that both levels of leakage leave detectable traces in foundation-model feature space (§4.2), using two structurally distinct encoders to show the effect is not an artifact of any single representation. Finally, we demonstrate that these traces translate into measurable benchmark inflation, across multiple published models (§4.3).
4.1 Cross-Dataset Overlap Audit↩︎
For every pairing of training corpus and evaluation benchmark, we compute both \(\rho_{\mathrm{case}}\) and \(\rho_{\mathrm{TSS}}\) (Eq. 5 ) by matching canonical TCGA case and tissue-source-site (TSS) identifiers. Figure 2 reports the resulting two-panel overlap matrix spanning eight instruction-tuning corpora and four TCGA-derived test benchmarks. Our audit covers the most widely used public WSI datasets, including SlideBench [2], WSI-Bench [3], WSI-VQA [9], HistGen [13], and PathGen-1.6M [8]. Three findings emerge:
First, case-level overlap on TCGA-derived benchmarks is extreme across most training corpora. WSI-Bench-train overlaps with \(92.3\%\), \(90.2\%\), \(100\%\), and \(83.1\%\) of SlideBench-VQA-TCGA, SlideBench-Cap-TCGA, WSI-VQA-test, and WSI-Bench-test cases, respectively. PathGen-1.6M shows comparable contamination at \(94.3\%\), \(94.1\%\), \(90.7\%\), and \(70.8\%\). WSI-VQA-test is fully subsumed by WSI-Bench-train at \(\rho_{\mathrm{case}} = 100\%\), meaning every test patient was seen during training. A particularly striking case is internal to a single benchmark: WSI-Bench-train and WSI-Bench-test themselves share \(83.1\%\) of their cases, indicating that the benchmark curators’ own data partition does not separate patients between the two halves.
Second, TSS-level overlap is uniformly higher than case-level overlap and reveals institutional contamination that a case-level audit alone would miss. Two pairs make this particularly clear. WSI-VQA-train and WSI-VQA-test exhibit \(0\%\) case overlap, indicating that the WSI-VQA authors correctly performed patient-level deduplication, yet their TSS overlap is \(100\%\). Every test patient comes from an institution already represented in training. SlideInstruction++ overlaps with SlideBench-VQA-TCGA at \(0\%\) case but \(85.5\%\) TSS, and with WSI-VQA-test at \(66.3\%\) case but \(94.7\%\) TSS. This pattern shows that patient-level deduplication, while necessary, leaves the same staining, scanning, and laboratory batch effects in both train and test splits, satisfying the formal definition of \(\mathcal{L}_{\mathrm{TSS}}\) leakage (Eq. 3 ).
Third, attempted decontamination is achievable at the case level but remains incomplete at the TSS level. SlideInstruction+ is the only training corpus we audited that achieves \(\rho_{\mathrm{case}} = 0\%\) against all four TCGA-derived test benchmarks. Even so, it retains \(11.0\%\) TSS overlap with SlideBench-VQA-TCGA, \(10.1\%\) with SlideBench-Cap-TCGA, and \(11.3\%\) with WSI-Bench-test. Eliminating patient-level overlap can be accomplished by metadata filtering, but eliminating institutional overlap requires sourcing test data from Tissue Source Sites not represented in any major public training corpus, which TCGA-derived benchmarks structurally cannot provide. This is why genuine zero-shot evaluation in the current ecosystem is feasible only on independently curated, non-TCGA cohorts.
4.2 Feature-Space Evidence of Leakage↩︎
The overlap documented above is defined at the level of metadata identifiers. A natural question is whether this identifier-level overlap translates into measurable redundancy in the learned feature representations that VLMs actually consume. We address this with two complementary experiments, a pairwise similarity analysis and a linear probing study, and run both on two structurally distinct slide encoders. The first is CONCH-v1 [7], a patch-level contrastive encoder whose slide representation we obtain by mean-pooling \(\ell_2\)-normalized patch features. The second is TITAN [11], a natively slide-level pretrained encoder. If leakage were merely an artifact of patch-level mean-pooling, it would not survive in TITAN’s native slide embeddings. Both experiments are computed on the \(6{,}756\) slides for which both encoders’ features are available, so the two encoders are always compared on an identical slide population.
4.2.0.1 Pairwise similarity.
For each encoder we compute cosine similarity between slide embeddings across three configurations of hierarchical proximity: (i) intra-patient pairs (sibling slides from the same case), (ii) intra-TSS, cross-patient pairs (different patients from the same Tissue Source Site), and (iii) cross-TSS pairs (slides from different institutions, \(n{=}20{,}000\) sampled pairs). The two encoders occupy feature spaces with different intrinsic scales. TITAN’s cross-TSS baseline similarity is \(0.233\), while CONCH-v1’s is \(0.664\). Absolute similarity values are therefore not directly comparable across encoders. We report each group relative to its own encoder’s cross-TSS baseline, isolating the excess similarity attributable to shared patient or institutional identity.
Figure 3 reveals a monotonic separation across the three groups for both encoders. For CONCH-v1, intra-TSS pairs sit \(+0.183\) above the cross-TSS baseline and intra-patient pairs \(+0.272\) above it. For TITAN the same ordering holds and the gaps are substantially larger, \(+0.506\) for intra-TSS and \(+0.669\) for intra-patient pairs. All four transitions are statistically significant (Mann-Whitney \(U\), \(p<10^{-200}\)). Two conclusions follow: First, the monotonic structure (cross-TSS \(<\) intra-TSS \(<\) intra-patient) reproduces across two encoders with entirely different pretraining objectives and input granularities. This establishes that the effect is a property of pathology slide content rather than of any single representation. Second, the natively slide-level encoder does not attenuate the effect. TITAN’s intra-patient excess similarity (\(+0.669\)) is more than double CONCH-v1’s (\(+0.272\)), indicating that slide-level pretraining concentrates rather than dilutes patient-identifying signal. These two configurations map directly onto the leakage levels. Intra-patient proximity realizes \(\mathcal{L}_{\mathrm{patient}}\), and intra-TSS proximity realizes \(\mathcal{L}_{\mathrm{TSS}}\).
4.2.0.2 Linear probing for latent confounders.
If patient and institutional signatures are encoded in the feature space, a linear classifier should recover them. We formulate two probing tasks on each encoder’s frozen slide embeddings, patient re-identification (ReID) restricted to multi-slide cases, and TSS detection. For each task we retain only classes with at least two slides, use a stratified \(70/30\) train-test split, and fit a logistic-regression probe. ReID is scored by top-1 accuracy against the chance rate \(1/(\text{\#patients})\). TSS detection is scored by one-versus-rest macro AUROC.
Figure 4 confirms that both confounders are linearly separable in both encoders. Patient ReID reaches \(14.0\%\) for CONCH-v1 and \(20.3\%\) for TITAN, corresponding to \(44.9\times\) and \(65.0\times\) their respective chance baselines. TSS detection yields macro AUROC of \(0.955\) and \(0.980\). Two points deserve emphasis. First, Howard et al. [26] reported in 2021 that ResNet-50 features could detect TSS with AUROC \(>0.95\). Four years and two generations of pathology foundation models later, the institutional confound has not been mitigated, but even sharpened. Second, the consistent ordering TITAN \(>\) CONCH-v1 on both tasks mirrors the pairwise-similarity finding. The encoder designed specifically for slide-level representation encodes patient and institutional identity most explicitly. Any downstream VLM consuming these features therefore inherits both the patient-identity leak and the institutional batch-effect leak. This concern is increasingly relevant because slide-level VLMs are increasingly adopting exactly this class of encoder.
4.3 Cross-Paper Meta-Analysis↩︎
The preceding experiments establish that overlap exists (§4.1) and that it produces detectable feature-space redundancy (§4.2). A remaining question is whether this redundancy inflates the performance numbers reported in the published literature.
| Model | Venue | Training data | Accuracy (%) | \(\rho_{\mathrm{case}}\) (%) | ||
|---|---|---|---|---|---|---|
| 4-5 (l)6-7 | SB-TCGA | WSI-VQA | SB-TCGA | WSI-VQA | ||
| Quilt-LLaVA [16] | CVPR ’24 | Quilt-1M (patch) | 40.2\(\diamond\) | 35.4\(\diamond\) | 0 | – |
| WSI-VQA [9] | ECCV ’24 | WSI-VQA-train | 27.6\(\diamond\) | 46.9\(\diamond\) | 10.4 | internal |
| SlideChat [10] | CVPR ’25 | SlideInstruct | 81.2 | 60.1* | 0 | 80.2\(\dag\) |
| PathGen-LLaVA [8] | ICLR ’25 | PathGen-1.6M | 46.5 | 33.1\(\diamond\) | 94.3 | 90.7 |
| WSI-LLaVA [3] | arXiv ’24/12 | WSI-Bench-train | 82.5\(\diamond\) | 54.6\(\diamond\) | 92.3 | 100.0 |
| MLLM-HWSI [31] | arXiv ’26/03 | WSI-Bench+ | 89.6 | 69.2 | 92.3 | 100.0 |
Table 1 reveals two patterns. First, the highest reported accuracies on SlideBench-VQA-TCGA and WSI-VQA-test consistently co-occur with severe contamination (\(\rho_{\mathrm{case}} > 90\%\), red cells). MLLM-HWSI reports \(89.6\%\) on SlideBench-VQA-TCGA with \(92.3\%\) case overlap. WSI-LLaVA reports \(82.5\%\) with identical overlap. Both report on WSI-VQA-test with \(100\%\) case overlap. Second, models whose training data is non-TCGA or patch-level (LLaVA-Med, Quilt-LLaVA, TITAN) show no detectable case-level overlap under our identifier-based audit. We stress that this is a lower bound rather than a certificate of cleanliness: the pretraining provenance of these models is either undisclosed or composed of patch-level web sources that carry no canonical case or slide identifiers, so the functions \(\mathrm{case}(\cdot)\) and \(\mathrm{tss}(\cdot)\) are undefined on their training data and Eq. 5 cannot resolve any overlap that may nonetheless exist. The horizontal rule in the table separates models with no detectable case-level contamination (above) from models with at least one contaminated (model, benchmark) pair (below).
A notable finding concerns the community’s partial awareness of this issue. SlideChat [10] (CVPR 2025) explicitly acknowledges overlap with WSI-VQA-test and constructs a filtered subset (WSI-VQA*), reporting \(60.1\%\) on it. However, the filter criterion is unspecified, the pre-filter accuracy is unreported, and no subsequent SOTA paper replicates this filtering step. Five models published after SlideChat, namely WSI-LLaVA, PathGen-LLaVA, CPath-Omni and MLLM-HWSI, report on the unfiltered, fully contaminated WSI-VQA-test as zero-shot external validation without acknowledgment.
These observations are consistent with the hypothesis that contaminated benchmarks inflate reported accuracy. However, cross-benchmark comparisons confound leakage with intrinsic task difficulty. The next experiment controls for this confound.
4.4 Within-Benchmark Stratification↩︎
To control for task difficulty, we exploit the fact that SlideBench-VQA-TCGA contains both leaked and audit-clean patients within the same test set. Our audit identifies leaked slides and audit-clean slides among its \(1{,}292\) total. We run inference using the published WSI-LLaVA checkpoint and stratify accuracy by leakage status across cancer type and question category.
| Subset | Leaked | Clean | Gap |
|---|---|---|---|
| Microscopy Qs | \(\mathbf{71.8\%}\) (\(n{=}78\)) | \(64.3\%\) (\(n{=}14\)) | \(+7.5\) |
| UCEC stratum | \(\mathbf{67.4\%}\) (\(n{=}190\)) | \(61.1\%\) (\(n{=}36\)) | \(+6.3\) |
| UCEC\(+\)COAD pool | \(\mathbf{65.9\%}\) (\(n{=}325\)) | \(62.1\%\) (\(n{=}58\)) | \(+3.8\) |
| Diagnosis Qs | \(\mathbf{64.8\%}\) (\(n{=}233\)) | \(61.4\%\) (\(n{=}44\)) | \(+3.4\) |
Table 2 reports the results. Across every well-powered subset, leaked accuracy exceeds clean accuracy. Gaps range from \(+3.4\) pp on Diagnosis questions to \(+7.5\) pp on Microscopy questions, the category most dependent on fine-grained visual recognition. The direction is positive in all four independent subsets. The consistency of the gap across four subsets that vary independently in cancer type and question category constitutes the reliable signal. This result closes the evidence chain. The overlap audit (§4.1) established that contamination is pervasive. The feature-space analysis (§4.2) established that it produces detectable memorization across two structurally distinct encoders. The meta-analysis (§4.3) established that the highest published accuracies co-occur with the most severe contamination. The within-benchmark stratification presented here isolates the effect. On the same benchmark, the same model, the same cancer types, and the same question categories, patients seen during training score \(3\) to \(8\) percentage points higher than unseen patients.
5 Discussion↩︎
5.1 Objections Considered↩︎
The first objection holds that strong performance on a TCGA-overlapping test set still demonstrates clinical competence, since TCGA broadly samples the global oncology distribution. This misidentifies the deployment barrier. The challenge in clinical translation is not diagnosing familiar presentations within a known laboratory but maintaining reliability under the domain shifts that deployment guarantees, including new scanner hardware, altered staining protocols, and unseen patient demographics [23], [27]. A model that retrieves memorized batch artifacts rather than reasoning over morphology will fail silently at a new institution, the failure mode that site-specific shortcuts produce across medical imaging [22], [26]. Evaluation on strictly held-out cohorts is therefore not artificially harsh but the minimum condition under which a generalization claim can be falsified. A second objection holds that exhaustive deduplication across millions of multimodal pairs is computationally impractical and that some leakage is an acceptable price of scale. This conflates two distinct operations. Deduplicating the training datasets is expensive, but auditing the train-test boundary is cheap and non-negotiable, and as Section 4 shows it is decidable in closed form from canonical TCGA identifiers (Eq. 5 ). The premise also fails for pathology in particular. In natural-image domains scaling absorbs diverse visual concepts, whereas uncurated scaling in pathology typically ingests near-redundant tiles of the same phenotype from the same patient and site, so scale amplifies the batch confounders we document rather than diluting them. If the benchmark itself is contaminated, no amount of training-set scaling can be verified as progress.
5.2 Toward Contamination-Free Evaluation↩︎
We outline three mutually reinforcing directions that together form a protocol for credible evaluation:
The first lever is benchmark construction. A credible replacement benchmark must draw its test cohort independently of every public instruction-tuning corpus. Because TCGA is the largest public repository meeting the joint requirement of diagnostic-grade WSIs, free redistribution, and paired reports, removing institutional overlap requires evaluation data from Tissue Source Sites absent from all major training datasets, which TCGA-derived sets structurally cannot provide. We therefore recommend independently curated multi-center holdout cohorts partitioned under preserved-site cross-validation, so that no Tissue Source Site is split across the train-test boundary [26]. Each benchmark should publish its canonical slide, case, and TSS identifiers, retain a privately held evaluation portion that is never released, and migrate over time to submission-based infrastructure in which models are uploaded to an isolated server rather than test data distributed to authors [32]. A public and private split with rate-limited submissions, the standard defense against leaderboard overfitting [32], [33], blunts the slow test-set memorization that even held-out benchmarks otherwise suffer.
The second lever is provenance disclosure. Contamination is invisible today because papers report a single headline zero-shot accuracy without disclosing what the model has seen. Following the documentation discipline of datasheets for datasets and the Data Provenance Initiative [34], we propose three mandatory disclosures for any zero-shot generalization claim: the complete list of training, instruction-tuning, and visual-encoder pretraining corpora consumed by the model; the case-level and TSS-level overlap, \(\rho_{\text{case}}\) and \(\rho_{\text{TSS}}\), between every disclosed dataset and the benchmark, computed with the released audit pipeline, and stratified accuracy on audit-clean and audit-contaminated subsets whenever overlap is non-zero. Emitting these fields as machine-readable metadata [35] would let reviewers verify provenance automatically rather than by manual cross-referencing.
The third lever is automated overlap auditing. Sustained decontamination requires community tooling rather than per-paper diligence. Identifier-level matching, the basis of our audit, is exact but presumes intact metadata, and should be complemented by content-level detection that survives re-export and relabeling, such as perceptual hashing and digital fingerprinting of whole-slide images [36], paralleling the n-gram and canary-based contamination detection now standard in language-model evaluation [33], [37]. For upstream encoder corpora that identifier matching cannot reach, the linear-probing strategy offers a black-box test: recoverability of patient or TSS identity from frozen features is itself a contamination signal even when the pretraining data is proprietary. Released as a continuously maintained open pipeline against which any new corpus or checkpoint can be checked, such tooling would convert contamination auditing from a one-off exercise into a standing community resource.
5.3 Limitations↩︎
Three scope conditions qualify our findings. First, the audit covers only the instruction-tuning stage, where structured TCGA metadata makes \(\rho_{\text{case}}\) and \(\rho_{\text{TSS}}\) computable in closed form; leakage propagated through the proprietary visual-encoder pretraining datasets, which are generally undisclosed, remains unquantified, though the linear probe offers a black-box signal for this upstream channel. Second, the within-benchmark stratification is computed on a single contaminated checkpoint by partitioning its test set into audit-clean and audit-contaminated slides within the same benchmark; because the audit-clean partition is comparatively small, the +3 to +8 pp gap is directionally consistent across four independently varying subsets but imprecisely estimated in magnitude. Third, our audit detects overlap via TCGA identifiers, so it assumes intact provenance metadata and yields a lower bound on true contamination: slides whose barcodes have been stripped, re-exported, or relabeled escape identifier matching and would require the content-level detection we recommend in Section 5.2. Despite these conditions, our core result remains robust across two distinct encoders via exact identifier matching. These boundaries simply mark where quantification ends and our proposed replacement benchmark begins.
6 Conclusion↩︎
We present the first systematic audit of train-test contamination across the public WSI vision-language ecosystem. Tracing TCGA case, slide, and TSS barcodes across the instruction-tuning datasets and TCGA-derived benchmarks, we document 92.3% to 100% case-level overlap. We show that both leakage levels are linearly decodable from the feature spaces of two structurally distinct foundation encoders. Notably, the natively slide-level encoder concentrates rather than dilutes patient-identifying signals, translating into a consistent accuracy gap between leaked and clean slides. Consequently, the highest reported state-of-the-art accuracies across WSI VLMs cluster on the most heavily contaminated benchmarks. These findings do not diminish the engineering achievements of recent pathology VLMs, but establish that current zero-shot generalization claims are unreliable: present benchmarks cannot distinguish genuine multimodal reasoning from nearest-neighbor retrieval over memorized institutional and patient-specific artifacts. The remedy is a strict change in evaluation practice. Until the community enforces slide-, patient-, and TSS-level deduplication, discloses cohort overlaps, and evaluates on independently curated multi-center holdouts, reported gains in WSI vision-language modeling cannot be separated from memorization artifacts.