1 Introduction↩︎

The central promise of computational pathology is to bring AI assistance to pathologists, rather than asking pathologists to adapt to AI infrastructure. This is increasingly urgent: global cancer incidence was close to 20 million new cases in 2022 and is projected to reach 35 million by 2050 [1]. At the same time, histopathology workforces are under pressure from rising workload, increasing diagnostic complexity and aging workforces [2]. AI systems could support screening, triage, and prescreening for therapy-relevant biomarkers, but only if they integrate into routine workflows without additional operational burden.

Current pathology foundation models (PFMs) have changed the performance landscape but not yet the deployment landscape. Their large vision-transformer backbones and gigapixel whole-slide image (WSI) pipelines often require dedicated GPUs, local clusters, or cloud processing. This creates a workflow mismatch: instead of augmenting existing pathology practice, many methods require additional digitization, transfer, preprocessing, and compute steps. For clinical adoption, models should run where tissue is examined and where data are produced: on existing pathology workstations, ordinary clinic computers, laptops, or eventually device-integrated microscopes. Such local inference also reduces data movement, allowing sensitive images to remain inside the pathology department.

Recent work highlights the feasibility and importance of this direction. Choudhury et al. developed a low-cost, open-source microscopy platform for slide capture and computational analysis, including Raspberry-Pi-based local compute for automated histopathology evaluation [3]. LitePath similarly argues that PFM accessibility depends not only on accuracy but also on deployability, reporting large reductions in parameters, FLOPs, energy, and runtime through distilled LiteFM models and adaptive patch selection [4]. These studies motivate a practical target for pathology AI: compact encoders that can run inside existing departmental infrastructure, including low-cost workstation and microscope-connected edge setups, rather than requiring external high-performance compute.

We propose Multi-teacher Contrastive Distillation (MuCoDi), a framework for learning edge-efficient pathology encoders (Fig. 1). MuCoDi distills complementary representations from multiple state-of-the-art PFMs into a substantially smaller student backbone designed for local inference. Instead of matching only individual teacher features, MuCoDi preserves teacher-induced relational structure through contrastive distillation, enabling a compact model to inherit complementary morphology-aware embeddings while reducing latency and hardware demands.

Our contributions are threefold. First, we introduce an efficient contrastive multi-teacher knowledge distillation pretraining framework for edge-oriented pathology representation learning using only 11.8K unlabeled TCGA WSIs for student pretraining. Second, we demonstrate extreme latency improvements while maintaining strong downstream classification performance, including sub-million-parameter students for severe edge constraints. Third, we benchmark the latency-accuracy trade-off of compact pathology encoders on low-cost edge hardware, targeting future integration into routine workstation and microscope-based pathology workflows.

2 Related work↩︎

3pt

2.0.0.1 Pathology foundation models.

Large self-supervised pathology foundation models (PFMs) such as UNI [8], Virchow [9], Prov-GigaPath [10], H-optimus-0/1 [5], [11], and Virchow2 [7] have substantially improved reusable histopathology representations. However, this progress largely follows a scaling trend: recent models range from hundreds of millions to over one billion parameters and process whole-slide images through large numbers of tiles, making training and deployment costly [12].

2.0.0.2 Knowledge distillation in pathology.

Knowledge distillation (KD) compresses teacher models into a student [13]. Contrastive representation distillation instead transfers relational structure in feature space [14]. In computational pathology, multi-model knowledge has been used to build stronger or more task-adapted representations: GPFM combines expert and self-KD [15], COBRA treats embeddings from multiple PFMs as feature-space augmentations for slide-level contrastive learning [16], and HistoMILKD distills multiple PFMs into a MIL-based WSI classifier [17]. These methods demonstrate the value of multi-PFM supervision, but they primarily target generalization or slide-level prediction rather than compact feature extractors for efficient edge inference.

2.0.0.3 Efficient pathology foundation models.

Recent work has begun to address PFM efficiency. H0-mini is an 86M-parameter ViT-B/14 distilled from a large pathology FM for robust and efficient inference [18]. Virchow2G Mini is a 22M-parameter distillation of the 1.9B-parameter Virchow2G [7]. LitePath introduces LiteFM, a multi-teacher distilled ViT family with adaptive patch selection for resource-efficient WSI analysis [4]. Nevertheless, these models remain ViT-based and are not primarily designed or benchmarked for direct deployment on low-cost edge devices integrated into routine pathology hardware.

2.0.0.4 Efficient architectures.

In general computer vision, mobile-first backbones explicitly optimize latency and hardware efficiency. MobileNetV3 uses hardware-aware search [19], MobileOne targets sub-millisecond mobile inference through reparameterized convolutional design [20], RepViT revisits mobile CNNs through ViT-inspired design [21], and EfficientFormer shows that transformer-like models can approach MobileNet-speed inference [22]. MuCoDi brings this design philosophy to computational pathology by distilling multiple large PFMs into edge-oriented students for efficient integration into microscope or workstation workflows.

3 Methods↩︎

3.1 MuCoDi pretraining↩︎

MuCoDi adapts the two-view contrastive learning recipe of MoCo v3 [23] to multi-teacher distillation. MoCo v3 contrasts queries against keys produced by a momentum encoder. MuCoDi keeps the contrastive objective but replaces this key encoder with cached embeddings from frozen pathology foundation models, which act as fixed teacher keys. For each training tile \(x\), we precompute teacher features \(k_t(x)\) for \(T=3\) teachers: Virchow2, UNI2, and H-Optimus-1. The teachers remain frozen. Only the lightweight student backbone \(f_\theta\) and teacher-specific linear projection heads \(g_t\) are optimized. The heads map the shared student feature to 1280 dimensions for Virchow2 and 1536 dimensions for UNI2 and H-Optimus-1. For each tile, we sample two MoCo-v3-style stochastic views \(x_1,x_2\) resized to (\(224\times224\)) px. Let \(q_{t,v}=\mathrm{norm}(g_t(f_\theta(x_v)))\) be the student query for teacher \(t\) and view \(v\in\{1,2\}\). Let \(K_t=\{k_t^{(i)}\}_{i=1}^{B_g}\) denote the L2-normalized teacher keys gathered across all GPUs for the current global batch of size \(B_g\). For a query from tile \(j\), the positive key \(k_t^{+(j)}\) is the cached teacher embedding of the same underlying tile. For one teacher and view, the InfoNCE term [24] is \[\ell_{t,v}^{(j)} = -\log \frac{\exp\left(q_{t,v}^{(j)\top} k_t^{+(j)} / \tau\right)}{\sum_{i=1}^{B_g} \exp\left(q_{t,v}^{(j)\top} k_t^{(i)} / \tau\right)} ,\] where \(\tau\) is the temperature. Following MoCo v3, the optimized per-sample loss is scaled as \(\mathcal{L}_{t,v}^{(j)}=2\tau\ell_{t,v}^{(j)}\), which makes the gradient magnitude independent of temperature. The final MuCoDi objective is the unweighted sum over teachers and views, \[\mathcal{L}_{\mathrm{MuCoDi}} = \sum_{t=1}^{T}\sum_{v=1}^{2}\frac{1}{B_g}\sum_{j=1}^{B_g}\mathcal{L}_{t,v}^{(j)} .\] Teacher embeddings are stored with the image tiles and loaded without gradients, and negatives are provided by cross-GPU gathering of the current batch.

3.1.0.1 Training details.

In the student sweep, all students are pretrained for 10 epochs with AdamW using PyTorch default betas, base learning rate \(2\times10^{-2}\), weight decay \(10^{-6}\), global batch size 2048, and temperature \(\tau=0.2\). The learning rate is linearly warmed up for the first 5% of optimizer steps and then cosine-decayed to zero. Gradients are clipped to a maximum norm of 1.0. Training uses bfloat16 automatic mixed precision, SyncBatchNorm, and distributed data parallelism on four GPUs. The augmentation pipeline follows the MoCo v3 two-crop scheme with random resized crops, color jitter, grayscale conversion, horizontal flips, asymmetric Gaussian blur, and solarization.

3.2 Data and task selection↩︎

MuCoDi is pretrained on TCGA tissue tiles with cached teacher features [25]. The pretraining set comprises 11,803 FFPE diagnostic WSIs and 14,280,892 tissue tiles. As shown in Tab. 1, this student distillation corpus uses approximately 6\(\times\) fewer WSIs than LiteFM and over 260\(\times\) fewer than Virchow2. Pretraining and evaluation use the same 2.0 \(\mu\)m/px (5\(\times\)) physical resolution, which reduces tile count and compute [26]. Downstream evaluation follows the weakly supervised tile-feature benchmarking paradigm used in recent slide-representation and pathology foundation-model studies [12], [16]. Task heads are trained on TCGA and deployed unchanged on matched CPTAC cohorts [27]. Evaluation uses the matched TCGA set (3,933 patients, 4,633 slides) and matched CPTAC set (1,307 patients, 3,736 deployable slides), covering 23 clinically curated binary endpoints with at least 20 minority-class patients. All three teacher PFMs achieve at least 0.65 AUROC on internal TCGA evaluation for each endpoint, ensuring that each endpoint has a morphology-predictable signal before external deployment. The selected tasks span BRCA (TP53, ESR1, and PGR), CRC (TP53 and MSI), GBM (NF1), KIRC (SETD2, grade, stage, and T stage), LUAD (EGFR, STK11, TP53, and T stage), NSCLC subtype, and UCEC (BRD4, KMT2A, MGA, NF1, PTEN, ROS1, TP53, and grade).

3.3 Evaluation protocol↩︎

Downstream performance is measured by freezing the student as a tile encoder, extracting one feature vector per tile, and training slide-level multiple-instance learning (MIL) heads with the STAMP workflow [28] on TCGA using patient-level stratified 5-fold cross-validation. External generalization is evaluated by applying each TCGA-trained fold checkpoint unchanged to the matched CPTAC cohort.

3.3.0.1 Slide-level heads and metrics.

Main results use the STAMP transformer MIL head. Performance is reported as AUROC, F1 score, and AUPRC with 95% confidence-interval half-widths.

3.3.0.2 Edge inference benchmark.

Deployment efficiency is measured on a Raspberry Pi 5 Model B with a quad-core Cortex-A76 CPU, 16 GB RAM, and no inference accelerator. Latency is measured in single-threaded float32 CPU mode without quantization and with batch size 1. MobileOne and RepViT models are first converted to their fused inference-time forms. For each model, we report parameters, GFLOPs for one (\(224\times224\)) px RGB tile, and median single-tile latency from 100 timed forward passes after 20 warmup passes.

4 Results↩︎

4pt

3pt

4.0.0.1 External validation.

We compare MuCoEdge students against their teacher PFMs and LiteFM competitors on the 23 selected downstream classification tasks (Tab. 2). Teacher PFMs retain the highest external AUROC, with Virchow2 reaching 71.8% and UNI2 71.7%. RepViT-based students close most of this gap at much smaller scale: MuCoEdge-R2.3 and MuCoEdge-R1.5 reach 71.0% AUROC, while MuCoEdge-R1.1 and MuCoEdge-R1.0 reach 70.9%. The strongest edge trade-off is achieved by smaller RepViT students. MuCoEdge-R0.9 reaches 70.5% AUROC, 51.3% F1, and 52.4% AUPRC with 4.7M parameters, outperforming LiteFM-L on all three external metrics despite using 18\(\times\) fewer parameters and 21\(\times\) fewer GFLOPs. MuCoEdge-R0.6 remains competitive with LiteFM while using only 2.2M parameters. These results are obtained from only 11.8K student-pretraining WSIs, approximately 6\(\times\) fewer than LiteFM and over 260\(\times\) fewer than Virchow2 (Tab. 1). LiteFM also includes both TCGA and CPTAC in its pretraining corpus. At the extreme edge end, sub-million MobileOne variants retain 66.5–66.9% external AUROC with only 0.2–0.7M parameters. Internal TCGA evaluation shows the same trend: MuCoEdge-R2.3 obtains the highest mean AUROC (77.1%), and several students remain within the teacher range. Per-organ results (Tab. 3) show that performance is not driven by a single tissue type. MuCoEdge variants obtain the best average F1 and are competitive across colon, kidney, lung, and uterine tasks.

4.0.0.2 Efficiency and edge deployment.

The efficiency gains are substantial (Tab. 1, Fig. 2a). Speedups are reported against Virchow2, the most efficient high-performing teacher reference in our benchmark. MuCoEdge-R0.9 improves over LiteFM-L while using 18\(\times\) fewer parameters, 21\(\times\) fewer GFLOPs, and 12.9\(\times\) faster Raspberry-Pi inference. MuCoEdge-R0.6 reaches 69.4% AUROC with 2.2M parameters and runs 6.2\(\times\) faster than LiteFM and 2.0\(\times\) faster than LiteFM-S. The sub-million MobileOne variants provide the most aggressive latency reduction, achieving 327–605\(\times\) single-tile speedups over Virchow2 while retaining 66.5–66.9% external AUROC. Together, these results define a practical edge-deployment spectrum, from higher-accuracy RepViT models to sub-million-parameter MobileOne models for strict latency budgets.

5 Conclusion↩︎

MuCoDi transfers multiple PFM feature spaces into compact, edge-oriented pathology encoders. In broad internal TCGA and external CPTAC evaluation, MuCoEdge-R0.9 improves over LiteFM-L with 18\(\times\) fewer parameters, MuCoEdge-R0.6 remains competitive with LiteFM at 2.2M parameters, and sub-million MobileOne variants retain 66.5–66.9% external AUROC with 327–605\(\times\) Raspberry-Pi speedups over Virchow2. From a clinical perspective, encoders that run directly on departmental workstations or microscope-attached edge devices remove the data-transfer and compute barriers that currently keep PFM-based decision support out of routine diagnostic workflows.

References↩︎