1 Introduction↩︎

In computational pathology, WSI analysis is a cornerstone for building decision-support diagnostic systems. The rapid progress in deep learning has accelerated the development of automated tools capable of analyzing and quantifying complex tissue and cellular characteristics in WSIs. However, due to the gigapixel-scale of WSIs, direct end-to-end optimization remains computationally intractable. Consequently, MIL methods have emerged as a practical pipeline, where a slide is decomposed into local patch-level instances, encoded into feature embeddings, and aggregated into a global slide-level representation for prediction. Throughout these standard frameworks, slide-level diagnosis has been overwhelmingly formulated as a flat, mutually exclusive classification task over discrete label indices, which contradicts the structured nature of clinical label space (Figure 1).

Clinical diagnosis is inherently hierarchical and often proceeds in a coarse-to-fine manner. A pathologist may first assign a broad diagnostic category (e.g., “Malignant”) and then refine the assessment into specific subtypes or grades (e.g., “Ductal Carcinoma In Situ, Invasive Carcinoma”) (Figure 1). Yet, when labels are treated as unrelated class indices, standard optimization objectives provide no explicit incentive to preserve coarse-to-fine (or parent–child) consistency or to encode semantic proximity among taxonomically related diagnoses. These gaps motivate moving beyond conventional flat classification toward formulations that natively capture the taxonomic relationships inherent in clinical reasoning.

Recent advances address this structural gap through two divergent paradigms: discrete hierarchical classifiers and unconstrained VLMs. Hierarchical classifiers [1] incorporate label taxonomies via multi-branch architectures but still rely on flat, discrete class indices, failing to capture the semantic continuity between diagnostic labels. Conversely, VLM-based models [2], [3] output clinically readable text that naturally accommodates varying granularities. However, robust diagnostic generation for WSIs remains challenging. Without explicit taxonomic constraints in the learned representation space, these generative models are prone to hallucinations, inconsistent diagnoses, and parent–child violations. Hence, neither paradigm simultaneously achieves structured reasoning, taxonomic faithfulness, and interpretable text generation.

To bridge this gap, we propose TaxoMIL, a taxonomy-constrained MIL framework that casts WSI diagnosis as hierarchical, taxonomy-consistent text generation. A standard MIL representation is modulated into coarse- and fine-level representations via a taxonomy-aware conditioning mechanism, and a dual-head Transformer decoder generates the corresponding diagnostic label text. To enforce clinical structure, taxonomy-guided objectives shape the label embedding space and align slide-level representations with taxonomy-consistent semantics, improving coarse-to-fine coherence and reducing hierarchical contradictions.

Our contributions are three-fold:

• We propose a taxonomy-constrained framework that reformulates WSI diagnosis as a multi-granularity text generation task to enable structured, hierarchy-aware clinical reasoning.

• We introduce taxonomy-guided hierarchical objectives that explicitly structure the label embedding space and ground slide representations within the clinical taxonomy.

• We evaluate on three WSI datasets (GastWSI, BRACS[4], and PANDA[5]), demonstrating that TaxoMIL consistently outperforms state-of-the-art MIL classifiers and VLM-based methods.

2 Related Work↩︎

MIL for WSI Classification. WSIs are commonly modeled as bags of patch instances, where MIL aggregates instance features into a slide-level representation. Early approaches relied on permutation-invariant pooling (e.g., mean/max pooling), while attention-based MIL introduced instance weighting and became a standard backbone for WSI analysis [6]. Subsequent works incorporated instance selection, multi-branch attention, or cluster-level constraints to handle intra-slide heterogeneity [7], while Transformer-based aggregators modeled inter-instance interactions via self-attention [8]. More recently, state space models [9], [10] offered linear-time aggregation for long-instance sequences. In parallel, multimodal and language-guided MIL has been explored to inject semantic priors into WSI representations, such as ViLa-MIL[11], which leverages vision–language pretraining to improve slide-level discrimination and interpretability.

Hierarchical MIL Learning in Pathology. To better capture clinical taxonomies, several WSI methods explicitly incorporate label hierarchies. HMIL [1] exploited label hierarchies at both instance and bag levels via a dual-branch design with class-wise attention, hierarchical alignment modules, and supervised contrastive learning. Beyond pathology, Chang et al.[12] utilized top-down traversal over a coarse-to-fine label hierarchy, showing benefits from disentangling coarse- and fine-level feature learning. While our approach conceptually aligns with these multi-granular, hierarchical formulations, we impose hierarchy directly on textual label embeddings and perform WSI-to-text generation rather than through multi-head classification over discrete IDs.

Vision–Language Modeling and Text Generation for Pathology. VLMs combine a visual encoder with a pretrained language model to generate interpretable outputs, and recent pathology works extend this paradigm to slide- or region-level retrieval, question answering, and instruction-following assistants [2], [3], [13]. WSI-VQA [3] reframes diverse slide-level objectives into a generative visual question answering setting, enabling question-guided reasoning on WSIs. CAMP[13] similarly provides patch- and slide-level pathology classification that casts prediction as text generation, capturing task-specific knowledge via trainable adaptors. SlideChat[2] further moves toward instruction-following assistants, utilizing two-stage cross-domain alignment and visual instruction learning. These approaches can provide clinically readable outputs and reduce the semantic gap between model predictions and diagnostic terminology. Nevertheless, accurate and taxonomically coherent WSI diagnosis by generation remains challenging.

3 Methods↩︎

3.1 Problem Formulation↩︎

In standard MIL practice, a WSI \(I\) is decomposed into a disjoint set of patches: \(\mathbf{P}=\{\mathbf{p}_i\}_{i=1}^{N_p}\), where \(N_p\) denotes the number of patches. A feature extractor \(\mathcal{E}(\cdot)\) encodes each patch into embedding space: \(\mathbf{x}_i = \mathcal{E}(\mathbf{p}_i)\in\mathbb{R}^{D}\), \(\mathbf{X}=\{\mathbf{x}_i\}_{i=1}^{N_p}\), where \(D\) is the embedding dimension. A MIL aggregator \(\texttt{Agg}(\cdot)\) pools these patch-level embeddings into a slide-level embedding: \(\mathbf{z}=\texttt{Agg}(\mathbf{X})\in\mathbb{R}^{D}\). For flat classification, \(\mathbf{z}\) is fed to a linear classifier to predict a slide-level label \(y \in \mathcal{Y}\), where \(\mathcal{Y}\) denotes a set of classes.

Clinical diagnosis, however, inherently follows a structured taxonomy. To formalize this, let \(\mathcal{C}\) and \(\mathcal{F}\) denote sets of coarse- and fine-level diagnostic text labels (i.e., natural language class names), respectively. The taxonomy mapping \(\mathcal{H}: \mathcal{F} \rightarrow \mathcal{C}\) specifies the parent–child relationship, such that ground truth text labels satisfy \(\mathcal{H}(\mathbf{y}^{(f)}) = \mathbf{y}^{(c)}, \mathbf{y}^{(c)} \in \mathcal{C}, \mathbf{y}^{(f)} \in \mathcal{F}\). The objective of TaxoMIL is to learn a mapping from the WSI space to taxonomically coherent text label pairs: \[(\hat{\mathbf{y}}^{(c)}, \hat{\mathbf{y}}^{(f)}) = f_{\theta}(I), \qquad \text{s.t.} \qquad \mathcal{H}(\hat{\mathbf{y}}^{(f)}) = \hat{\mathbf{y}}^{(c)}\] where \((\hat{\mathbf{y}}^{(c)}, \hat{\mathbf{y}}^{(f)})\) are the generated coarse- and fine-level diagnostic text strings, and \(f_{\theta}\) represents the proposed taxonomy-constrained MIL framework.

3.2 Overview of TaxoMIL↩︎

TaxoMIL reframes WSI diagnosis as a taxonomy-constrained hierarchical text generation problem. As illustrated in Figure 2, the framework comprises three major components: (1) a MIL backbone (\(\mathcal{E}(\cdot)\), \(\texttt{Agg}(\cdot)\)) producing a global slide-level representation \(\mathbf{z}\in\mathbb{R}^{D}\); (2) a dual-branch multimodal conditioning module \(\mathcal{V}\) that transforms \(\mathbf{z}\) into coarse- and fine-level visual representations conditioned on label context; and (3) a dual-head decoder \(\mathcal{T}\) generating the corresponding diagnostic label text. The algorithm of TaxoMIL is provided in Supp. 7.

3.3 Dual-branch Multimodal Conditioning Module↩︎

Given the global slide-level representation \(\mathbf{z}\in\mathbb{R}^{D}\) and semantic priors from the clinical taxonomy, the dual-branch conditioning module \(\mathcal{V}\) produces two granularity-specific conditioning sequences for the dual-head decoder \(\mathcal{T}\). This supports decoupled label-text generation and provides an interface for injecting taxonomy semantics.

Embedding Decoupling. The MIL backbone produces the global slide-level representation \(\mathbf{z}\in\mathbb{R}^{D}\). To support decoupled hierarchical reasoning, we split \(\mathbf{z}\) into two equal-sized embeddings along the feature dimension: \(\mathbf{z} = [\mathbf{z}_c ; \mathbf{z}_f]\), where \(\mathbf{z}_c\), \(\mathbf{z}_f \in \mathbb{R}^{D/2}\) denote coarse- and fine-level embeddings, respectively. Each embedding is projected into the embedding space of the dual-head text decoder \(\mathcal{T}\) via a dedicated lightweight adaptor: \(\mathbf{v}^{(c)}=g_c(\mathbf{z}_c), \mathbf{v}^{(f)}=g_f(\mathbf{z}_f)\), where \(\mathbf{v}^{(c)}, \mathbf{v}^{(f)} \in \mathbb{R}^H\), and \(H\) is the embedding dimension of \(\mathcal{T}\). Each adaptor is a linear projection followed by layer normalization and GELU activation.

Label Embedding Construction. TaxoMIL uses label names not only as generation targets but also as semantic priors for taxonomy-constrained optimization. Recall that \(\mathcal{C} = \{\mathbf{y}_i^{(c)}\}_{i=1}^{|\mathcal{C}|}\) and \(\mathcal{F} = \{\mathbf{y}_i^{(f)}\}_{i=1}^{|\mathcal{F}|}\) denote the coarse- and fine-level text labels, respectively. Each text label is tokenized into a sequence of token IDs using \(\texttt{TOK}\), the designated tokenizer of \(\mathcal{T}\). The corresponding token embeddings are retrieved and combined via mean-pooling to form base label-text representations \(\{\mathbf{b}_i^{(c)}\}_{i=1}^{|\mathcal{C}|}\) and \(\{\mathbf{b}_i^{(f)}\}_{i=1}^{|\mathcal{F}|}\). A shared learnable projection head \(\phi(\cdot)\) is then applied to produce coarse- and fine-level label-text embeddings: \(\{\mathbf{t}_i^{(c)}\}_{i=1}^{|\mathcal{C}|} = \{\phi (\mathbf{b}_i^{(c)})\}_{i=1}^{|\mathcal{C}|}\) and \(\{\mathbf{t}_i^{(f)}\}_{i=1}^{|\mathcal{F}|} = \{\phi (\mathbf{b}_i^{(f)})\}_{i=1}^{|\mathcal{F}|}\). These embeddings serve as the semantic anchors for the taxonomy-constrained objectives.

Conditioning Sequence Construction. To condition hierarchical label-text generation on both visual evidence and taxonomy semantics, we construct a short decoder prefix for each granularity by concatenating four components: (1) a granularity-specific visual token embedding (\(\mathbf{v}^{(c)}\) or \(\mathbf{v}^{(f)}\)), (2) a learnable image token embedding (\(\mathbf{e}_{\text{img}}\)), (3) a granularity-specific label-context token embedding (\(\mathbf{e}_{\text{ctx}}^{(c)}\) or \(\mathbf{e}_{\text{ctx}}^{(f)}\)), and (4) prompt token embeddings (\(\mathbf{E}_{\text{prompt}}\)). Formally, the coarse- and fine-level conditioning sequences are formed as follows: \[\mathbf{E}^{(c)} = [\,\mathbf{v}^{(c)};\, \mathbf{e}_{\text{img}};\, \mathbf{e}^{(c)}_{\text{ctx}};\, \mathbf{E}_{\text{prompt}}\,], \qquad \mathbf{E}^{(f)} = [\,\mathbf{v}^{(f)};\, \mathbf{e}_{\text{img}};\, \mathbf{e}^{(f)}_{\text{ctx}};\, \mathbf{E}_{\text{prompt}}\,].\] Here, \(\mathbf{e}_{\text{img}} \in \mathbb{R}^{H}\) is a learnable token embedding for the special image token <|img|>, which serves as a fixed visual reference for decoding. The label-context token embeddings \(\mathbf{e}_{\text{ctx}}^{(c)}, \mathbf{e}_{\text{ctx}}^{(f)} \in \mathbb{R}^{H}\) provide global semantic priors over their respective label space, grounding the label-text generation in the clinical taxonomy. We compute these embeddings by averaging the corresponding base label embeddings across all classes at each granularity: \(\mathbf{e}_{\text{ctx}}^{(c)}=\frac{1}{|\mathcal{C}|}\sum_{i=1}^{|\mathcal{C}|}\mathbf{b}_i^{(c)}\) and \(\mathbf{e}_{\text{ctx}}^{(f)}=\frac{1}{|\mathcal{F}|}\sum_{i=1}^{|\mathcal{F}|}\mathbf{b}_i^{(f)}\). The prompt token embeddings \(\mathbf{E}_{\text{prompt}} \in \mathbb{R}^{L_p \times H}\) serve as a natural language instruction that steers the dual-head text decoder \(\mathcal{T}\) toward generating short label-like outputs (i.e., class name phrases). The prompt is a fixed template string, tokenized and mapped to the embedding space of \(\mathcal{T}\) with a prepended BOS token. To reduce sensitivity to specific prompt phrasing, we adopt a template sampling strategy during training, randomly sampling from a curated set of 10 semantically equivalent but syntactically diverse templates. At inference, a fixed prompt is used for deterministic decoding. The prompt templates are listed in Supp. 8 and Table 5.

3.4 Dual-Head Text Decoder↩︎

The dual-head text decoder \(\mathcal{T}\) generates coarse- and fine-level diagnostic label text from the branch-specific conditioning sequences \(\mathbf{E}^{(c)}\) and \(\mathbf{E}^{(f)}\). To encourage consistent label text generation, we use a shared autoregressive backbone \(\mathcal{T}_{shared}\) and attach two lightweight heads to perform granularity-specific generation. Specifically, given \(\mathbf{E}^{(c)}\) and \(\mathbf{E}^{(f)}\), \(\mathcal{T}_{shared}\) produces hidden states: \[\mathbf{M}^{(c)}=\mathcal{T}_{shared}(\mathbf{E}^{(c)}) \in \mathbb{R}^{L^{(c)} \times H}, \qquad \mathbf{M}^{(f)}=\mathcal{T}_{shared}(\mathbf{E}^{(f)}) \in \mathbb{R}^{L^{(f)} \times H}.\] where \(L^{(c)}\) and \(L^{(f)}\) denote the maximum token lengths for coarse- and fine-level predictions, respectively. Each hidden state is passed through a generation head (one Transformer decoder block) and a separate unembedding projection to produce token logits, from which the coarse- and fine-level label texts \(\hat{\mathbf{y}}^{(c)}\) and \(\hat{\mathbf{y}}^{(f)}\) are decoded autoregressively.

3.5 Training Strategies↩︎

TaxoMIL is trained end-to-end with a primary hierarchical label-text generation loss for both granularities, complemented by a supervised contrastive loss on visual embeddings and taxonomy-guided losses that (1) explicitly structure the label embedding space according to the clinical hierarchy and (2) ground visual embeddings in this taxonomy-structured label space.

Overall Objective. The overall training objective is given by: \[\mathcal{L} = w_{gen}\mathcal{L}_{GEN} + w_{hier}\mathcal{L}_{HIER} + w_{ital}\mathcal{L}_{ITAL} + w_{cl}\mathcal{L}_{CL},\] where \(\mathcal{L}_{GEN}\) supervises hierarchical label-text generation, \(\mathcal{L}_{HIER}\) structures the label embedding space according to the clinical taxonomy, \(\mathcal{L}_{ITAL}\) grounds slide-level representations to the taxonomy-structured label space, and \(\mathcal{L}_{CL}\) regularizes the visual embedding space. \(w_{gen}\), \(w_{hier}\), \(w_{ital}\), and \(w_{cl}\) denote the corresponding loss weights.

Let \(\big\{(I_i, \mathbf{y}^{(c)}_i, \mathbf{y}^{(f)}_i)\big\}_{i=1}^{B}\) be a mini-batch of \(B\) WSIs, where \(I_i\) is a WSI and \(y^{(c)}_i \in \mathcal{C}\) and \(y^{(f)}_i \in \mathcal{F}\) denote the corresponding coarse- and fine-level text labels, respectively. For each WSI \(I_i\), the MIL backbone (\(\mathcal{E}(\cdot)\) and \(\texttt{Agg}(\cdot)\)) and dual-branch multimodal conditioning module \(\mathcal{V}\) produce granularity-specific visual token embeddings \(\mathbf{v}_i^{(c)}\) and \(\mathbf{v}_i^{(f)}\) and coarse- and fine-level conditioning sequences \(\mathbf{E}_i^{(c)}\) and \(\mathbf{E}_i^{(f)}\). The dual-head text decoder \(\mathcal{T}\) uses \(\mathbf{E}_i^{(c)}\) and \(\mathbf{E}_i^{(f)}\) to generate coarse- and fine-level text labels \(\hat{\mathbf{y}}_i^{(c)}\) and \(\hat{\mathbf{y}}_i^{(f)}\), respectively.

Hierarchical Label-Text Generation Objective. For the label-text generation, we employ standard token-level cross-entropy for each granularity as \(\mathcal{L}_{GEN}=\mathcal{L}_{GEN}^{(c)}+\mathcal{L}_{GEN}^{(f)}\), which serves as the primary loss.

Label Hierarchical Loss. Given the projected coarse- and fine-level label-text embeddings \(\{\mathbf{t}_i^{(c)}\}_{i=1}^{|\mathcal{C}|}\) and \(\{\mathbf{t}_i^{(f)}\}_{i=1}^{|\mathcal{F}|}\), we enforce them to respect the predefined clinical taxonomy \(\mathcal{H}\). Formally, we define the label hierarchy loss (HIER), which consists of (1) a hierarchy alignment loss \(\mathcal{L}_{\text{HAL}}\) and (2) a sibling-margin loss \(\mathcal{L}_{\text{SML}}\): \(\mathcal{L}_{HIER}=\mathcal{L}_{\text{HAL}}+\mathcal{L}_{\text{SML}}\).

Hierarchy Alignment Loss. To enforce the clinical taxonomy in the label embedding space, the hierarchy alignment loss \(\mathcal{L}_{\text{HAL}}\) is defined as: \[\mathcal{L}_{\text{HAL}} = -\frac{1}{|\mathcal{C}|} \sum_{k=1}^{|\mathcal{C}|} \log \left( \frac{S_3(k)}{S_3(k)+S_1(k)+S_2(k)} \right),\] where \(S_1(k) = \sum_{k' \neq k} \exp\!\big( \mathbf{t}_k^{(c)}\cdot \mathbf{t}_{k'}^{(c)} \big)\), \(S_2(k) = \frac{1}{n_k(n_k-1)} \sum_{\substack{i,j \in \mathcal{I}_k\\ i \neq j}} \exp\!\big( \mathbf{t}_i^{(f)}\cdot \mathbf{t}_j^{(f)} \big)\), \(S_3(k) = \frac{1}{n_k} \sum_{i \in \mathcal{I}_k} \exp\!\big( \mathbf{t}_k^{(c)}\cdot \mathbf{t}_i^{(f)} \big)\), and \(\mathcal{I}_k = \{i \mid \mathcal{H}(i) = k\}\) defines the set of child fine-level indices for a coarse-level \(k\), with \(n_k = |\mathcal{I}_k|\).

This encourages that each coarse-level label embedding aligns with its descendant fine-level label embeddings (\(S_3\)) and remains separated from other coarse-level labels (\(S_1\)), while sibling fine-level label embeddings are separated from each other (\(S_2\)).

Sibling Margin Loss. In \(\mathcal{L}_{\text{HAL}}\), although \(S_2\) implicitly encourages the relative separation of sibling fine label embeddings, these embeddings frequently collapse into highly clustered regions of embedding space. This likely occurs when the coarse–fine alignment term (\(S_3\)) dominates \(\mathcal{L}_{\text{HAL}}\). To explicitly prevent sibling collapse, we formulate the sibling margin loss \(\mathcal{L}_{\text{SML}}\) based on a hard Euclidean margin: \[\mathcal{L}_{\text{SML}} = \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \psi\!\left( m^2 - \|\mathbf{t}_i^{(f)}-\mathbf{t}_j^{(f)}\|_2^2 \right),\] where \(m>0\) is a target margin, \(\psi(\cdot)\) is a smooth hinge implemented with softplus, and the sibling pair set is defined as \(\mathcal{P}=\cup_k \mathcal{P}_k\), with a parent-specific subset \(\mathcal{P}_k=\{(i,j)\mid i<j,\; \mathcal{H}(i)=\mathcal{H}(j)=k\}\).

Image–Text Alignment Loss. To ground the visual embeddings \(\mathbf{v}_i^{(c)},\mathbf{v}_i^{(f)}\) in taxonomy-consistent label space, we align these embeddings to their corresponding label-text embeddings. For a branch \(b \in \{c,f\}\), the image–text alignment loss (ITAL) is computed as: \[\mathcal{L}_{ITAL}^{(b)} = -\sum_{i=1}^{B} \log \frac{ \exp\!\left( \mathbf{v}_i^{(b)\top} \mathbf{t}^{(b)}_{\mathbf{y}_i^{(b)}} / \tau \right) }{ \sum_{j=1}^{N_b} \exp\!\left( \mathbf{v}_i^{(b)\top} \mathbf{t}_j^{(b)} / \tau \right) }\] where \(\tau\) is a temperature and \(N_b\) denotes total classes (\(N_c=|\mathcal{C}|\) and \(N_f=|\mathcal{F}|\)). The total ITAL is obtained by summing branch-specific objectives: \(\mathcal{L}_{ITAL} = \mathcal{L}_{ITAL}^{(c)} + \mathcal{L}_{ITAL}^{(f)}\). This anchors visual embeddings to their ground-truth label semantics, forcing them to inherit the label-space hierarchy.

Supervised Contrastive Loss. To further improve discriminability under weak slide-level supervision, we regularize the visual embeddings via supervised contrastive learning. For a branch \(b \in \{c,f\}\), the supervised contrastive objective (CL) is formulated as: \[\mathcal{L}_{CL}^{(b)} = - \sum_{i=1}^{B} \frac{1}{|\mathcal{G}^{(b)}(i)|} \sum_{g \in \mathcal{G}^{(b)}(i)} \log \frac{\exp\!\left(\mathbf{v}_i^{(b)\top} \mathbf{v}^{(b)}_g / \tau \right)}{\sum_{j \neq i} \exp\!\left(\mathbf{v}_i^{(b)\top} \mathbf{v}^{(b)}_j / \tau \right)}\] where \(\mathcal{G}^{(b)}(i)\) denotes indices of positives for an anchor \(i\). We compute \(\mathcal{L}_{CL}^{(b)}\) independently for both branches and sum them to obtain the final CL loss: \(\mathcal{L}_{CL} = \mathcal{L}_{CL}^{(c)} + \mathcal{L}_{CL}^{(f)}\).

Cyclic Loss Scheduling. The overall objective includes four distinct loss terms. We fix \(w_{gen}=1.0\) as generation is the primary task. Since \(\mathcal{L}_{HIER}\) and \(\mathcal{L}_{CL}\) enforce intra-modal structuring while \(\mathcal{L}_{ITAL}\) promotes cross-modal alignment, optimizing all three jointly empirically induces instability. To mitigate this, we employ an alternating cyclic loss schedule that separates representation structuring from cross-modal alignment. Let \(u=t/T \in [0,1]\) denote normalized training progress at epoch \(t\) out of \(T\) total epochs. We define a cyclic modulator \(c(u)=\cos^2(\pi n_{\text{cycles}}u)\) to dynamically adjust the corresponding weights: \(w_{hier}(u)=w_{hier}^{\max}c(u)\), \(w_{cl}(u)=w_{cl}^{\max}c(u)\), and \(w_{ital}(u)=w_{ital}^{\max}(1-c(u))\). In this manner, \(\mathcal{L}_{HIER}\) and \(\mathcal{L}_{CL}\) are emphasized when \(\mathcal{L}_{ITAL}\) is down-weighted, and vice versa.

4 Experiments and Results↩︎

4.1 Datasets and Label Hierarchies↩︎

We evaluate TaxoMIL on three WSI datasets, each structured according to a two-level taxonomy with coarse-level labels and fine-level subtypes or grades. GastWSI is a private dataset containing 7,228 gastric WSIs organized into 4 coarse-level and 23 fine-level categories. BRACS [4] and PANDA [5] are public breast and prostate datasets, respectively. BRACS includes 545 WSIs with 3 coarse-level and 7 fine-level labels. PANDA comprises 10,614 WSIs annotated with 6 fine-level labels (normal and five ISUP grades). The fine-level labels are grouped into three coarse-level risk strata to construct a two-level hierarchy. The details of the datasets and label hierarchies are summarized in Supp. 9 and Table 6.

4.2 Evaluation Protocol↩︎

For MIL-based baselines, predictions are evaluated using standard top-1 accuracy at both coarse and fine levels. For text generation methods, including TaxoMIL and VLM-based baselines, we adopt a strict exact match evaluation protocol to ensure fair comparison. A generated text is correct only if the entire token sequence exactly matches the ground-truth label string, after decoding and removing special tokens. Partial or prefix matches and semantically similar but non-identical outputs are strictly counted as incorrect to prevent diagnostic confusion in clinical settings.

4.3 Baselines↩︎

We compare TaxoMIL against a total of twelve baselines spanning MIL-based flat and hierarchical methods as well as VLM-based methods. All baselines were retrained from their official implementations under identical data splits, frozen UNI features, and evaluation protocols to ensure a fair comparison.

MIL-based Baselines. We employ five single-label MIL baselines (TransMIL[8], CLAM-SB[7], ABMIL[6], MambaMIL[10], and S4MIL[9]) and four hierarchical MIL baselines (HMIL[1], Chang et al.[12], ViLa-MIL[11], and HiClass[14]). ViLa-MIL is originally a dual-magnification single-label classifier. We adapt its dual-scale design to a dual-granularity setting by training one branch on coarse-level labels and the other on fine-level labels.

VLM-based Baselines. We include three VLM baselines (WSI-VQA[3], SlideChat[2], and CAMP[13]), which perform text-based reasoning over WSIs.

4.4 Implementation Details↩︎

For all three WSI datasets, we extract patch embeddings using the frozen UNI[15] foundation model as the feature extractor \(\mathcal{E}\). The shared autoregressive text decoder backbone \(\mathcal{T}_{shared}\) is initialized from the pretrained GPT-2 small (124M)[16] with 12 decoder layers. We use the standard GPT-2 tokenizer as \(\texttt{TOK}\) for all text processing. During training, we set the temperature \(\tau=0.07\) for both \(\mathcal{L}_{ITAL}\) and \(\mathcal{L}_{CL}\), and we apply a margin \(m=1.5\) for \(\mathcal{L}_{SML}\). For the cyclic loss scheduling, we set the number of cycles to \(n_{\text{cycles}}=3\) and define the maximum loss weights as \(w_{hier}^{\max}=0.3\), \(w_{ital}^{\max}=0.3\), and \(w_{cl}^{\max}=0.1\) in all experiments. Further implementation and training details are available in Supp. 10.

4.5 Main Results↩︎

3.5pt

3.5pt

We assess TaxoMIL against state-of-the-art MIL classifiers and generative VLMs across three evaluation scenarios: (1) Holistic evaluation (exact match required at both coarse and fine levels), (2) Coarse-level evaluation, and (3) Fine-level evaluation. We report accuracy (ACC), weighted F1 (W-F1), and Cohen’s kappa (\(\kappa\)) to capture overall correctness, robustness to class imbalance, and agreement beyond chance, respectively. The computational efficiency of TaxoMIL and competing baselines is provided in Table S5.

Holistic Evaluation. Table 1 reports holistic performance across all three datasets, serving as a strict benchmark for clinical usability, as it requires both coarse- and fine-level predictions to be correct. TaxoMIL achieved the best holistic performance on all three datasets by consistent margins in all evaluation metrics. Compared to the strongest per-dataset baseline, TaxoMIL yielded substantial gains by +1.29% ACC, +0.0139 W-F1, and +0.0336 \(\kappa\) on GastWSI (vs. CAMP), +7.26% ACC, +0.0638 W-F1, and +0.0815 \(\kappa\) on BRACS (vs. ABMIL), and +4.32% ACC, +0.0419 W-F1, and +0.0479 \(\kappa\) on PANDA (vs. WSI-VQA). Notably, none of the baselines demonstrated consistent performance across all three datasets. While CAMP, ABMIL, and WSI-VQA exhibited competitive results on a single dataset, their performance degraded substantially on the remaining two. Overall, these results indicate stronger end-to-end coarse-to-fine consistency of TaxoMIL.

Coarse-level Evaluation. As shown in Table 2, the overall performance of all models increased substantially at the coarse-level compared to the holistic evaluation, reflecting the reduced complexity of broader diagnostic categories. Nevertheless, TaxoMIL consistently outperformed both MIL and VLM baselines. Specifically, TaxoMIL provided performance gains of 1.58% to 5.12% ACC for GastWSI, 1.85% to 14.81% ACC for BRACS, and 0.93% to 3.56% ACC for PANDA, relative to baselines. Consistent improvements were also observed for W-F1 and \(\kappa\), further validating the reliability of TaxoMIL.

Fine-level Evaluation. At the fine-level, TaxoMIL maintained its superior performance (Table 2), demonstrating its ability to resolve the subtle morphological ambiguities. Relative to the coarse-level evaluation, TaxoMIL obtained even larger performance margins over baselines; for instance, 2.33% to 8.93% ACC for GastWSI, 7.41% to 25.93% ACC for BRACS, and 3.94% to 7.60% ACC for PANDA. These consistently larger improvements at the fine-level suggest that taxonomy-constrained text generation and the hierarchy-guided objectives particularly benefit fine-level discrimination, where inter-class boundaries are more ambiguous and precise hierarchy-aware reasoning is most critical.

4.6 Ablation Studies↩︎

We conducted systematic ablation studies to assess the impact of the proposed training strategies and decoder prefix design.

3.0pt

3.2pt

Effect of Training Strategies. Table 3 presents a progressive ablation study, incrementally adding each objective to a base model trained with the text generation loss (\(\mathcal{L}_{GEN}\)). Adding the supervised contrastive learning loss (\(\mathcal{L}_{CL}\)) yielded modest but consistent gains across datasets, likely due to improved discriminability among visually similar slides. Introducing the label hierarchical loss (\(\mathcal{L}_{HIER}\)) and image–text alignment loss (\(\mathcal{L}_{ITAL}\)) produced the largest improvements, concentrated at the fine-level and holistic evaluations. On BRACS, fine-level and holistic ACCs improved by approximately +10.00%, demonstrating the efficacy of hierarchy-guided objectives for fine-level discrimination. Notably, adding \(\mathcal{L}_{CL}\) alone on BRACS achieved stronger coarse-level performance but degraded fine-level results, suggesting that \(\mathcal{L}_{CL}\) without hierarchical constraints can lead to over-clustering of coarse categories at the expense of sibling discrimination. Finally, the inclusion of cyclic scheduling (full model) delivered the highest performance across all datasets. These results indicate that each component contributes complementary benefits for robust coarse-to-fine learning.

Effect of Decoder Design. Table 4 examines the impact of the decoder conditioning sequence by removing the learnable image token (\(\texttt{<|img|>}\)) or label-context token (\(\mathbf{e}_{\text{ctx}}\)). Removing either component consistently degraded performance across all granularities. For example, under holistic evaluation, the exclusion of \(\texttt{<|img|>}\) decreased ACC by 2.50%, 5.41%, and 1.50% on GastWSI, BRACS, and PANDA, respectively. Similarly, omitting \(\mathbf{e}_{\text{ctx}}\) yielded comparable drops of 2.04%, 5.41%, and 4.04% on the same datasets. These consistent drops highlight their roles in structural grounding. Specifically, \(\texttt{<|img|>}\) helps to disentangle visual conditioning from prompt tokens, while \(\mathbf{e}_{\text{ctx}}\) provides a global semantic prior. We further examined two architectural variants: replacing the text decoder with classification heads (TaxoMIL-Cls) and varying the decoder backbone (Supp. 11 and Table 7). The generative formulation consistently improves holistic performance over its classification counterpart, and the lightweight GPT-2 small backbone yields the best accuracy–efficiency trade-off, indicating that the gains stem primarily from taxonomy-aware representation learning rather than decoder capacity.

4.7 Qualitative Assessments↩︎

Label Embedding Geometry. To investigate the effectiveness of the proposed hierarchy-guided objectives, we visualize the label-text embeddings \(\{\mathbf{t}_k^{(c)}\} \cup \{\mathbf{t}_i^{(f)}\}\) by projecting them into a lower-dimensional manifold using multidimensional scaling (MDS). Figure 3a and b compare the resulting manifolds before and after taxonomy-constrained training. Prior to training, fine-level labels intermingle across coarse-level categories, and sibling fine-level labels frequently collapse, demonstrating the lack of explicit taxonomic structure in the raw pretrained embedding space. After training, we observe a markedly more taxonomy-aligned organization: (1) coarse-level labels gravitate toward the centroids of their child fine-level labels, serving as structural anchors for the hierarchy, (2) unrelated coarse-level categories become better separated, reducing high-level diagnostic confusion, and (3) sibling fine-level labels become more dispersed, facilitating fine-level discrimination. These observations closely match the design principles of the proposed hierarchy-guided objectives.

Image Embedding Centroid Geometry. To further examine the organization of the learned image representations, we apply the same MDS technique to project slide-level embeddings from the GastWSI dataset. Figure 3c visualizes the class centroids for both granularities, computed by averaging slide-level embeddings from the same class, and compares TaxoMIL against a representative MIL model (CLAM-SB) and a representative VLM model (CAMP). Relative to these baselines, TaxoMIL exhibits clearer separation among coarse-level categories while producing a highly structured fine-class layout within each coarse category, which is consistent with the label-text geometry. In contrast, CLAM-SB and CAMP show intermingled and overlapping class structures, indicating weaker hierarchical organization. These observations align with the improved coarse- and fine-level silhouette scores achieved by TaxoMIL (Figure 3c).

5 Limitations and Future Work↩︎

TaxoMIL assumes a fixed taxonomy defined by label-text embeddings and parent–child relations. Although it can be extended to deeper hierarchies via level-wise alignment and sibling-margin constraints, adapting to evolving taxonomies would require dynamic taxonomy construction, which we leave for future work. TaxoMIL also reduces but does not fully eliminate parent–child violations, as shown by the PCVR analysis in Supp. 11 and Table 8, and incurs additional parameter and memory overhead from the autoregressive decoder, as discussed in Supp. 12 and Table 9.

6 Conclusion↩︎

We present TaxoMIL, a taxonomy-constrained WSI-to-text label generation framework for hierarchical pathology diagnosis. Built on a standard MIL backbone for slide-level representations, TaxoMIL employs a dual-head autoregressive decoder to jointly generate coarse-level categories and corresponding fine-level subtypes within a unified generative pipeline. To enforce clinical structure, hierarchy-guided objectives organize the label embedding space according to the clinical taxonomy, align slide-level representations with taxonomy-consistent semantics, and improve parent–child coherence while reducing hierarchical contradictions. Across three WSI benchmarks, TaxoMIL yields more accurate and hierarchically consistent diagnostic predictions, outperforming state-of-the-art MIL classifiers and VLM methods. These results highlight the benefit of embedding clinical taxonomic structure directly into the learning objective, supporting interpretable and clinically meaningful diagnostic reasoning.

Acknowledgements↩︎

This work was supported by a grant of the National Research Foundation of Korea (NRF) (No. RS-2025-00558322 and RS-2024-00397293).

7 Algorithm↩︎

8 Prompt Templates↩︎

We use 10 base prompt templates, which are randomly selected during training. For multi-dataset training, the organ term is substituted by the dataset domain: gastric \(\rightarrow\) breast for BRACS and gastric \(\rightarrow\) prostate for PANDA. At inference time, we use the first template as the default prompt. The prompt templates are provided in Table 5.

9 Datasets↩︎

We evaluate TaxoMIL on three WSI benchmarks spanning distinct organs and hierarchical diagnostic protocols. Each dataset provides a two-level taxonomy with a coarse-level category and a fine-level subtype or grade. GastWSI is a private dataset containing 7,228 gastric WSIs with 4 coarse-level categories and 23 fine-level diagnoses. We adopt a \(7:1.5:1.5\) train/val/test split to ensure sufficient samples for rare fine-level diagnoses. BRACS [4] is a public breast pathology benchmark with 3 coarse-level categories and 7 fine-level labels across 545 WSIs with an \(8:1:1\) train/val/test split. PANDA [5] is a public prostate dataset that comprises 10,614 WSIs annotated with ISUP grading. ISUP grading is a modern, simplified prostate cancer classification scheme that is standard in digital pathology. We retain all 10,614 WSIs without excluding ambiguous cases, a comparatively challenging setting that contributes to the lower absolute performance observed across all models on PANDA. We adopt a \(8:1:1\) train/val/test split and construct a two-level hierarchy by grouping ISUP grades into three risk strata at the coarse-level and using the original ISUP grades as fine-level labels. The specific coarse- and fine-level labels are provided in Table 6.

10 Implementation Details↩︎

All WSIs are processed at \(20\times\) magnification. We use a patch size of \(512 \times 512\) pixels for GastWSI and BRACS, and \(256 \times 256\) pixels for PANDA. We extract patch embeddings using the frozen UNI[15] foundation model as the feature extractor \(\mathcal{E}\). The shared autoregressive text decoder backbone \(\mathcal{T}_{shared}\) is initialized from pretrained GPT-2 small (124M)[16], consisting of 12 decoder layers. We use the standard GPT-2 tokenizer as \(\texttt{TOK}\) for all text processing. For text generation, greedy decoding is implemented with maximum generation lengths of \(L^{(c)}=5\) and \(L^{(f)}=15\) tokens for coarse- and fine-level predictions, respectively. The patch- and slide-level embedding dimension is \(D = 1024\), and the decoder hidden state size is \(H = 768\).

The model is trained up to 100 epochs with early stopping based on validation performance. We use AdamW with weight decay \(0.01\) and a cosine annealing learning-rate scheduler. The learning rate is set to \(1\times10^{-5}\) for GastWSI and \(1\times10^{-4}\) for BRACS and PANDA, with a batch size of 64. During training, we set the temperature \(\tau=0.07\) for both \(\mathcal{L}_{ITAL}\) and \(\mathcal{L}_{CL}\), and we apply a margin \(m=1.5\) for \(\mathcal{L}_{SML}\). For the cyclic loss scheduling, we set the number of cycles to \(n_{\text{cycles}}=3\) and define the maximum loss weights as \(w_{hier}^{\max}=0.3\), \(w_{ital}^{\max}=0.3\), and \(w_{cl}^{\max}=0.1\) in all experiments. All experiments are conducted on an NVIDIA A6000 GPU.

11 Additional Ablation Analyses↩︎

3.5pt

Non-generative control. To isolate the contribution of taxonomy-constrained text generation, we compare TaxoMIL with TaxoMIL-Cls, a variant that replaces the autoregressive decoder with classification heads while keeping the same backbone and taxonomy-aware objectives. As shown in Table 7, the generative formulation consistently improves holistic performance across all three datasets, and TaxoMIL-Cls alone still surpasses most baselines, confirming that the core architecture and taxonomy-aware supervision are effective independently of the generation head.

Decoder choice. As our outputs are short, closed-taxonomy label strings, we adopt GPT-2 small (124M) as a lightweight decoder. We compare it against a domain-specific decoder (BioGPT[17], 347M) and a larger general decoder (GPT-2 medium, 355M) in Table 7. Both alternatives are inferior to GPT-2 small, with GPT-2 medium showing notably lower coarse-level performance on BRACS, likely due to the limited dataset size.

6pt

Hierarchical consistency. In this closed-taxonomy setting, all generation-based methods yield a 0% invalid-label rate under the exact-match protocol. To further assess consistency, we report the parent–child violation rate (PCVR) in Table 8, defined as the proportion of predictions whose fine-level label is inconsistent with the predicted coarse-level label. CAMP is excluded as its paired output structurally prevents violations. TaxoMIL maintains a competitive PCVR while achieving the highest holistic accuracy, indicating a favorable accuracy–consistency trade-off.

12 Computational Efficiency↩︎

6pt

References↩︎