1 Introduction↩︎

Deep learning has achieved radiologist-level accuracy on several medical imaging tasks [1], [2], yet clinical adoption remains limited. Two barriers stand out. First, black-box models provide no evidence for their predictions, making it impossible for a physician to verify or override the reasoning. Second, even “interpretable” models may silently propagate spatial errors when doctors provide corrective feedback, creating a patient-safety risk.

Prototype-based models [3], [4] address the first barrier by making predictions through similarity to learned visual examples. Recent work on Concept-based Similarity Reasoning (CSR) [5] extends this to multi-prototype per-concept reasoning with doctor-in-the-loop interaction. However, three structural limitations remain:

1. Independent concepts. The task head is a flat linear layer applied to concatenated concept similarity scores. It ignores known anatomical correlations between findings (e.g., active tuberculosis (TB) and cavity formation co-occur reliably), discarding clinically meaningful structure.

2. Unsafe spatial interaction. When a physician draws a bounding box to guide the model’s attention, the system applies the spatial feedback regardless of the model’s confidence in that region. If the model is confused and associates a different concept with the indicated region, the feedback amplifies an error.

3. Closed concept vocabulary. Adding a new clinical finding (e.g., a rare variant not seen in training) requires fully retraining the prototype atlas — a prohibitive cost in clinical settings.

We propose GRAPE, a unified architecture that resolves all three limitations within a coherent four-stage training framework. Our contributions are:

• A Graph Attention Task Head (Module A, §3.2) that constructs a data-driven concept co-occurrence graph from training labels and refines prototype similarity scores via two rounds of Graph Attention Network (GAT) message passing before classification.

• A Concept-Mismatch Safety Check (Module B, §3.3) — the first such mechanism in prototype-based medical classifiers — that computes per-concept mean similarity inside a physician-drawn region and warns when a different concept dominates, preventing silent error amplification before feedback is applied.

• Open-Vocabulary Prototype Anchoring (Module C, §3.4) that aligns visual prototypes to frozen BioViL-T text embeddings, enabling a new clinical concept to be added at test time from a single labeled image without modifying any other component.

• A four-stage training pipeline (§3.5) that integrates concept supervision, contrastive prototype learning, VLM alignment, and graph-structured classification.

• Experiments on TBX11K and NIH ChestX-ray14 (§4) covering classification, Pointing Game localization, inference speed, safety check validation, zero-shot concept addition, and Insertion/Deletion AUC faithfulness.

2 Related Work↩︎

ProtoPNet [3] learns class-specific visual prototypes. Extensions add deformability [4], decision trees [6], part-sharing [7], and patch variants [8]. In medical imaging, MProtoNet [9] and Ma et al. [10] apply prototype bottlenecks to chest X-rays. GRAPE builds on CSR [5], replacing its linear head with a graph-structured classifier. Concept Bottleneck Models [11]–[13] interpose concept scores before prediction; Margeloiu et al. [14] show they can be semantically misaligned, GRAPE grounds prototypes to BioViL-T [15] text anchors instead. GCN classifiers [16], [17] model corpus-derived label co-occurrence. GRAPE uses training-label statistics with prototype-similarity node features and learned GAT attention [18]. Bayesian uncertainty [19], [20] requires sampling overhead. GRAPE’s prototype-variance signal is deterministic [21]. Medical VLMs (MedCLIP [22], BiomedCLIP [23]) target classification. GRAPE uses text embeddings as geometric anchors for prototype initialization, preserving spatial interpretability. Interactive annotation methods [24], [25] lack concept-mismatch checks. Faithfulness is evaluated via Insertion/Deletion AUC [26], [27]. B-cos networks [28] offer an alternative faithful attribution approach by replacing standard convolutions with alignment-based transforms. Unlike GRAPE, they do not support interactive concept-level correction or open-vocabulary extension.

3 Method↩︎

3.1 Preliminaries: Prototype-Based Concept Reasoning↩︎

Backbone \(F\) followed by two-layer MLP projector \(P\) yields L2-normalized spatial embeddings: \[\mathbf{f}' = P(\mathbf{f}) \in \mathbb{R}^{D \times h \times w}, \quad \|\mathbf{f}'_{:,i,j}\|_2 = 1 \;\;\forall i,j.\]

Each concept \(k\) has \(M\) L2-normalized prototypes \(\{\mathbf{p}_{km}\} \subset \mathbb{R}^D\); the similarity map \(\mathcal{S}_{km}(i,j) = \langle \mathbf{p}_{km}, \mathbf{f}'_{:,i,j} \rangle\) (unit-norm cosine; Appendix 17) aggregates to a scalar score: \[s_{km} = \max_{i,j}\, \mathcal{S}_{km}(i,j). \label{eq:sim95score}\tag{1}\]

The baseline CSR model feeds the score matrix \(\mathbf{s} \in \mathbb{R}^{K \times M}\) through a linear task head to produce class logits. GRAPE replaces this with the three modules described below.

3.2 Module A: Graph Attention Task Head↩︎

Clinical concepts are not independent, active TB and cavity formation co-occur. Cardiomegaly correlates with pleural effusion, yet a flat linear head over concatenated \(s_{km}\) discards this structure. We build a data-driven co-occurrence graph from training labels \(\mathbf{y}^{(n)} \in \{0,1\}^K\): the \(K \times K\) conditional probability matrix

\[\mathbf{A}_{kk'} = P(k \mid k') = \frac{\sum_{n} y^{(n)}_k \cdot y^{(n)}_{k'}}{\sum_{n} y^{(n)}_{k'} + \epsilon},\]

is thresholded at \(\tau=0.1\) to obtain a directed binary adjacency with self-loops (the matrix is not symmetrized: \(\mathbf{A}_{kk'} \neq \mathbf{A}_{k'k}\) in general), then row-normalized so that each node’s incoming edge weights sum to one. The equality \(\mathbf{A}_{kk'} = P(k\mid k')\) follows from the frequentist identity \(P(k\mid k') = P(k,k')/P(k')\): since \(\mathbf{y}^{(n)}\in\{0,1\}^K\) is binary, the numerator \(\sum_n y_k^{(n)}y_{k'}^{(n)}\) counts samples where both concepts co-occur, and the denominator \(\sum_n y_{k'}^{(n)}\) counts samples where \(k'\) is present; the common \(1/N\) factor cancels, yielding the maximum-likelihood estimate of the conditional probability [16]. The constant \(\epsilon\) (implemented as denominator clamping to \(\min=1\), i.e.\(\epsilon=1\)) prevents division by zero for concepts absent from the training split and has negligible effect whenever \(\sum_n y_{k'}^{(n)}\gg 1\).

Graph Attention Network. Each concept is a node with initial feature \(\mathbf{h}_k^{(0)} = \mathbf{s}_{k,:} \in \mathbb{R}^M\). Two GAT layers [18] refine these via learned attention (Appendix 17, Eqs. 8 –9 ). A LayerNorm–Dropout–Linear readout produces class logits. Layer dimensions and head counts are described in Appendix 8 (Table ¿tbl:tab:hparams95fixed?).

3.3 Module B: Concept-Mismatch Safety Check↩︎

3.3.0.1 Prototype variance maps.

The disagreement among \(M\) prototypes for concept \(k\) at patch \((i,j)\) provides a spatial consistency estimate without sampling: \[\begin{align} \mu_k(i,j) &= \frac{1}{M} \sum_{m=1}^M \mathcal{S}_{km}(i,j), \\ U_k(i,j) &= \frac{1}{M} \sum_{m=1}^M \!\left(\mathcal{S}_{km}(i,j) - \mu_k(i,j)\right)^2. \label{eq:uncertainty} \end{align}\tag{2}\] High \(U_k(i,j)\) indicates that the \(M\) prototypes strongly disagree about whether patch \((i,j)\) belongs to concept \(k\), signaling ambiguous evidence. At training time this guides prototype masking (see below); at inference time the safety check uses the mean map \(\mu_k\) directly to detect concept mismatch.

3.3.0.2 Spatial safety check.

When a doctor provides a bounding box \(\mathcal{B}\) intending to reinforce concept \(k^*\), GRAPE checks whether the dominant concept inside the region matches. It first computes the mean similarity per concept inside \(\mathcal{B}\): \[\bar{s}_k(\mathcal{B}) = \frac{1}{|\mathcal{B}|} \sum_{(i,j) \in \mathcal{B}} \mu_k(i,j) \quad \forall k,\] identifies the dominant concept \(k^\dagger = \arg\max_k \bar{s}_k(\mathcal{B})\), and issues a safety warning if \(k^\dagger \neq k^*\) and \(\bar{s}_{k^\dagger}(\mathcal{B}) - \bar{s}_{k^*}(\mathcal{B}) > \eta\) (threshold \(\eta = 0.05\)). The feedback is not applied until the doctor confirms. This is a concept-mismatch check: it detects when the model’s dominant response inside the region conflicts with the claimed concept label, preventing silent error amplification in the prototype atlas. The check uses the spatial mean \(\bar{s}_k(\mathcal{B})\) rather than the spatial max of \(\mu_k\) inside \(\mathcal{B}\). Mean-based aggregation is more robust to isolated high-activation outliers than max-based gating (quantitative comparison: Appendix 22); \(U_k\) also guides prototype masking during training (Appendix 17).

3.4 Module C: Open-Vocabulary Prototype Anchoring↩︎

We eliminate the closed-vocabulary bottleneck by anchoring each prototype to a clinical text description via the frozen BioViL-T encoder [15]. A learnable projection \(\mathbf{W}_{\text{txt}} \in \mathbb{R}^{D \times 768}\) maps each concept’s text embedding \(\mathbf{t}_k\) into the visual prototype space: \[\hat{\mathbf{t}}_k = \frac{\mathbf{W}_{\text{txt}}\mathbf{t}_k}{\|\mathbf{W}_{\text{txt}}\mathbf{t}_k\|_2} \in \mathbb{R}^D.\] The alignment loss (Stage 3) pulls all \(M\) prototypes toward their text anchor while keeping the BioViL-T backbone frozen: \[\mathcal{L}_\text{align} = \frac{1}{KM} \sum_{k=1}^K \sum_{m=1}^M \left(1 - \langle \mathbf{p}_{km},\, \hat{\mathbf{t}}_k \rangle\right). \label{eq:align95loss}\tag{3}\] Zero-shot concept addition. At test time, a clinician adds a new concept \(k'\) without image annotations by initializing \(M\) perturbed prototypes from the text anchor (\(\sigma_{\text{init}}=0.1\)): \[\mathbf{p}_{k'm}^{(0)} = \frac{\hat{\mathbf{t}}_{k'} + \sigma_{\text{init}}\boldsymbol{\epsilon}_m}{\|\hat{\mathbf{t}}_{k'} + \sigma_{\text{init}}\boldsymbol{\epsilon}_m\|_2}, \quad \boldsymbol{\epsilon}_m \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_D),\; \boldsymbol{\epsilon}_m \leftarrow \boldsymbol{\epsilon}_m / \|\boldsymbol{\epsilon}_m\|_2.\] These prototypes are immediately usable for inference or adaptable with a small number of labeled examples via concept-local fine-tuning.

3.5 Training Pipeline↩︎

GRAPE trains in four stages (Algorithm 4 in Appendix 16): concept supervision, prototype learning, optional VLM alignment, and GNN task head fine-tuning. All stages use BF16 AMP on A100 hardware.

Stage 1 — Concept Supervision. The backbone \(F\) and per-concept CAM heads are trained with BCE on Global Average Pooled activations (Appendix 17, Eq. 6 ).

Stage 2 — Concept Vector Generation. CAM-weighted spatial averages of frozen backbone features produce per-concept vectors \(\mathbf{v}_k\) (Appendix 17, Eq. 7 ) that seed the Stage 3 prototype atlas.

Stage 3 — Prototype Learning with VLM Alignment. The projector \(P\), prototypes \(\{\mathbf{p}_{km}\}\), and text projection \(\mathbf{W}_{\text{txt}}\) are trained jointly. Each projected concept vector \(\mathbf{v}'_k = P(\mathbf{v}_k)\) serves as a query; the multi-prototype contrastive loss assigns it to its ground-truth concept \(\tilde{k}\) via soft assignment: \[\begin{align} q_m^k(\mathbf{v}') &= \text{softmax}_m\!\left(\gamma \langle \mathbf{p}_{km},\mathbf{v}' \rangle\right), \\ \text{sim}_k(\mathbf{v}') &= \sum_m q_m^k(\mathbf{v}') \langle \mathbf{p}_{km},\mathbf{v}' \rangle, \\ \mathcal{L}_\text{con} &= -\log \frac{\exp\!\left(\lambda(\text{sim}_{\tilde{k}}(\mathbf{v}'_{\tilde{k}}) + \delta)\right)}{\sum_{k} \exp\!\left(\lambda\, \text{sim}_k(\mathbf{v}'_{\tilde{k}})\right)}, \label{eq:contrastive} \end{align}\tag{4}\] with \(\lambda=10\), \(\gamma=5\), \(\delta=0.1\); the Stage 3 loss is \(\mathcal{L}_3 = \mathcal{L}_\text{con} + 0.1\,\mathcal{L}_\text{align}\).

Stage 4 — Graph-Structured Classification. With all prior components frozen, the GNN is trained with \(\mathcal{L}_4 = \text{CE}(\text{GNN}(\mathbf{s}, G),\, y)\) (E2E variant: Appendix 9).

4 Experiments↩︎

4.1 Datasets↩︎

TBX11K [2] is a 3-class TB chest X-ray dataset with 10,400 images: 4,600 healthy, 4,600 sick-but-non-TB, and 1,200 active-TB images, with 1,211 bounding box annotations for three TB finding types (Active Tuberculosis, Obsolete Pulmonary Tuberculosis, Pulmonary Tuberculosis). We use the official 6,600/1,800 train/val split and hold-out a dedicated bbox_eval split (200 active-TB images with at least one annotated box) for Pointing Game evaluation; the 1,800-image test split is used for final F1 reporting. NIH ChestX-ray14 [29] contains 100k+ frontal chest X-rays with 14 finding labels. Primary multi-seed comparison (Table 2) uses a 10k-patient patient-level stratified sub-sample (5 seeds, no patient leakage); single-seed ablations (Table 1) use a 20k-image shard (seed 42). Pointing Game uses 709 shard images with 964 bbox annotations from BBox_List_2017.²

Evaluation metrics. Macro F1 measures classification performance averaged across classes, accounting for class imbalance. Pointing Game (PG) [30] evaluates spatial interpretability: for each (image, concept) pair with a ground-truth bounding box, it checks whether the spatial maximum of the concept’s similarity map falls inside the box; we report the hit rate across all evaluated pairs. Full implementation details and hyperparameter settings are in Appendix 8 (Table ¿tbl:tab:hparams95fixed?).

4.2 Main Results↩︎

6pt

4.2.0.1 Key observations.

(1) GNN is the dominant module for multi-class tasks. The GNN head alone saturates the staged gain (\(+13.8\) pp F1 over 5 seeds, Table 2; \(+14.4\) pp on the single-seed ablation, Appendix 12); VLM alone yields no improvement on the sparse \(K=3\) concept set. E2E training reaches \(0.990 \pm 0.002\) F1 (Appendix 9) but at the cost of spatial faithfulness (PG \(0.091 \pm 0.059\) vs.staged \(0.111 \pm 0.038\)); a plain ResNet-50 linear head achieves \(0.993\) F1, confirming E2E gains are driven by unconstrained backbone fine-tuning rather than concept reasoning. The per-class gain is concentrated on majority classes (Healthy \(+21.5\) pp, Sick-non-TB \(+20.6\) pp; Appendix 15); Active TB F1 is comparable between GRAPE and CSR, suggesting the GNN reshapes the output distribution rather than improving disease-specific detection. (2) VLM alignment is complementary for diverse concept sets. On NIH, VLM contributes to the single-seed gain (Table 1), but multi-seed evaluation (Table 2) shows GRAPE and CSR are statistically comparable (\(p=0.894\)), indicating the gain does not persist under the rigorous patient-level split protocol. (3) Pointing Game and spatial faithfulness. GRAPE’s prototype maps achieve \(\mathbf{2.6\times}\) higher PG than CBM (\(0.140\) vs.\(0.055\)) and PIPNet (\(0.140\) vs.\(0.053\); Appendix 14): end-to-end fine-tuning sacrifices prototype-to-region alignment for classification accuracy. On the single-seed run (seed 42), GRAPE achieves PG \(=0.1401\) vs.CSR \(0.0628\) (\(+7.73\) pp; Figure 3); multi-seed evaluation finds no statistically significant difference vs.CSR (\(p=0.64\)), reflecting high variance (\(\sigma \approx 0.03\)) inherent to Pointing Game at \(7{\times}7\) resolution.

4.3 Graph Head Comparison and Inference Speed↩︎

We ablate the choice of graph task head by comparing GRAPE’s GAT with ML-GCN [16] and ADD-GCN [17]; all three heads outperform the linear baseline (+10.7–14.6 pp F1), confirming that concept co-occurrence structure is the key driver, not the specific message-passing mechanism. We retain the GAT head for its directly interpretable, learned edge-conditioned attention weights. Full results are in Appendix 12; a per-edge importance ablation quantifying each edge’s contribution to final F1 is in Appendix 13. Graph structure vs.added nonlinearity. To isolate whether the GAT gain comes from graph-relational reasoning or simply from additional parameters, we train a parameter-matched 2-layer MLP (\({\approx}47\)K parameters) directly on the flattened \(K{\times}M\) score vector, replacing graph structure with a generic nonlinear function of the same capacity. The MLP achieves F1 \(=0.865\), closing \(\sim\!69\%\) of the gap between the linear head (\(0.798\)) and the GAT (\(0.896\)); the remaining \(+3.1\) pp (\(0.865 \to 0.896\)) is attributable to the co-occurrence graph structure itself. On TBX11K (\(K=3\)) where self-loops dominate, this confirms that the graph contributes meaningfully beyond nonlinearity, while the majority of the gain reflects the benefit of a learnable nonlinear readout over the prototype score matrix. The edge ablation (Appendix 13) shows self-loops dominate on TBX11K (\(K=3\)); richer co-occurrence effects are expected on NIH (\(K=14\), 103 edges). The GNN head adds only \(+1\) ms over the linear head at batch size 1, satisfying real-time interaction requirements (full latency benchmarks: Appendix 12).

8pt

4.4 Comparison with CSR Baseline↩︎

Table 2 summarizes GRAPE vs.CSR [5] over five seeds on both datasets (official patient-level split for NIH). On TBX11K, GRAPE’s F1 ranges 0.798–0.916 across seeds (std = 0.042); the paired \(t\)-test yields \(t(4)=8.03\), \(p=0.0013\). Single-seed ablation values (Table ¿tbl:tab:tbx11k?) are included for comparability only.

6pt

4.5 Module B: Safety Check Evaluation↩︎

We simulate miscorrections on the TBX11K bbox_eval split (207 annotated triples): with probability \(p\), a bounding box is reassigned to a wrong concept; the safety check (the dominance rule following Eq. 6, \(\eta=0.05\)) issues a warning when the dominant concept inside the box differs from the claimed one. Bootstrap 95% CIs use \(B=1000\) resamples. With the \(U_k\)-gated variant (\(\theta_V = 2{\times}10^{-4}\)) as the default operating point, Module B catches 85.1% of misdrawn boxes with an FP rate of 16.4% [12.1%, 20.9%] (Table 12). The ungated baseline achieves higher TP (88–89%) at the cost of a higher FP rate (21.7% [16.4%, 27.1%]). Further tightening to \(\theta_V=10^{-4}\) drops FP to 12.6% but TP falls to 78.0%, below the 85% clinical threshold. The \(U_k\)-gated configuration is the recommended deployment setting. Table 17 in Appendix 21 shows that prototype variance substantially outperforms both MC-Dropout variants (TP 90.5% vs.%/23.4%) while requiring only a single deterministic forward pass. On NIH (\(K=14\)), TP remains high (\(0.921\pm0.012\)) but FP rises to \(0.731\pm0.011\) due to dense concept overlap; per-concept breakdown and discussion are in Appendix 23.

4.6 Further Analysis: Zero-Shot Concept Addition and Faithfulness↩︎

Module C: Zero-shot concept addition. We evaluate Module C via a leave-one-concept-out experiment on TBX11K (\(K=3\)): for each held-out concept, we reinitialize its \(M=100\) prototypes from the text anchor (Eq. 9) and fine-tune them with \(n \in \{0,1,5,20\}\) labeled images, measuring Pointing Game on the bbox_eval split. Full results are in Appendix 19 (Table 14). Text-only initialization yields non-zero localization for active_tb (PG \(=0.030\)), and a single labeled image reaches PG \(=0.068 \pm 0.045\), on average 41% of full supervision (68% CI: 14–69%). With five seeds the few-shot trend is essentially monotonic: 1-shot (PG \(=0.068\)) \(<\) 5-shot (PG \(=0.079\)) \(\approx\) 20-shot (PG \(=0.078\)); the earlier non-monotonicity observed with three seeds was a seed artefact. We additionally evaluate on NIH ChestX-ray14 (\(K=14\)): 3 of 4 target concepts have bbox coverage (Infiltration, Effusion, Atelectasis); Effusion’s 1-shot PG matches full supervision (PG \(=0.072\), 100%), and Infiltration’s 20-shot reaches 53% of full supervision. Full results are in Appendix 19.1 (Table 15). Similarity map faithfulness. Faithfulness is evaluated via Insertion and Deletion AUC [26] on the 200-image TBX11K test split (full results: Appendix 20, Table 16). GRAPE achieves higher Insertion AUC (\(+1.3\) pp), confirming prototype maps capture genuinely predictive regions. Higher Deletion AUC is attributed to the prototype atlas spreading evidence across \(M=100\) spatial prototypes per concept, each contributing weakly to the saliency signal. A contextual comparison with CheXNet, MedCLIP, DenseNet, CBM, and PIPNet is in Appendix 14 .

5 Discussion↩︎

The over-regularization effect of VLM alignment on sparse concept sets (\(K=3\)) is analyzed in Appendix 8. Figure 5 quantifies this: after Stage 3, all \(M=100\) prototypes cluster tightly near their text anchors (mean prototype–anchor cosine \(=0.995\pm0.008\) for TBX11K vs.\(0.992\pm0.007\) for NIH), and inter-prototype pairwise cosines are similarly high (\(0.990\) vs.\(0.984\)), confirming that the alignment loss collapses intra-concept prototype diversity in both settings. The slightly lower NIH inter-prototype cosine (-0.006) is consistent with 14 distinct text anchors providing more diverse initialization than 3. Safety implications of the concept-mismatch check. The concept-mismatch check (the dominance rule following Eq. 6) does not improve macro-F1 directly, its value is in preventing harmful interactions. In a deployment scenario where a doctor mistakenly draws a box over a region dominated by a different finding, the safety warning gives the system a chance to flag the inconsistency before the feedback is applied. This property cannot be easily quantified in standard benchmarks but is critical for clinical trust.

5.0.0.1 Limitations.

Pointing Game scores remain low in absolute terms (7–14%) because similarity maps at \(7 \times 7\) resolution must be compared to bounding boxes annotated in 224×224 space. Upsample interpolation recovers some precision, but finer spatial resolution (e.g., with a ViT backbone) would improve localization. Module B’s recommended \(U_k\)-gated configuration (§4.5) achieves 16.4% FP rate; the baseline (ungated) rate of 21.7% is the upper bound. Further reduction would require a learned threshold or a clinician user study to calibrate tolerance. Module C is additionally validated on NIH (\(K=14\), 3 concepts with bbox coverage; Appendix 19.1); full \(K=14\) leave-one-out and a clinician user study remain future work. Training variance. TBX11K F1 ranges \(0.798\)–\(0.916\) (std \(=0.042\), 5 seeds), reflecting Stage 4 GNN sensitivity on the \(n{=}6{,}600\) dataset; Table 2 reports multi-seed NIH results on the official patient-level split (\(p=0.894\)).

Broader Impact↩︎

Interpretable AI systems for medical imaging have the potential to meaningfully improve clinical practice. By making every prediction traceable to a specific visual concept and prototype, GRAPE enables clinicians to verify or override model reasoning rather than accepting opaque outputs. The concept-mismatch safety check reduces the risk that erroneous clinician feedback silently degrades the prototype atlas, a failure mode that could otherwise go undetected in real deployments. The open-vocabulary extension lowers barriers to adoption in resource-limited settings by allowing new clinical concepts to be introduced without full retraining. At the same time, prototype-based explanations that appear clinically plausible may not be faithful to the model’s actual decision boundary, and clinicians who over-rely on similarity maps risk anchoring diagnostic judgment to potentially misleading visualizations. This work does not evaluate performance across demographic subgroups; prototype atlases learned from imbalanced distributions may exhibit worse spatial alignment for underrepresented populations, and practitioners should audit subgroup performance before deployment.

5.0.0.2 Ethics statement.

This work uses two publicly released datasets: TBX11K [2], released under a research license, and NIH ChestX-ray14 [29], released by the U.S.National Institutes of Health (NIH) under a public-access license. No new human-subject data were collected for this research. All experiments are conducted retrospectively on de-identified images. The system described in this paper is a research prototype and is not deployed in any clinical setting.

6 Conclusion↩︎

We presented GRAPE, an interactive medical image diagnosis architecture combining concept-mismatch interaction safety (Module B), graph-structured concept reasoning (Module A), and open-vocabulary prototype anchoring (Module C) in a unified four-stage training framework. Module B, the first concept-mismatch safety check in prototype-based medical classifiers, catches 85% of simulated erroneous bounding-box annotations at 16.4% FP rate (\(U_k\)-gated, §4.5), preventing unsafe feedback amplification without adding measurable inference latency. Module A yields +13.8 pp macro-F1 (\(0.866 \pm 0.042\), 5 seeds, \(p=0.0013\)) over the prototype baseline; the gain is concentrated on majority classes (Appendix 15) and the Pointing Game advantage does not replicate consistently across seeds (\(p=0.64\), Table 2). Module C is the first prototype-based mechanism to add a new clinical concept from a single labelled image without retraining: on NIH ChestX-ray14, Effusion 1-shot exactly matches fully supervised Pointing Game accuracy (100% recovery with no model modification); on TBX11K, 1-shot recovers 41% of full supervision (§4.6, Appendix 19.1). Insertion/Deletion AUC faithfulness evaluation (§[sec:faithfulness]) confirms that prototype similarity maps capture genuinely predictive spatial regions. Code and training scripts will be released upon acceptance.

Appendix overview↩︎

The supplementary material is organized as follows.

7	Notation table	p.
8	Hyperparameter sensitivity (\(\tau\) and \(\eta\) sweeps)	p.
10	Dataset details	p.
11	Graph structure	p.
9	End-to-end training ablation	p.
15	Per-class F1 and confusion matrix	p.
12	Graph-head comparison: GAT vs.ML-GCN vs.ADD-GCN	p.
13	GAT edge-importance ablation	p.
14	Contextual comparison with published methods	p.
17	Mathematical details	p.
16	Staged training algorithm	p.
18	Per-concept Pointing Game results	p.
19	Zero-shot concept addition	p.
20	Faithfulness evaluation	p.
21	MC-Dropout uncertainty comparison	p.
22	Module B: mean vs.max spatial aggregation ablation	p.
23	Module B safety check on NIH ChestX-ray14 (\(K=14\))	p.
24	Prototype–anchor alignment and inter-prototype dispersion	p.
25	GAT vs.parameter-matched MLP ablation (N6)	p.

7 Notation Table↩︎

8pt

8 Hyperparameter Sensitivity↩︎

Implementation. Backbone: ResNet-50 pretrained on ImageNet at 224×224. Prototype dimension \(D=256\), \(M=100\) prototypes per concept, batch size 128, BF16 AMP on NVIDIA A100 80GB.

We report sensitivity to the two most impactful thresholds: the graph co-occurrence threshold \(\tau\) and the safety check threshold \(\eta\). All experiments use TBX11K unless noted. Fixed hyperparameters are listed in Table ¿tbl:tab:hparams95fixed?.

8.0.0.1 Graph threshold \(\tau\).

For each \(\tau \in \{0.01, 0.05, 0.10, 0.20, 0.30\}\) we rebuild the co-occurrence graph and retrain only Stage 4 (GNN head, 20 epochs, all other parameters frozen). Table 3 reports Macro F1 and Pointing Game on the TBX11K validation set.

14pt

Results are stable across the full sweep (F1 range: 0.912–0.918; PG range: 0.135–0.150), confirming that GRAPE is not sensitive to the exact value of \(\tau\). The default \(\tau=0.10\) achieves the highest PG.

8.0.0.2 Safety threshold \(\eta\).

Using the trained GRAPE model (no retraining), we sweep \(\eta \in \{0.01, 0.03, 0.05, 0.10, 0.20\}\) in the safety check experiment of §4.5 (miscorrection rate \(p=0.5\), \(T=5\) trials).

14pt

TP rate varies by less than 1 pp across the sweep, and FP rate is essentially constant (decreasing only at \(\eta=0.20\) where very small gaps are ignored). The default \(\eta=0.05\) is a conservative choice that maximises TP without inflating FP.

10pt

7pt

8.0.0.3 VLM alignment on sparse vs.dense concept sets.

TBX11K has only \(K=3\) sparse TB-type concepts with high class imbalance (1,200 active-TB vs.,600 healthy images across all splits; 800 vs.,800 in train+val). The alignment loss \(\mathcal{L}_\text{align}\) pulls all 300 prototypes toward three text anchors, which may over-constrain the prototype geometry when the visual diversity is limited and the task signal is dominated by the binary presence/absence of TB lesions. On NIH with \(K=14\) diverse findings, the text anchors provide meaningful semantic grounding that prevents prototype collapse and improves generalization.

9 End-to-End Training Ablation (N4)↩︎

To examine the role of the staged curriculum, we train GRAPE (GNN + Uncertainty, no VLM) with all parameters jointly from scratch for 70 epochs — the same total epoch budget as staged training (30 + 20 + 20). No staged objectives are used; the loss is \(\mathcal{L} = \mathcal{L}_\text{BCE} + \mathcal{L}_\text{proto} + \mathcal{L}_\text{cls}\) throughout, where \(\mathcal{L}_\text{proto}\) is the multi-prototype contrastive loss from Eq. 4 (see §3.5), and the backbone receives task-classification gradients from epoch 1. We run both strategies over three seeds \(\{42, 0, 1\}\) to compare variance.

12pt

Result. E2E achieves substantially higher and more stable F1 (\(0.990 \pm 0.002\)) compared with staged training (\(0.809 \pm 0.084\)), confirming that the ResNet-50 backbone is the bottleneck in staged training. However, the picture reverses for spatial interpretability: staged training achieves higher mean Pointing Game accuracy (\(0.111 \pm 0.038\) vs.\(0.091 \pm 0.059\)), and with lower variance. The seed-42 E2E run reported elsewhere (\(\text{PG}=0.174\)) was an outlier; the multi-seed mean is below staged.

12pt

Zero-shot comparison. Table 7 shows that E2E weights also support Module C’s zero-shot initialization: the text anchor produces non-zero localization at \(n=0\) (\(\text{PG}=0.062\), higher than staged \(0.030\)). However, E2E zero-shot PG improves only marginally as \(n\) increases (\(+0.022\) from \(n=0\) to \(n=20\)), while staged training shows a steeper gain from \(n=0\) to \(n=1\) (\(+0.049\)). We attribute this to prototype entanglement: in E2E training, backbone and prototypes co-evolve, making prototype geometry less amenable to targeted few-shot fine-tuning after training.

When to use each. Despite higher F1, we retain the staged curriculum as the primary configuration for three reasons:

1. Modular interpretability. Staged training decouples concept learning, prototype learning, and task classification; each stage can be inspected, retrained, or replaced independently.

2. Spatial faithfulness. Staged training achieves higher and more stable PG (0.111 vs.), directly reflecting more faithful prototype localisation.

3. Few-shot responsiveness. Staged prototypes respond more cleanly to few-shot fine-tuning for new concepts, a property required for Module C in clinical deployment.

We recommend E2E training for practitioners where classification accuracy is the primary goal and interpretability and zero-shot addition are not required.

10 Dataset Details↩︎

TBX11K concept descriptions (used for VLM alignment):

• ActiveTuberculosis: “Active pulmonary tuberculosis with cavitary lesions, nodular opacities, and upper-lobe infiltrates, often with signs of consolidation on chest X-ray.”

• ObsoletePulmonaryTuberculosis: “Healed or inactive tuberculosis with calcified granulomas, fibrous scarring, and volume loss in the upper lobes on chest X-ray.”

• PulmonaryTuberculosis: “Pulmonary tuberculosis presenting as patchy or confluent consolidation, tree-in-bud opacities, or miliary nodules on chest X-ray.”

Bounding box scaling. TBX11K COCO annotations store boxes in original image coordinates (variable resolution). We scale to the 224×224 input space as \(x_1' = \lfloor x_1 / W_\text{orig} \times 224 \rfloor\). NIH bounding box annotations are provided in 1024×1024 coordinates and scaled with the same formula.

11 Graph Structure↩︎

On TBX11K (\(K=3\)), the co-occurrence graph has 3 nodes and 4 edges (threshold \(\tau=0.1\)). The three TB-type concepts form a dense triangle since they frequently co-occur within the same image. On NIH (\(K=14\)), the graph has 14 nodes and 103 edges at threshold \(\tau=0.05\) (as confirmed in multi-seed training logs). Concepts like Infiltration–Consolidation and Effusion–Atelectasis form strong edges.

12 Graph Head Comparison: GAT vs ML-GCN vs ADD-GCN↩︎

GRAPE uses a GAT task head [18] for its learned, edge-conditioned attention. Table 8 ablates this choice by replacing the GAT with two established graph classification baselines: ML-GCN [16] (standard 2-layer GCN with symmetric normalisation) and ADD-GCN [17] (static co-occurrence graph combined with a per-sample dynamic cosine-similarity graph). All three heads receive the same input — prototype similarity scores \((B, K, M)\) from frozen Stage 1–3 weights — and are trained for 20 epochs (Stage 4 only).

10pt

All three graph heads substantially outperform the linear baseline (+10.7–14.6 pp F1), confirming that the concept co-occurrence structure is the key driver of GRAPE’s gain, not the specific message-passing mechanism. ML-GCN achieves marginally the highest F1 (0.8983 vs. for GAT), but the difference is within the seed variance range (std \(\approx 0.04\), 5-seed estimate) observed in Table 2 and is not statistically significant. ADD-GCN achieves the highest PG (0.1449) at the cost of slightly lower F1 (0.8589), suggesting its dynamic per-sample graph better preserves spatial prototype structure at the expense of discriminative power. We retain the GAT head as GRAPE’s default because its learned, edge-conditioned attention weights are directly interpretable — each edge weight quantifies how strongly one concept’s activation influences another’s classification — a property absent in standard GCN variants.

8pt

13 GAT Edge-Importance Ablation↩︎

The TBX11K co-occurrence graph (\(K=3\), threshold \(\tau=0.10\)) has four edges: three self-loops (one per concept) and one directed cross-concept edge \((0{\to}1)\) connecting ActiveTuberculosis to ObsoletePulmonaryTuberculosis. To measure the contribution of each edge to the task head, we retrain Stage 4 four times, each time removing a single edge from the graph while keeping Stages 1–3 frozen.

The self-loop on concept 0 (ActiveTuberculosis) is the most informative edge: removing it causes the largest drop of \(-6.49\) pp F1, indicating that the GAT head relies heavily on self-aggregation of the dominant positive class. The PulmonaryTB self-loop contributes moderately (\(-1.99\) pp), while the ObsoleteTB self-loop and the sole cross-concept edge \((0{\to}1)\) each contribute fewer than 0.5 pp, suggesting that inter-concept propagation plays a secondary role when self-referential concept features are already strong.

14 Contextual Comparison with Published Methods↩︎

On TBX11K, CBM and PIPNet achieve higher raw F1 than GRAPE’s staged configuration owing to end-to-end backbone fine-tuning (observation 4 in §4.2); GRAPE’s primary advantage is its spatial faithfulness (\(2.6\times\) PG improvement) and unique capabilities (GNN co-occurrence, safety check, zero-shot concept addition). On NIH, the multi-seed evaluation (Table 2) finds GRAPE and CSR statistically comparable (\(p=0.894\)); the primary claim rests on TBX11K (\(+13.8\) pp F1, \(p=0.0013\); single-seed PG \(+7.73\) pp, multi-seed PG \(p=0.64\)). Table ¿tbl:tab:sota? contextualizes GRAPE against published methods. Because each prior work uses a different train/test split, label set, or task formulation, direct numerical comparison is not always valid; the “Comparable?” column indicates the degree of comparability.

6pt

On TBX11K, CBM and PIPNet achieve higher raw F1 (\(0.9778\) and \(0.9644\)) than GRAPE’s staged configuration (\(0.8963\) best single-seed): both baselines train end-to-end without the prototype-preserving curriculum. Chen et al.’s DenseNet ensemble (AUC 0.901) uses a different metric; GRAPE’s F1 confirms competitiveness while providing prototype-level explanations and safe interaction. On NIH, CheXNet (mean AUC 0.841 over 14 findings) and MedCLIP use different splits and task formulations; they are listed for context. The primary quantitative claim rests on TBX11K (\(+14.4\) pp F1, \(+7.73\) pp PG, \(p=0.0013\)).

15 Per-Class F1 and Confusion Matrix (N2)↩︎

Table 11 reports per-class macro-F1 for GRAPE and CSR across five seeds on TBX11K, revealing the source of the macro-F1 gap. Table ¿tbl:tab:confusion? shows the confusion matrix for seed 42 (best GRAPE run).

10pt

GRAPE’s macro-F1 improvement is driven almost entirely by the two majority classes: Healthy (\(+0.215\) mean) and Sick-non-TB (\(+0.206\) mean). Active TB F1 is already high in both models (\(\approx 0.93\)) and barely changes. This pattern is consistent with the GNN head exploiting co-occurrence context — active TB strongly co-occurs with the other TB findings, providing discriminative signal that benefits Healthy/Sick separation even without direct Active-TB context. CSR’s high Active-TB score but low Healthy/Sick scores reflects the known class-imbalance pattern in TBX11K (1,200 active-TB vs.,600 healthy and sick images).

8pt

16 Staged Training Algorithm↩︎

10pt

17 Mathematical Details↩︎

This appendix collects standard or prior-work formulas that were moved from the main body to keep the presentation concise.

17.0.0.1 Cosine similarity map (§3.1).

Given L2-normalised patch features \(\mathbf{f}'_{:,i,j} \in \mathbb{R}^D\) and prototype \(\mathbf{p}_{km} \in \mathbb{R}^D\), the similarity map is: \[\mathcal{S}_{km}(i,j) = \langle \mathbf{p}_{km},\, \mathbf{f}'_{:,i,j} \rangle. \label{eq:simmap}\tag{5}\] Because both vectors are L2-normalised, this inner product is equivalent to cosine similarity.

17.0.0.2 Stage 1 concept supervision loss (§3.5).

The backbone and per-concept CAM head are trained with binary cross-entropy on Global Average Pooled activations: \[\mathcal{L}_1 = \frac{1}{K} \sum_{k=1}^K \text{BCE}\!\left(\text{GAP}(\text{CAM}_k(\mathbf{f})),\, y_k\right). \label{eq:l1}\tag{6}\] This is a standard multi-label BCE objective applied to per-concept binary labels.

17.0.0.3 Stage 2 CAM-weighted concept vector (§3.5).

For each training image, the local concept vector is the CAM-attention-weighted spatial average of backbone features: \[\mathbf{v}_k = \sum_{i,j} \text{softmax}_{ij}(\text{CAM}_k(i,j)) \cdot \mathbf{f}_{:,i,j}. \label{eq:camvec}\tag{7}\] This is the standard CAM aggregation formula; the resulting vectors seed the \(k\)-means initialisation for Stage 3 prototypes.

17.0.0.4 GAT message-passing equations (§3.2).

Standard Veličković et al.attention over the concept co-occurrence graph. For layer \(\ell\) and head \(h\), the unnormalised attention score, normalised weight, and updated node representation are: \[\begin{align} e_{kk'}^h &= \text{LeakyReLU}\!\left(\mathbf{a}_h^\top \left[\mathbf{W}_{\text{GAT}}^h\mathbf{h}_k^{(\ell)} \,\|\, \mathbf{W}_{\text{GAT}}^h\mathbf{h}_{k'}^{(\ell)}\right]\right), \quad k' \in \mathcal{N}(k), \tag{8} \\ \alpha_{kk'}^h &= \frac{\exp(e_{kk'}^h)}{\sum_{j \in \mathcal{N}(k)} \exp(e_{kj}^h)}, \\ \mathbf{h}_k^{(\ell+1)} &= \text{ELU}\!\left(\sum_{k' \in \mathcal{N}(k)} \alpha_{kk'}^h\, \mathbf{W}_{\text{GAT}}^h\mathbf{h}_{k'}^{(\ell)}\right). \tag{9} \end{align}\]

17.0.0.5 Prototype masking (§3.3).

At training time, uncertainty maps additionally enable prototype discarding: a prototype \(\mathbf{p}_{km}\) is masked from gradient updates if its per-image variance exceeds a threshold \(\theta_U = 0.05\), preventing high-variance prototypes from destabilizing contrastive training.

18 Per-Concept Pointing Game Table↩︎

10pt

19 Zero-Shot Concept Addition Results↩︎

We perform a leave-one-concept-out experiment on TBX11K (\(K=3\)): for each held-out concept \(k'\), we load the fully trained GRAPE model, reinitialize only concept \(k'\)’s \(M=100\) prototype vectors using the text-anchor initialization (Eq. 9, \(\sigma_\text{init}=0.1\)), and fine-tune those prototypes alone with \(n \in \{0, 1, 5, 20\}\) labeled images while keeping all other parameters frozen.

10pt

For obsolete_tb, the \(n=0\) result (PG \(=0.093\)) reflects random prototype initialization: without VLM alignment the text anchor falls back to a random unit vector (§3.4), so this value does not represent semantic grounding and should not be compared against the few-shot rows. With labeled examples, performance is inconsistent: 1-shot reaches PG \(=0.047 \pm 0.036\) but drops sharply at \(n=5\) (\(0.005 \pm 0.009\)), recovering to \(0.023 \pm 0.029\) at \(n=20\). Full supervision for obsolete_tb is itself very low (PG \(=0.023\), 43 pairs), so the concept is intrinsically hard to localize; the high variance across seeds reflects this difficulty.

19.1 NIH ChestX-ray14 Partial Leave-One-Out↩︎

We extend the Module C evaluation to NIH ChestX-ray14 (\(K=14\)), targeting the four most annotated concepts by bounding-box count: Infiltration (113 pairs), Effusion (138 pairs), Atelectasis (166 pairs), and Consolidation. Consolidation has zero bbox-eval pairs after patient-level splitting and is excluded. For each of the three evaluable concepts, we load the fully trained NIH GRAPE model (seed 0), reinitialize that concept’s \(M=100\) prototypes from the BioViL-T text anchor (Eq. 9), and fine-tune with \(n \in \{0,1,5,20\}\) labeled images over 3 seeds.

8pt

The NIH results corroborate and extend the TBX11K finding. Effusion achieves full-supervision PG at \(n=1\) (\(0.072\), 100% of the \(0.072\) upper bound), suggesting the BioViL-T text anchor for pleural effusion aligns closely with the visual prototypes learned under full supervision. Infiltration reaches 53% of full supervision at \(n=20\) (\(0.094\) vs.\(0.177\)); Atelectasis at \(n=20\) (\(0.050\)) marginally exceeds its fully supervised PG (\(0.048\), 166 pairs). Trends at low \(n\) remain noisy: all three concepts show a dip at \(n=1\) or \(n=5\) before recovering at \(n=20\), consistent with the prototype-overfitting effect documented on TBX11K. The high variance at \(n=5\) for Infiltration (\({\pm}0.071\)) reflects sensitivity to the specific images selected by each seed.

20 Faithfulness Evaluation↩︎

We measure faithfulness using Insertion and Deletion AUC [26] over the 200-image TBX11K test split, using \(\text{sal}(i,j) = \max_{k,m} \mathbf{S}_{km}(i,j)\) as the saliency signal. Deletion AUC measures how quickly predicted class probability drops as salient pixels are replaced by channel mean (lower = more faithful); Insertion AUC measures how quickly it rises as salient pixels are revealed from a Gaussian-blurred baseline (higher = more faithful).

12pt

GRAPE’s higher Deletion AUC (\(+3.1\) pp) indicates that removing the maximally salient patches does not fully disrupt prediction, attributed to the prototype atlas distributing evidence across \(M=100\) spatial prototypes per concept: each prototype contributes weakly to the saliency aggregate \(\max_{k,m}\mathbf{S}_{km}(i,j)\), so the saliency map is more diffuse than a single-activation method. Both models maintain high confidence throughout the deletion curve (AUC \(>0.92\)), reflecting ResNet-50’s global texture encoding; the Insertion curve provides the more informative signal in this regime [26].

21 MC-Dropout Uncertainty Comparison (N5)↩︎

A natural alternative to prototype variance as the uncertainty signal for Module B is MC-Dropout [19]: run \(T=30\) stochastic forward passes with dropout enabled at inference, then use either the variance of similarity scores across passes (Variant A) or the predictive entropy (Variant B) to trigger warnings. Table 17 reports TP and FP rates for all three methods, each evaluated at the operating point on its threshold sweep closest to the prototype-variance FP rate (\(\approx 21.7\%\)).

8pt

Prototype variance substantially outperforms both MC-Dropout variants: Variant A achieves TP \(=51.0\%\) at FP \(=25.1\%\) with a highly threshold-sensitive signal, while Variant B achieves only 23.4% TP because global entropy is weakly correlated with local spatial misdraw. GRAPE’s prototype-level disagreement reaches 90.5% TP at the same FP level using a single deterministic forward pass rather than \(T=30\) stochastic passes.

22 Module B: Mean vs.Max Spatial Aggregation Ablation↩︎

The safety check (§3.3) computes \(\bar{s}_k(\mathcal{B})\) as the spatial mean of \(\mu_k\) inside the doctor-drawn box, rather than the spatial maximum. To validate this choice, we evaluate an alternative max-based gate: replace \(\bar{s}_k(\mathcal{B}) = \frac{1}{|\mathcal{B}|}\sum_{(i,j)\in\mathcal{B}}\mu_k(i,j)\) with \(\bar{s}_k^{\max}(\mathcal{B}) = \max_{(i,j)\in\mathcal{B}}\mu_k(i,j)\) in the dominant-concept comparison, keeping all other parameters (\(\eta=0.05\), \(T=5\) trials) identical.

The two variants produce nearly identical results. Mean aggregation yields marginally higher TP (\(0.895\) vs.\(0.891\)) at a negligible FP cost (\(0.217\) vs.\(0.213\)), confirming the design choice: averaging over the annotated region is more robust to isolated high-activation patches that would trigger spurious max-based alarms, while matching the max-based gate on true-positive detection.

23 Module B Safety Check on NIH ChestX-ray14 (\(K=14\))↩︎

We repeat the simulated miscorrection study on the NIH bbox split (888 triples, 709 images, 6 concepts with annotations). With \(K=14\) overlapping concepts, the safety check achieves TP \(=0.921\pm0.012\) — comparable to the TBX11K rate — but the FP rate rises to \(0.731\pm0.011\) (vs.% on TBX11K). The elevated FP rate reflects the dense concept landscape: when a correct box is drawn for any single finding, one of the other 13 active concepts often achieves higher mean similarity inside the region (e.g., Pneumonia FP \(=0.90\), Cardiomegaly FP \(=0.85\)). High sensitivity is preserved in the multi-label setting, while specificity degrades as concept count and co-occurrence density increase. A learned or per-concept threshold would be needed to deploy Module B in \(K{>}3\) settings.

24 Prototype–Anchor Alignment and Inter-Prototype Dispersion↩︎

25 GAT vs.Parameter-Matched MLP Head Ablation (N6)↩︎

10pt

Interpretation. The MLP closes \({\sim}69\%\) of the Linear\(\to\)GAT F1 gap (\(+6.7\) pp out of \(+9.8\) pp total), confirming that added nonlinearity accounts for most of the gain on this small \(K=3\) label set. The remaining \(+3.1\) pp (MLP\(\to\)GAT) is attributable to the co-occurrence graph structure itself. PG is identical between Linear and GAT, and slightly higher for the MLP, indicating that graph structure does not systematically harm localisation.

References↩︎

---

1. Corresponding author. Email: hkurban@hbku.edu.qa↩︎

2. BBox_List_2017 has 880 annotated images; 709 overlap with our shard.↩︎