1 Introduction↩︎

Neoadjuvant chemotherapy (NAC) is a standard treatment for patients with locally advanced or high-risk breast cancer, and pathological complete response (pCR) after NAC is a clinically important endpoint associated with favorable prognosis in several breast cancer subtypes [1], [2]. However, pCR is determined only after surgery by pathological assessment. Therefore, accurate prediction of pCR from multimodal data may support individualized NAC treatment [3].

Recent work on NAC response prediction has evolved from imaging-only models to multimodal frameworks that combine MRI with clinical and pathological information [4], [5]. MRI-based studies demonstrated the feasibility of imaging-driven prediction, while multicenter studies also revealed sensitivity to scanner and protocol heterogeneity, highlighting the risk of center-specific bias and limited cross-center generalization [6], [7]. For pre-treatment pCR prediction, prior work has shown that integrating baseline MRI with clinical variables improves performance compared with unimodal settings [6]. Subsequent studies further suggested that adding clinical and pathological information can improve predictive accuracy and reinforce the value of multimodal baseline integration [8], [9]. These results support multimodal modeling as a promising direction for pCR prediction. While some recent methods consider longitudinal or multi-time data for pCR prediction [2], [10], [11], we focus on the pre-treatment setting because it is most relevant for early treatment escalation for likely non-responders and de-escalation strategies for patients with high response probability.

However, several limitations remain for robust multicenter pre-treatment pCR prediction. First, many methods still rely on simple fusion operators (e.g., direct concatenation), which may not fully exploit clinically structured cross-modal dependencies. Related limitations have also been noted in recent multimodal studies [2], [11]. Second, explicit domain-invariant learning to mitigate multicenter MRI heterogeneity is not consistently incorporated [7]. Third, most pCR models output only a scalar pCR probability without case-based evidence retrieval, which limits evidence-grounded interpretation in clinical workflows, motivating retrieval-augmented generation (RAG) inference strategies [12].

To address these limitations, we propose ClinRAG-GRAPH, a clinically informed retrieval-augmented graph framework for pre-treatment pCR prediction from multimodal data, including DCE-MRI, structured clinical variables, and biopsy-derived pathological biomarkers. The main contributions of our work are as follows. First, we encode clinically motivated prior knowledge into the graph construction and learn multimodal features with a relation-aware attention graph convolutional network (GCN). Second, we design adversarial learning strategy to mitigate multicenter MRI protocol heterogeneity and improve task-relevant representations. We further incorporate an LLM-based subgraph RAG module to support evidence-grounded and interpretable pCR inference. Third, we conduct a multicenter evaluation on a large-scale NAC breast cancer cohort comprising two public datasets and three in-house datasets, demonstrating competitive and consistent performance against five representative baselines.

2 Method↩︎

We proposed ClinRAG-GRAPH for multicenter breast pCR prediction by integrating heterogeneous multimodal data into a unified intra-patient graph. For each patient \(i\), we denote the MRI volume as \(X_i\), the clinical/pathological attributes as \(t_i\), the binary pCR label as \(y_i\in\{0,1\}\), and the acquisition center as \(c_i\in\{1,\ldots,C\}\). Our goal is to learn a predictor that is discriminative for \(y_i\) while being robust to domain shifts induced by \(c_i\).

As shown in Fig. 1, ClinRAG-GRAPH first encodes the MRI volume using a DenseNet-121 encoder and constructs a clinical-prior graph \(G_i\) with edge-wise prior strengths \(p_{uv}\). It then applies a prior-guided relational attentional R-GCN to obtain a patient embedding and predict pCR, while enforcing center-invariant imaging representations via a gradient-reversal domain classifier. We further employ an LLM-driven case-based subgraph RAG module to retrieve clinically analogous historical cases based on learned embeddings and attention signatures, and then fuse the neighbor-derived estimate with the R-GCN prediction for a more evidence-grounded and robust inference.

2.1 Hierarchical Semantic Clinical-prior Graph Construction↩︎

In multi-modal pCR prediction, purely data-driven graph learning faces two practical challenges: (i) homogeneous connectivity (e.g., fully connecting image-table/table-table pairs) blurs heterogeneous multi-modal medical semantics, and (ii) enforcing priors via auxiliary losses constraints can destabilize optimization and introduce extra hyperparameters.

We therefore construct a hierarchical semantic clinical-prior graph at the patient level. For each patient \(i\), we define an intra-patient directed graph \(G_i=(V,E)\) with one imaging node and multiple clinical-\(/\)pathological- variable nodes. Each directed edge \((u,v)\in E\) is associated with a relation type \(r(u,v)\) and a scalar prior weight \(p_{uv}>0\) that explicitly encodes the confidence of medical knowledge about this interaction. Guided by clinical practice guidelines and radiologist consensus [13]–[16], we organize edges into three semantic tiers:

• Strong prior. Guideline-level, stable associations (e.g., strong couplings among key molecular biomarkers), encoded with a fixed large \(p_{uv}=2\).

• Soft prior. Clinically agreed relations with inter-patient variability, encoded with a relatively large initialization of \(p_{uv}=0.8\).

• Learnable prior. Uncertain relations, weakly initialized (small \(p_{uv}=0.2\)) and primarily determined by data (e.g., patient age with breast caner subtypes).

To inject priors without auxiliary loss terms, we incorporate \(p_{uv}\) as a log-bias on attention logits only at the first graph layer (Sec. 2.2), providing early calibration toward clinically plausible message passing while keeping deeper layers fully learnable. We fix the temperature \(t=0.2\) in all experiments.

2.2 PRAttn-RGCN: Prior-guided Relational Attentional R-GCN↩︎

Follow R-GCN [17], given \(G_i=(V,E)\), we perform relation-aware attentive message passing to fuse the imaging node with heterogeneous clinical/pathological variable nodes. Each directed edge \((u\!\to\!v)\) carries a semantic relation type \(r(u,v)\), and nodes belong to different modalities. The prior strength \(p_{uv}\) is injected only at the first layer to guide early-stage cross-modal information flow.
Relation-aware attention. Let \(\mathbf{h}^{(l)}_v\in\mathbb{R}^d\) be the node representation at layer \(l\), and \(\mathbf{t}_{\tau(v)}\) be the embedding of node type \(\tau(v)\). For each edge \((u\!\to\!v)\), we compute a relation-specific attention logit \[e^{(l)}_{uv}= \mathbf{a}_{r(u,v)} ( [\mathbf{h}^{(l)}_{u};\mathbf{h}^{(l)}_{v};\mathbf{t}_{\tau(u)};\mathbf{t}_{\tau(v)}] ), \label{eq:rel95attn95logit}\tag{1}\] where \(\mathbf{a}_{r}(\cdot)\) is a lightweight relation-specific scorer. We inject the prior log-bias only when \(l=0\): \[\tilde{e}^{(l)}_{uv}=e^{(l)}_{uv}+\mathbb{I}[l=0]\cdot \tau \log(p_{uv}), \qquad \alpha^{(l)}_{uv}=\operatorname{softmax}_{u\in\mathcal{N}(v)}(\tilde{e}^{(l)}_{uv}). \label{eq:rel95attn95weight}\tag{2}\]
Relational message passing. Messages are transformed per relation and aggregated with attention: \[\mathbf{h}^{(l+1)}_{v}= \phi\!( \mathbf{W}_{0}\mathbf{h}^{(l)}_{v}+ \sum_{u\in\mathcal{N}(v)} \alpha^{(l)}_{uv}\,\mathbf{W}_{r(u,v)}\mathbf{h}^{(l)}_{u} ), \label{eq:rel95rgcn95update}\tag{3}\] where \(\mathbf{W}_{0}\) means self-loop transform and \(\mathbf{W}_{r}\) is relation-specific (optionally using basis decomposition with large \(|\mathcal{R}|\)). \(\phi(\cdot)\) denotes LayerNorm\(\&\)GELU.
Graph readout. After \(L\) layers, node embeddings are pooled to obtain a patient representation \(\mathbf{g}_i=\mathrm{MeanPool}(\{\mathbf{h}^{(L)}_v\}_{v\in V})\), which is fed to an Multilayer Perceptron (MLP) to predict pCR. The first-layer attention \(\{\alpha^{(0)}_{uv}\}\) is retained as the attention signature for downstream case retrieval.

2.3 Adversarial Learning for Decoupling MRI Domain Noise↩︎

Multicenter DCE-MRI inevitably contains scanner/protocol-specific artifacts that can dominate image embeddings and harm cross-center generalization. To suppress such domain cues while preserving pCR-discriminative information, we impose a domain-adversarial constraint on the imaging representation. Let \(\mathbf{h}_{\text{img}}^{(0)} = f_{\text{img}}(X_i) \in \mathbb{R}^d\) denotes the embedding used to initialize the imaging node. We attach an adversarial branch consisting of a lightweight projector \(f_{\mathrm{adv}}(\cdot)\) and a domain classifier \(D(\cdot)\), trained through a gradient reversal layer (GRL) [18]: \[\mathbf{z}_{\text{adv}} = f_{\text{adv}}(\mathbf{h}_{\text{img}}^{(0)}), \qquad \hat{\mathbf{c}} = D(\operatorname{GRL}(\mathbf{z}_{\text{adv}})), \label{eq:adv95branch}\tag{4}\] where \(\hat{\mathbf{c}}\in\mathbb{R}^{C}\) denotes the center logits and GRL multiplies the back-propagated gradient by a negative constant, enabling end-to-end optimization. Then, the training objective combines pCR supervision and domain confusion is: \[\mathcal{L}_{\mathrm{total}} =\mathcal{L}_{\mathrm{pCR}}(\hat{y}_i, y_i) +\lambda_{\mathrm{adv}} \mathcal{L}_{\mathrm{adv}}(\hat{\mathbf{c}}, c_i),\] where \(\mathcal{L}_{\mathrm{pCR}}\) is BCE loss, \(\mathcal{L}_{\mathrm{adv}}\) is CE loss with \(\lambda_{\mathrm{adv}}\) set to 0.05 through empirical tuning.

2.4 LLM-driven Subgraph RAG for Evidence-grounded Inference↩︎

To improve the robustness under distribution shift, we augment the trained PRAttn-RGCN with an LLM (DeepSeek-V3.2 [19]) retriever, design an RAG module that grounds inference on similar patient evidence. After training, we infer the training set and export intermediate results as an evidence buffer. For each patient \(i\), we store (i) a graph-level embedding \(\mathbf{g}_i\), (ii) an MRI embedding \(\mathbf{z}_i\), (iii) a clinical-pathological embedding \(\mathbf{m}_i\), and (iv) edge-level attention weights \(\boldsymbol{\alpha}_i\) from the frozen PRAttn-RGCN. We then build FAISS [20] inner-product indices over \(\ell_2\)-normalized embeddings for approximate nearest-neighbor search.

At inference, a query patient \(q\) is encoded as \((\mathbf{g}_q,\mathbf{z}_q,\mathbf{m}_q,\boldsymbol{\alpha}_q)\). DeepSeek outputs a schema-constrained retrieval plan from query-side signals (e.g., candidate pool sizes, filters (reduce false neighbors), fusion weights, and top-\(K\)), without making predictions. Candidates are retrieved via multi-route search and ranked by a fused score combining representation similarity and a subgraph match induced by \(\boldsymbol{\alpha}\). The top-\(K\) neighbors are deterministically aggregated and combined with the PRAttn-RGCN output to yield the final prediction, enabling evidence-grounded inference while preserving backbone calibration. Optionally, DeepSeek generates a structured post-hoc explanation based solely on retrieval metadata, fused scores, and driver-edge cues, exported as a verifiable JSON record.

3 Experiments↩︎

Datasets and Implementation Details. We evaluated ClinRAG-GRAPH on two public datasets and three in-house datasets, with dataset statistics summarized in Table ¿tbl:dataset?. Specifically, data from three source centers (DUKU [21], ISPY1 [22], and Zcenter) were used for model development and split at the patient level in a 7:1:2 ratio for training, validation, and internal testing, respectively, while Qcenter and Ycenter were held out for external generalization evaluation. All models were implemented in PyTorch and trained on an NVIDIA A40 GPU (48 GB memory). For preprocessing, multi-phase DCE-MRI scans were cropped to tumor-centered regions and concatenated along the temporal dimension to form an input volume of size \((192, 256, 256)\), with intensities processed using z-score normalization. The network was optimized using AdamW (learning rate \(2\times10^{-5}\), weight decay \(1\times10^{-4}\), batch size 12) for 50 epochs.

Evaluation Metrics. The quantitative evaluation was performed using standard classification metrics, i.e., AUC (95% CI) to assess discrimination, balanced accuracy (bAcc) to account for class imbalance, sensitivity (SEN) to evaluate positive-case detection, and specificity (SPE) to assess negative-case recognition.

6pt max width=0.7

0.4pt max width=, center

0.5pt width=, center

Internal Comparison. We evaluate ClinRAG-GRAPH against state-of-the-art methods including CLSTM [7], M2Fusion [11], LMF [2], iMRhpc [10]. As shown in Table ¿tbl:tab:main95results95compact?, on the internal test set (aggregated from three internal test sets), ClinRAG-GRAPH achieves the best AUC of 0.815 (95% CI: 0.738–0.885), outperforming R-GCN (0.706) and other fusion strategies such as LMF (0.678) and iMRhpc (0.699). It also improves balanced accuracy (0.691) and specificity (0.882) while maintaining sensitivity (0.500), indicating a more favorable trade-off. Paired DeLong tests confirm significant AUC gains (\(p<0.05\)).

External evaluation. ClinRAG-GRAPH remains robust under cross-domain shift. On Ycenter, it attains the highest specificity (0.946) while preserving the best AUC, reducing false positives in unseen domains. On the more challenging Qcenter cohort, it yields the best balanced accuracy (0.676) and markedly higher sensitivity (0.455) than all baselines, reflecting improved detection without sacrificing overall discrimination. The PLS-DA projection in Fig. [fig:right] further shows that model effectively distinguishes pCR patients across internal and external cohorts. Overall, ClinRAG-GRAPH consistently enhances both in-domain performance and multicenter generalization for pre-treatment pCR prediction.

Ablation Study. Table ¿tbl:tab:ablation95compact? shows that each proposed component contributes to performance. Introducing the clinical-prior graph improves over the R-GCN backbone, highlighting the value of explicit clinical-pathological relational priors for multimodal fusion. PRAttn-RGCN and adversarial decoupling provide further gains, suggesting the benefits of prior-aware message passing and more domain-invariant MRI representations. In addition, LLM-driven subgraph RAG further improves performance on external test sets, indicating enhanced robustness under distribution shift. Modality ablation (Fig. 2) further shows that pathology contributes most to pCR prediction.

Qualitative analysis. Fig. 3 presents subtype-specific SHAP analysis of directed edges in ClinRAG-GRAPH, showing that the model captures clinically meaningful cross-modal dependencies rather than relying on isolated features. The influential edges vary across Luminal A/B, HER2-pos, and TripleNeg subgroups, suggesting subtype-aware message passing. Consistent contributions from edges linked to ER, PR, HER2, Ki67, age, and MRI features support the effectiveness of our proposed hierarchical clinical-prior graph. The directional importance patterns further indicate that the model benefits from structured relational reasoning instead of simple feature concatenation.

LLM-driven RAG case study. As illustrated in Fig. 4, the Deepseek serves as both a retrieval planner and a post-hoc explainer. The planner generates a schema-constrained retrieval policy that selects clinically matched neighbors for the pCR=0 query case. The retrieved samples show high fused similarity and consistent clinical profiles. After fusion, the top-\(3\) neighbors provide coherent supporting evidence with aligned driver-edge patterns. The LLM explainer then summarizes this evidence into a structured, verifiable rationale. The final prediction agrees with the retrieved evidence and remains stable for this case.

4 Conclusion↩︎

In this study, we explore pre-treatment pCR prediction under multimodal heterogeneity and multicenter domain shift. We propose ClinRAG-GRAPH, a clinically informed framework that integrates DCE-MRI, clinical variables, and pathological biomarkers through hierarchical graph modeling, adversarial learning, and RAG inference. Extensive experiments demonstrate improved predictive performance, cross-center generalization, and clinically interpretability. Future work will collect longitudinal data to further improve pCR prediction.

References↩︎