1 Introduction↩︎

Hepatocellular carcinoma (HCC) is a leading cause of cancer-related mortality [1], with prognosis governed by two interacting yet biologically distinct axes: hepatic functional reserve and tumor-specific oncologic behavior [2]. Patients with similar survival can have different underlying drivers. In some patients, poor outcomes are driven by severe hepatic dysfunction, whereas in others they are attributable to tumor biology. Disentangling these pathways is critical for risk attribution and personalized treatment planning, yet most existing data-driven prognostic models compress both axes into a single entangled representation. Recent vision-language models (VLMs) improve radiology representation learning through image–report alignment, and finer-grained semantic grounding at the sentence, entity, and concept level has been explored [3]–[6]. However, these approaches organize representations by semantic granularity rather than clinically grounded biological pathways, implicitly mixing hepatic and tumor signals, limiting prognostic modeling performance and biological interpretability in heterogeneous cohorts such as HCC. In clinical practice, radiology reports routinely describe hepatic functional reserve and tumor-related characteristics. Yet standard representation learning does not explicitly disentangle these factors. We therefore introduce pathway-specific structure as a domain-informed inductive bias to encourage biologically aligned representations. Parameter-efficient adaptation via LoRA [7] provides a mechanism for lightweight specialization, but a single shared adapter applies the same transformation regardless of biological heterogeneity. Recent LoRA–MoE approaches, including MoLoRA [8], LoRAMoE [9], and MixLoRA [10], learn expert routing and specialization implicitly from downstream task objectives, without incorporating prior structural differentiation among experts. Such designs are effective for multi-task adaptation but are less suited to settings where meaningful domain structure exists. In HCC prognostic modeling, biological axes are present in multimodal pretraining data yet are not explicitly encoded in survival labels.

We present BioFact-MoE to address this gap by enforcing expert specialization through biologically defined pathway inputs during unsupervised contrastive pretraining, establishing domain-informed inductive bias before survival supervision is introduced. Contributions: (1) Biologically factorized pretraining. Large language model (LLM)-guided report decomposition paired with anatomical patch masking trains pathway-specific LoRA adapters that independently encode hepatic and tumor representations. (2) Improved prognostic performance. BioFact-MoE achieves 12-, 18-, and 24-month AUCs of 75.33%, 75.85%, and 73.96%, outperforming image-only, multimodal, vision–language pretraining, and MoE baselines. (3) Residual MoE survival architecture. A gated residual MoE head dynamically routes patient-specific risk contributions across hepatic and tumor pathways, enabling phenotype-aware risk stratification. Exploratory analysis shows the liver-dominant low-risk subgroup benefits from transarterial chemoembolization (TACE), consistent with clinical expectations. (4) Biological validation. Hepatic embeddings align with Platelet-Albumin-Bilirubin (PALBI), while tumor embeddings align with bilobar disease and immunoscore (all p<0.05), without supervision, confirming biological specificity.

2 Method↩︎

2.1 Stage 1: Pathway-Specific Biological Factorized Pretraining↩︎

BioFact-MoE framework operates in two stages (Fig. 1). In the first stage, biologically factorized pathway-specific adapters are pretrained using a contrastive vision–language objective on MRI–report pairs. We build on BiomedCLIP [11] (ViT-B/16 image encoder, PubMedBERT text encoder), with all backbone weights frozen. For volumetric MRI, we adopt a 2.5D strategy: each axial slice is encoded independently, a lightweight 1-layer self-attention transformer with learnable positional embeddings aggregate inter-slice context, and a two-layer MLP attention module produces a weighted sum of slice embeddings as the volume-level representation.

LLM-Guided Report Decomposition Each radiology report \(t\) is parsed offline by Qwen [12] into three segments: \(t \to (s_{\text{liver}},\, s_{\text{tumor}},\, s_{\text{neutral}})\), where \(s_{\text{liver}}\) captures hepatic parenchymal quality, portal hypertension signs, and splenic findings; \(s_{\text{tumor}}\) captures tumor morphology, vascularity, and aggressiveness; and \(s_{\text{neutral}}\) retains remaining context. During pretraining, the neutral segment is concatenated to both \(s_{\text{liver}}\) and \(s_{\text{tumor}}\), forming pathway-specific textual inputs that preserve shared clinical context while maintaining biological specialization.

Anatomical Patch Masking Organ masks from TotalSegmentator [13] weight patch tokens by organ occupancy \(w_i = \text{(organ pixels in patch)} \,/\, \text{(patch area)}\), so the pathway embedding \(v = \sum_i w_i t_i / \sum_i w_i\) is restricted to pathway-relevant anatomy. The liver pathway uses liver and spleen masks; the tumor pathway uses liver, portal vein, and inferior vena cava (IVC). Pathway-Specific LoRA Three independent LoRA parameter sets \((A_p, B_p)\), \(p \in \{\text{general, liver, tumor}\}\) (rank \(r{=}4\), \(\alpha{=}8\)) are trained separately via symmetric InfoNCE loss [14] \(\mathcal{L} = \frac{1}{2}(\mathcal{L}_{\text{img}\to\text{txt}} + \mathcal{L}_{\text{txt}\to\text{img}})\) at \(\tau{=}0.1\), with AdamW at a learning rate of \(1\times 10^{-4}\) for 250 epochs with a batch size 32.

2.2 Stage 2: Residual MoE Survival Modeling↩︎

All pathway encoder checkpoints (including LoRA parameters) are frozen. The general encoder produces \(z_{\text{base}} \in R^{1024}\). Pathway checkpoints yield \(z_{\text{liver}}, z_{\text{tumor}} \in R^{1024}\). Two expert MLPs map pathway embeddings to 256-dim representations \(e_{\text{liver}}, e_{\text{tumor}}\). A core component of the MoE framework is gating function \(G\) for dynamically routing patient-specific contributions across experts. In our case, \(G: R^{3072} \to R^2\) is a learned linear projection.

The gate input \(g_i=[z_{\text{base}};\; \delta z_i;\; |\delta z_i|]\) concatenates the global patient context \(z_{\text{base}}\), the signed inter-pathway discrepancy \(\delta z_i=z_{\text{liver}} - z_{\text{tumor}}\), and its absolute magnitude \(|\delta z_i|\), allowing the gate to jointly reason about the patient’s overall profile and the degree of disagreement between hepatic and tumor representations. The resulting soft weights \(w_i = [w_{liver}^{i}, w_{tumor}^{i}] = \text{softmax}(G(g_i))\) (summing to 1) reflect the relative dominance of each pathway for patient \(i\): a large \(w_{liver}^{i}\) indicates a liver-dominant phenotype, while a large \(w_{tumor}^{i}\) indicates a tumor-driven one. These weights are later used for phenotype-aware patient stratification, enabling clinically interpretable subgroup discovery beyond a scalar risk score.

A per-patient coefficient \(\alpha_i = \sigma(\text{MLP}_\alpha([z_{\text{base}}; e_{\text{liver}}; e_{\text{tumor}}]))\) modulates expert contribution, with the final risk score: \[r_i = \underbrace{h_{\text{base}}(z_{\text{base}})}_{\text{base risk}} + \;\alpha_i\!\left( w_{i1}\,\delta r_i^{\text{liver}} + w_{i2}\,\delta r_i^{\text{tumor}} \right).\]

The MoE survival head is optimized using the Cox partial likelihood with AdamW (learning rate \(1\times 10^{-4}\)) for 50 epochs with a batch size of 16.

3 Experiments↩︎

3.1 Datasets and Data Preprocessing↩︎

We curated two cohorts from local hospitals. This retrospective study was IRB approved. Pretraining: 4,582 abdominal MRI–report pairs from 1,896 patients, used for vision–language pretraining without survival supervision. Downstream: 588 HCC patients with baseline multiparametric MRI-report pairs and longitudinal follow-up for overall survival, randomly split into 400/188 for training/test. Subsets have structured biomarkers: PALBI (\(n{=}118\)), ALBI (\(n{=}275\)), Immunoscore (\(n{=}62\)), and bilobar disease (\(n{=}386\)), none used for supervision.

MRIs were resampled to isotropic resolution, normalized, and cropped to 224×224×60. TotalSegmentator [13], [15] organ masks were only used during pretraining. Reports were de-identified, retaining Findings and Impression sections.

3.2 Baselines and Metrics↩︎

We compare four categories of baselines: image-only survival modeling, multimodal fusion, state-of-the-art biomedical VLMs, and MoE architectures. Image-only and multimodal fusion baselines are trained solely on the downstream survival cohort, while VLMs and MoE models first leverage large-scale image–report pretraining. Performance is evaluated using time-dependent AUC at 12, 18, and 24 months based on three independent runs with different random seeds.

Image-only survival A pretrained 2D ResNet processes axial MRI slices independently. Slice-level features are averaged to obtain a patient-level representation, which is optimized under the Cox proportional hazards objective [16].

Multimodal survival Image and Report Fusion follows standard late-fusion multimodal modeling [17]. A pretrained 2D image encoder processes individual axial MRI slices, which are averaged to obtain a patient-level embedding. A pretrained text encoder extracts a global report embedding. Both embeddings are concatenated and fed into a Cox regression head for survival prediction [16].

Vision-language fine-tuning. BiomedCLIP [11] is fine-tuned end-to-end for survival by appending a Cox regression head to the joint image–report embedding. GLoRIA [18] extends global contrastive learning with local alignment, contrasting image sub-regions with individual report words; a Cox head is applied to patch- and global-level features. Finally, we implement an entity-aligned variant that exploits organ segmentation masks to explicitly supervise alignment between report-extracted anatomical entities and their corresponding segmented regions, providing a more robust grounding signal.

MoE SparseMoE implements a standard sparsely-gated MoE layer [19]. Given the frozen BiomedCLIP fused embedding, a learned gating network produces routing logits over \(E{=}4\) experts. We select only the two highest-scoring experts and combine them using the normalized gate weights as output.

3.3 Survival Performance↩︎

4pt

BioFact-MoE outperforms all baselines at 12, 18, and 24 months (Table 1). Based on predicted risk scalar, we can train a decision tree model on the training set to achieve high-risk and low-risk stratification on the test set. In addition, gate weights provide patient-level biological interpretation. Patients are classified as liver-driven (\(w_{liver}\) \(\geq 0.5\)) or tumor-driven based on the dominant expert pathway. Despite comparable clinical stage, TACE-treated patients in the test set demonstrate marked heterogeneity in survival outcomes. As shown in Fig. 2, while all patients receiving TACE benefit with log-rank \(p{=}\)​0.024, the liver-dominant phenotype subgroup show improved survival under TACE with log-rank \(p{=}\)​0.009, while the tumor-dominant phenotype subgroup show no difference in terms of survival with log-rank \(p{=}\)​0.834.

Exploratory treatment analysis is shown in Fig. 3. Hepatic low-risk patients show stronger survival association with TACE, consistent with clinical expectations. In tumor-dominant patients, no significant treatment interaction was detected (all \(p > 0.3\)); note subgroup sizes in the tumor-dominant TACE and non-TACE arms were limited (all \(n \leq 12\) per arm).

6pt

3.4 Biological Validation of Pathway Specialization↩︎

Table 2 provides evidence that the learned pathway embeddings capture biologically meaningful signals aligned with clinical biomarkers. Linear probes (5-fold cross-validation) indicate PALBI, a marker of hepatic functional reserve, is more accurately predicted from frozen liver-pathway embeddings than tumor-pathway embeddings (\(\Delta = 9.0\), \(p < 0.05\)), confirming that the liver expert has specialized toward hepatic biology. Conversely, both tumor-related variables, immunoscore and bilobar disease extent, are better captured by the tumor-pathway embeddings (\(\Delta = 4.9\) and \(4.1\) respectively, \(p < 0.05\)), validating that the tumor expert has acquired complementary tumor-specific representations. This cross-validated biological alignment provides evidence that BioFact-MoE achieves genuine factorization rather than redundant parallel encoding.

3.5 Ablation Studies↩︎

Table 3 evaluates the role of biological grounding from both text and images across training and inference. Random Text Split and Swapped Conditioning disrupt biologically meaningful report assignment, while No Text Segmentation removes textual pathway supervision entirely. No Anatomical Masking eliminates organ-aware visual grounding. Full Report at Inference keeps pathway-specific text segmentation during downstream training but substitutes the full unsegmented report at inference. Full Report uses the unsegmented report for both downstream training and inference. The inductive bias introduced by biological grounding from both text and images is vital for expert specialization in BioFact-MoE.

6pt

Table 4 isolates the contribution of each pathway component. Base Only removes both pathway-specific experts and retains only the base survival head, establishing the performance floor of general image-report fusion without biological factorization. Liver Only and Tumor Only retain a single expert pathway and remove the other, quantifying the individual prognostic value of hepatic and tumor-specific representations. Joint Learning (No MoE) trains liver, tumor, and base pathways jointly with equal contribution and no mixture routing, testing whether adaptive expert weighting via MoE is necessary beyond simple multi-pathway fusion.

We observe that Tumor & Base slightly outperforms Liver & Base, possibly because the shared pathway already captures substantial liver-related global signals, while tumor-specific features benefit more from dedicated specialization. Notably, Liver & Tumor (No Base) achieves performance comparable to the full BioFact-MoE at 18- and 24-month horizons, with only modest differences at 12 months. This suggests that biologically specialized pathways capture most of the prognostic signal, while the base pathway mainly provides additional robustness rather than driving performance.

Overall, these results indicate that biological factorization is the primary contributor, and the residual MoE design serves to enhance stability rather than act as the main source of performance gain.

8pt

4 Conclusion↩︎

We presented BioFact-MoE, a biologically factorized MoE framework for vision-language prognostic modeling in hepatocellular carcinoma. By explicitly decomposing radiology reports into liver- and tumor-specific pathways and aligning pathway-specific LoRA adapters with anatomically masked visual representations, BioFact-MoE learns biologically structured multimodal embeddings without requiring biomarker supervision. A residual MoE survival head dynamically routes patient-specific risk contributions across both pathways, improving survival prediction while enabling phenotype-aware stratification. Biological validation confirms that pathway embeddings align with clinically recognized hepatic and tumor markers, supporting our factorization as an emergent property of the pretraining design.

Several limitations limited further investigation. First, the LLM-guided report decomposition relies on the capability of the LLM; errors in pathway assignment may introduce noise into the biological factorization signal. Second, structured clinical biomarkers used for biological validation are sparsely available, limiting the statistical power of pathway specialization analysis.

References↩︎