1 Introduction↩︎

Foundation Models (FMs) have demonstrated remarkable cross-domain generalization by training large-scale architectures that integrate multiple domain-specific sub-models [1]. In medical imaging, building such FMs requires pooling numerous datasets from diverse institutions. However, strict privacy regulations such as GDPR [2] and HIPAA [3] prohibit the centralized aggregation of sensitive clinical data, creating a fundamental tension between model capability and data governance. Federated Foundation Models (FFMs) [4] resolve this tension by enabling collaborative training across institutions without exposing raw data, making them a natural paradigm for privacy-preserving medical AI.

Recent FFMs for medical imaging predominantly fine-tune pretrained natural-image models, including ViT [5], CLIP [6], and SAM [7], on medical downstream tasks. Liu et al. [8] fine-tuned SAM for federated medical image segmentation; Wu et al. [9] adapted CLIP for federated classification. Despite their convenience, these methods inherit a critical limitation: empirical evidence [10] shows that natural-image representations transfer poorly to the medical domain, where subtle inter-class variations, acquisition noise, and precise anatomical context demand domain-native knowledge that pretrained features cannot provide.

An alternative line of work trains FFMs from scratch on curated medical data [11], [12], acquiring richer domain-specific representations, echoing evidence from other modalities that federating the from-scratch training of a foundation model can match centralized counterparts while neutralizing per-domain bias [13]. Yet these efforts target single imaging modalities and single tasks. Real-world clinical AI demands a unified framework that simultaneously handles classification, caption-supervised representation learning, and visual question answering across heterogeneous modalities, under both Overlapped configurations (clients share modalities but with non-IID label distributions) and Non-overlapped configurations (each client holds a completely disjoint modality). Existing federated frameworks address at most a subset of these requirements. The compounding of classical data heterogeneity (non-IID labels) with Imaging Modality Heterogeneity, where feature distributions, noise characteristics, and semantic granularity differ fundamentally across clients, makes standard personalized FL methods, designed for label skew alone, insufficient for this setting.

This paper proposes FM\(^2\) (Federated Medical Multimodal foundation model), a framework that trains from scratch under Imaging Modality Heterogeneity and extends them to multimodal medical understanding. The core of FM\(^2\) is a dual Mixture-of-Experts (MoE) architecture: a Class-wise MoE retains personalized per-category knowledge locally, while a Domain-wise MoE captures shared cross-modality representations that are globally aggregated. A Heterogeneous Modality Alignment regularizer constrains the domain-wise experts toward global consensus, for which we prove a convergence rate of \({\mathcal{O}}(1/\sqrt{T})\) and an explicit generalization bound. Beyond classification, FM\(^2\) supports two multimodal extensions: (i) Caption-Enhanced Learning (CEL), where GPT-4o-generated captions (retained locally for privacy) provide federated vision-language contrastive supervision, serving as a shared semantic bridge that enriches representations even across clients with disjoint visual modalities; and (ii) Federated Medical VQA, demonstrating FM\(^2\)’s extensibility to natural-language querying of medical images across institutions. Our contributions are summarized as follows:

• We formalize Imaging Modality Heterogeneity under both Overlapped and Non-overlapped modality regimes, and propose FM\(^2\), the first unified multi-task federated framework addressing both settings simultaneously.

• We design dual MoE modules with a Heterogeneous Modality Alignment regularizer targeting modality-specific expert drift, and prove \({\mathcal{O}}(1/\sqrt{T})\) convergence and an explicit generalization bound (Theorem 1–Proposition 2).

• We introduce Caption-Enhanced Learning (CEL), where locally retained GPT-4o captions provide a shared textual bridge: language descriptions share a common vocabulary across modalities, enabling representation transfer even when clients hold entirely disjoint visual domains.

• We construct the MIMH benchmark (five modalities, four configurations) for federated Classification and Caption Enhanced Learning, and additionally evaluate on real-world medical VQA datasets for Federated Medical VQA, confirming consistent superiority across all three tasks with strong out-of-modality generalization.

2 Related Work↩︎

2.0.0.1 Heterogeneous Federated Learning.

FL trains models collaboratively across decentralized clients without exposing raw data [14]. Statistical heterogeneity (non-IID distributions) is a central challenge: naive aggregation causes client drift and degraded convergence. Personalized FL (PFL) addresses this through several families of techniques. Regularization-based methods [15], [16] add proximal penalties to keep local models close to the global reference while permitting per-client drift; Ditto [16] exemplifies this by solving a personalized objective with a global proximal term, and FFTS [17] extends the idea to foundation models by regularizing both client and server side to align knowledge across statistically heterogeneous datasets. Partial-sharing methods split the network into a globally aggregated backbone and a local task-specific head: FedBN [18] keeps batch normalization layers local to absorb feature shift, FedRep [19] retains only the classifier locally, and prompt- or adapter-based approaches [20], [21] share only lightweight parameters while keeping the backbone on-device. Adaptive aggregation [22] weights client contributions by similarity, and meta-learning methods such as Per-FedAvg [23] learn a shared initialization that enables rapid per-client fine-tuning. Despite their diversity, all of these approaches assume a homogeneous feature space and optimize for a single task, making them unsuitable when clients differ fundamentally in imaging modality. Mixture-of-experts approaches have also been explored in FL [24], [25], yet none disentangle class-level from modality-level heterogeneity simultaneously; centralized missing-modality strategies [26] are further inapplicable since FL prohibits raw cross-site feature aggregation.

2.0.0.2 FL for Medical Images.

Federated learning has been applied to privacy-preserving medical image classification [27]–[30] and, more recently, to vision-language tasks including medical VQA [31], [32]. For classification, methods such as FedFMS [8] and FedTCA [33] are typically task-specific and backbone-dependent, requiring re-engineering for each setting and assuming a single shared imaging modality. For vision-language understanding, FedDAT [34], PromptFL [35], and F\(^3\)OCUS [36] adapt pretrained VLMs to federated settings via dual adapters, prompt tuning, and client-layer meta-learning, but remain confined to single-modality client scenarios. No existing method jointly supports classification and VQA under a protocol that handles cross-modality structure. FM\(^2\) fills this gap by disentangling class-level personalization from modality-level consensus via dual MoE modules, enabling unified multi-task training across heterogeneous imaging modalities without task-specific re-engineering.

3 Methodology↩︎

3.1 Overview↩︎

Fig. 1 illustrates the FM\(^2\) framework. Each client first trains a local visual encoder \({\mathcal{F}}(\cdot) = \{{\mathcal{F}}_b, {\mathcal{F}}_h\}\) on its own data. The backbone \({\mathcal{F}}_b\) then serves as a frozen Observer \({\mathcal{V}}\) that performs Top-\(K\) Sampling to extract representative subsets \({\mathcal{D}}_1, \ldots, {\mathcal{D}}_N\) per class, forming a sampled dataset \({\mathcal{D}}_s = \bigcup_{i=1}^N {\mathcal{D}}_i\). Two MoE modules, namely Class-wise MoE \({\mathcal{M}}_c\) and Domain-wise MoE \({\mathcal{M}}_d\), are trained on \({\mathcal{D}}_s\) following \({\mathcal{V}}\). After local updates, \({\mathcal{V}}\) and \({\mathcal{M}}_d\) are uploaded for global aggregation; \({\mathcal{M}}_c\) remains local for personalization. \({\mathcal{V}}\) is frozen during stage-2 MoE training but uploaded at round end so the server can aggregate and redistribute a refined backbone for the next round’s stage-1 initialization. For multimodal extension, an optional text encoder \({\mathcal{T}}\) processes expert-generated captions, and a VQA head \({\mathcal{H}}_{\text{vqa}}\) enables question answering, both sharing the same MoE-enhanced visual backbone.

3.2 Dual Mixture-of-Experts for Heterogeneity↩︎

3.2.0.1 Observer and Top-\(K\) Sampling.

In the first training stage, each client trains its local model \({\mathcal{F}}\) with labeled images. The trained backbone, designated as the Observer \({\mathcal{V}}= {\mathcal{F}}_b\), computes a representativeness score \(S = \texttt{Sigmoid}({\mathcal{V}}({\boldsymbol{X}}_z))\) for each class \(z\). Top-\(K\) Sampling selects the \(K\) highest-scoring samples per class, forming subsets \({\mathcal{D}}_1, \ldots, {\mathcal{D}}_N\) (sensitivity analysis of \(K\) in Appendix B). This observer-guided selection addresses local label drift by ensuring that only the most informative samples enter the second stage.

3.2.0.2 Class-wise MoE (CMoE)

To handle label heterogeneity, the CMoE \({\mathcal{M}}_c\) allocates \(N_c\) experts (one per local class, \(N_c = N\)), each specializing in class-specific feature transformations: \[\label{eq:cmoe} \hat{{\boldsymbol{X}}} = \sum_{{\mathbf{n}}=1}^{N_c} W_{{\mathbf{n}}} \cdot E_{{\mathbf{n}}}({\boldsymbol{X}}_{\text{mid}}),\quad \text{where}\; E({\boldsymbol{X}}_{\text{mid}}) = \mathrm{\small ReLU}(\mathrm{\small MLP}({\boldsymbol{X}}_{\text{mid}})),\tag{1}\] where \({\boldsymbol{X}}_{\text{mid}}\) and \(\hat{{\boldsymbol{X}}}\) denote latent and output representations, \(E_{{\mathbf{n}}}\) is the \({\mathbf{n}}\)-th expert network, and \(W_{{\mathbf{n}}}\) is the gating weight. By decomposing the representation space into class-specific subspaces, CMoE transforms the classification problem into learning multiple potential class-specific knowledge bases, each tailored to the client’s local label distribution. The CMoE remains on each client and is never uploaded, preserving personalization.

3.2.0.3 Domain-wise MoE (DMoE)

To handle modality heterogeneity, the DMoE \({\mathcal{M}}_d\) allocates \(N_d\) experts (one per imaging modality), each capturing modality-specific feature transformations. Its architecture mirrors CMoE but with experts indexed by imaging domain rather than class. Expert indices are pre-assigned via a shared global modality vocabulary declared at federation setup, reflecting the practical assumption that each clinical site knows its imaging modality (e.g., a radiology department knows it operates CT scanners); extension to unknown-modality scenarios via online feature clustering is discussed in Appendix C. The DMoE is globally aggregated, enabling cross-client knowledge transfer of domain representations.

For inference, the final prediction combines both MoE modules: \[\label{eq:eq2} y = \gamma_d \cdot {\mathcal{M}}_d({\mathcal{V}}({\boldsymbol{X}})) + \gamma_c \cdot {\mathcal{M}}_c({\mathcal{V}}({\boldsymbol{X}})),\tag{2}\] where \(\gamma_d, \gamma_c \in [0,1]\) with \(\gamma_d + \gamma_c = 1\) balance domain consensus and personalized expertise.

3.3 Heterogeneous Modality Alignment↩︎

Aggregating domain-wise experts across clients with fundamentally different imaging modalities risks introducing consensus bias. We mitigate this through a regularization that explicitly constrains the local DMoE parameters \({\mathcal{M}}_d^i\) toward the global aggregate \(\hat{{\mathcal{M}}}_d\): \[\label{eq:optimization} \begin{align} &\min_{\{\theta_1, \ldots, \theta_Z\}} \sum_{i=1}^Z \frac{n_i}{n} \bigl[{\mathcal{L}}_i(\theta_i;{\mathcal{D}}^i, {\mathcal{D}}_s^i) + \lambda \|{\mathcal{M}}_d^i - \hat{{\mathcal{M}}}_d\|^2\bigr], \\ &\text{s.t.}\quad \hat{{\mathcal{M}}}_d = \mathop{\mathrm{arg\,min}}_{{\mathcal{M}}_d} G({\mathcal{M}}_d^1, \ldots, {\mathcal{M}}_d^Z), \\ &\text{where}\quad {\mathcal{L}}_i(\theta_i;{\mathcal{D}}^i, {\mathcal{D}}_s^i) = -\sum_{c=1}^C y_c \log(\hat{y}_i(x)),\; \theta_i = \{{\mathcal{V}}^i, {\mathcal{M}}_d^i, {\mathcal{M}}_c^i\}. \end{align}\tag{3}\] Here \(\lambda > 0\) controls the alignment strength, \(n_i / n\) weights clients by data volume, and \(\hat{{\mathcal{M}}}_d\) is obtained via FedAvg. The proximal term \(\lambda\|{\mathcal{M}}_d^i - \hat{{\mathcal{M}}}_d\|^2\) serves two purposes: it prevents local domain experts from drifting too far from the global consensus while still allowing modality-specific adaptation. Unlike FedProx [37] and Ditto [16], which apply a global proximal penalty to the entire model, HMA constrains only the DMoE parameters, exploiting the dual-MoE structure to selectively align modality-specific representations while leaving CMoE fully free for per-client personalization. We establish formal convergence guarantees for this in Sec. 3.5.

3.4 Multimodal Extensions↩︎

The dual MoE architecture provides a modality-aware visual backbone that naturally supports richer output modalities beyond classification. We introduce two extensions that operate on the same MoE-enhanced features without modifying the core framework.

3.4.0.1 Caption-Enhanced Learning (CEL)

Medical images encode rich diagnostic information that class labels alone cannot capture, such as lesion morphology, tissue texture, and spatial relationships. This limitation is particularly acute in the Non-overlapped setting (MIMH-SA), where clients hold entirely disjoint imaging modalities: visual feature distributions share no common structure across clients, yet their textual descriptions share a common diagnostic vocabulary regardless of modality. CEL exploits this asymmetry by aligning local visual features with LLM-generated captions through a federated contrastive objective, so that language serves as a shared semantic bridge enabling cross-client representation transfer even when visual domains are completely disjoint. To inject this knowledge, we generate diagnostic captions for each image in the MIMH dataset using GPT-4o [38] via a ReAct-based [39] agent pipeline (full prompts and quality-filtering criteria in Appendix D) in a zero-shot setting (e.g., “Fundus image showing microaneurysms and hard exudates in the macular region, consistent with moderate non-proliferative diabetic retinopathy”). A lightweight text encoder \({\mathcal{T}}\) (PubMedBERT [40]) extracts caption embeddings \({\boldsymbol{t}}= {\mathcal{T}}(\text{caption})\), which are aligned with the visual features \({\boldsymbol{v}}= {\mathcal{V}}({\boldsymbol{X}})\) through the federated contrastive objective: \[\label{eq:cel} {\mathcal{L}}_{\text{CEL}} = -\frac{1}{B}\sum_{j=1}^{B} \log \frac{\exp(\text{sim}({\boldsymbol{v}}_j, {\boldsymbol{t}}_j) / \tau)}{\sum_{k=1}^{B} \exp(\text{sim}({\boldsymbol{v}}_j, {\boldsymbol{t}}_k) / \tau)},\tag{4}\] where \(\text{sim}(\cdot, \cdot)\) denotes cosine similarity, \(\tau\) is a temperature parameter, and \(B\) is the batch size. The total local objective becomes \({\mathcal{L}}_i + \alpha \cdot {\mathcal{L}}_{\text{CEL}}\), where \(\alpha\) balances classification and contrastive losses. Crucially, \({\mathcal{L}}_{\text{CEL}}\) enriches the Observer’s representations with fine-grained semantic structure without requiring caption data to leave the client, preserving privacy. The text encoder parameters remain local; only the visual backbone and DMoE are aggregated.

3.4.0.2 Federated Medical VQA

To demonstrate FM\(^2\)’s capacity for multimodal reasoning, we extend it to Visual Question Answering on medical images. Given an image \({\boldsymbol{X}}\) and a natural-language question \(q\), we extract visual features \({\boldsymbol{v}}= {\mathcal{M}}_d({\mathcal{V}}({\boldsymbol{X}})) \oplus {\mathcal{M}}_c({\mathcal{V}}({\boldsymbol{X}}))\) (where \(\oplus\) denotes concatenation weighted by \(\gamma_d, \gamma_c\)) and question features \({\boldsymbol{q}}= {\mathcal{T}}_q(q)\) via a question encoder \({\mathcal{T}}_q\). A lightweight answer module \({\mathcal{H}}_{\text{vqa}}\) fuses both: \[\label{eq:vqa} a = {\mathcal{H}}_{\text{vqa}}({\boldsymbol{v}}\odot {\boldsymbol{q}}),\tag{5}\] where \(\odot\) denotes element-wise gating. The VQA loss \({\mathcal{L}}_{\text{VQA}}\) (binary cross-entropy over candidate answers) is added to the local objective. The DMoE’s domain-specific experts naturally handle modality-dependent questions (e.g., questions about retinal layers vs.tissue staining), while the CMoE provides category-aware visual reasoning. The question encoder and answer head are locally trained; the shared visual backbone benefits from federated aggregation across all tasks.

3.5 Theoretical Analysis↩︎

We provide convergence and generalization guarantees for the FM\(^2\) optimization defined in Eq. 3 . For clarity, we write the global objective as \(F(\theta) = \sum_{i=1}^Z \frac{n_i}{n}\bigl[{\mathcal{L}}_i(\theta_i) + \lambda\|{\mathcal{M}}_d^i - \hat{{\mathcal{M}}}_d\|^2\bigr]\), where each client’s alignment penalty is included inside the per-client term.

The bound shows that \(\sigma^2/Z\) diminishes with more clients; \(\delta^2\) captures modality heterogeneity reduced by dual MoE disentanglement; and \(2\lambda L\Delta_{{\mathcal{M}}}/T\) vanishes at \({\mathcal{O}}(1/T)\), confirming HMA progressively eliminates modality drift.

Proposition 2 yields three insights: (1) small \(R_d^*,R_c^*\) justify the dual MoE design; (2) HMA keeps \(d_{\mathrm{JS}}\) small by encouraging complementary rather than conflicting specialization; and (3) the CEL objective (Eq. 4 ) directly reduces \(\epsilon_{{\mathcal{V}}}\) by aligning visual features with caption embeddings, forcing the Observer beyond classification supervision toward finer-grained diagnostic representations.

4 Experiments↩︎

4.1 Federated Medical Image Classification↩︎

4.1.0.1 Datasets.

We construct the large-scale Medical Imaging Modality Heterogeneity (MIMH) benchmark from five public MedMNIST datasets [41]: RetinaMNIST (Fundus Camera, 1,600 images), BloodMNIST (Blood Cell Microscope, 17,092 images), TissueMNIST (Kidney Cortex Microscope, 236,386 images), PathMNIST (Colon Pathology, 107,180 images), and DermaMNIST (Dermatoscope, 10,015 images), all at \(224 \times 224\) resolution, loaded from the native MedMNIST+ release and not the \(28 \times 28\) thumbnails commonly associated with MedMNIST. We create four dataset configurations (Fig. 2): MIMH-5/4/3-Domain: 10 clients with overlapping modalities, using \(\texttt{Dirichlet}(\varphi, {\mathbf{p}})\) [42] to control non-IID severity (\(\varphi \in \{0.1, 0.3, 100\}\)); MIMH-SA (Stand-Alone): 5 clients, each holding data from a single unique modality with zero overlap, representing the hardest setting.

4.1.0.2 Baselines and Implementation.

We compare FM\(^2\) against 6 advanced FL methods: FedAvg [14], FedProx [37], PerFedAvg [23], FedBN [18], FedProto [43], and FedRep [19]. All methods use ResNet-4 [44] with SGD [45] (lr \(= 5 \times 10^{-3}\)), 100 communication rounds, and 1 local epoch. FM\(^2\)’s two-stage protocol (50 Observer + 50 MoE rounds, 1 epoch each) matches all baselines’ total budget; see Appendix A for a full computational breakdown. We set \(\gamma_d = \gamma_c = 0.5\) and \(\lambda = 0.1\). FM\(^2\) is instantiated at three scales: FM\(^2\)-Tiny/Medium/Large (\(2\times\)/\(4\times\)/\(8\times\) MLP layers per expert). Results are averaged over five runs with standard deviations reported. All experiments on MIMH datasets are conducted on an NVIDIA A5500.

4.1.0.3 Main Results.

Table 1 reports classification accuracy on MIMH-5/4/3-Domain. FM\(^2\) consistently outperforms all baselines across most settings. The performance improves as expert capacity increases (Tiny \(\to\) Medium \(\to\) Large), confirming that scaling the lightweight MoE modules is an effective strategy.

4.1.0.4 Out-of-Modality Generalization.

We evaluate two generalization aspects on MIMH-SA (Table 2): E1, can modalities benefit from each other when clients hold completely disjoint modalities? E2, can FM\(^2\) generalize to unseen modalities? For E2, we introduce three held-out MedMNIST datasets: PneumoniaMNIST [46] (Chest X-Ray, 5,856), OCTMNIST [46] (Retinal OCT, 109,309), and OrganAMNIST [47] (Abdominal CT, 58,830), fine-tuning with only 5% of data. Results show: (1) baselines collapse on E1 due to inability to capture cross-modality invariances, while FM\(^2\) significantly outperforms even local-only training; (2) on E2, FM\(^2\) transfers effectively to all three unseen modalities, demonstrating genuine foundation model capabilities.

4.2 Caption-Enhanced Learning↩︎

4.2.0.1 Dataset and Setup.

To evaluate the multimodal extension in Sec. 3.4, we augment every image in the MIMH benchmark with a concise diagnostic caption generated by GPT-4o [38] following the ReAct-based pipeline in Appendix D (e.g., “A retinal fundus image showing mild drusen deposits consistent with early AMD”). The visual backbone is upgraded to CLIP ViT-B/16 [6], pretrained on natural image–text pairs, while caption embeddings are extracted by PubMedBERT [40] and fused with image features through a two-head cross-attention layer (dim\(=\)​256). The FM\(^2\) dual MoE modules then operate on the fused representation, and an auxiliary InfoNCE contrastive loss aligns image and caption embeddings locally before aggregation. Training follows the protocol in Sec. 4.1, except that we use AdamW [48] (lr\(=10^{-4}\)), 50 communication rounds, and fix \(r{=}100\%\). For cross-modality adaptation, we reuse the three held-out datasets from Sec. 4.1 with 5% fine-tuning data.

4.2.0.2 Results.

Table 3 compares ten image-only FL baselines with seven caption-enhanced variants. Several observations stand out. First, appending captions yields consistent improvements for every FL method: even the weakest baseline FedAvg gains +4.47% on the Hard setting and +7.35% on MIMH-SA, confirming that language grounding provides complementary information regardless of the aggregation strategy. Second, the improvement is most pronounced on MIMH-SA and cross-modality transfer, where visual domains are completely disjoint and captions serve as a shared semantic bridge; FM\(^2\)-Large+CEL reaches 95.04% cross-modality average, a +4.56% margin over its image-only counterpart. Third, the Tiny\(\to\)Medium\(\to\)Large scaling pattern is preserved under CEL (85.20\(\to\)​87.25\(\to\)​88.73 on MIMH-SA), indicating that MoE capacity scaling and caption supervision contribute orthogonally to performance. Among all image-only methods, FM\(^2\)-Large already surpasses the best non-FM\(^2\) baseline (FedProto) by a large margin, and the addition of CEL further widens this gap.

4.3 Visual Question Answering↩︎

4.3.0.1 Datasets and Setup.

Following the federated medical VQA setup of OmniFM [50], we construct three settings using SLAKE [31], VQA-RAD [32], and VQA-Med 2019/2020/2021 [51], [52]. Task 1 uses SLAKE with modality-based IID/non-IID splits across three clients under varying LLaVA fine-tuning strategies. Task 2 constructs eight modality-specific clients to test cross-domain DMoE specialization. Task 3 treats each dataset as an independent mixed-modality client (five total) under different PEFT regimes. All tasks use LLaVA-1.5 with CLIP ViT-B/32 augmented by our dual MoE modules and LLaMA-3.2-3B fine-tuned via LoRA [53] (\(r{=}8\), \(\alpha{=}32\)). FedDAT, PromptFL, and F\(^3\)OCUS are re-implemented on the identical LLaVA-1.5 backbone with the same LoRA configuration, with only their respective federated adaptation strategies applied, ensuring backbone-level parity across all methods.

2pt 2pt

2pt 3pt

4.3.0.2 Results.

FM\(^2\) consistently outperforms all baselines across all three tasks and tuning strategies (Tables 4–6). On Task 1, gains are largest under F-CL (0.844 vs. for the best baseline in Non-IID), validating that HMA effectively mitigates distribution discrepancies during VQA training. On Task 2, FM\(^2\) achieves the highest average accuracy of 79.60, confirming DMoE’s cross-domain specialization. On Task 3, FM\(^2\) maintains consistent advantages across all PEFT regimes and outperforms specialized federated VLM methods FedDAT [34], PromptFL [35], and F\(^3\)OCUS [36] by clear margins.

4.4 Framework Analysis↩︎

We analyze FM\(^2\) from six complementary angles: (i) component ablation (Fig. 3a), (ii) hyperparameter sensitivity (Fig. 3b), (iii) caption quality and coverage (Fig. 3c–d), (iv) communication cost (Table 7), (v) convergence behavior, and (vi) learned feature structure (Fig. 4). Unless noted, ablation and sensitivity results span three tasks: classification (MIMH-5-Domain, \(\varphi{=}0.1\), FM\(^2\)-Tiny), VQA (Task 1, F-CL, Non-IID), and CEL (MIMH-SA, FM\(^2\)+CEL), to verify that findings generalize across objectives.

4.4.0.1 Component Ablation.

We progressively disable FM\(^2\)’s core components (Fig. 3a). Among single-component removals, disabling CMoE costs 3.4–7.8% and disabling HMA costs 2.1–4.3%. The most harmful variant, Only CMoE, removes both DMoE and HMA and drops 5.1–10.4%, showing that personalization alone is insufficient without global domain consensus. Since w/o-HMA and Only-CMoE differ only in DMoE, removing DMoE alone still costs 5.4 and 1.8 points on MIMH-5-Domain and MIMH-SA, so DMoE contributes beyond a personalized CMoE plus the federated backbone. The importance ordering (Only CMoE \(>\) w/o CMoE \(>\) w/o HMA) is consistent across all three tasks, confirming architectural robustness independent of downstream objective.

4.4.0.2 Hyperparameter Sensitivity.

Fig. 3 (b) examines the effect of varying \(\gamma_c\), \(\gamma_d\), and \(\lambda\) away from the default setting (\(\gamma_c{=}0.5\), \(\gamma_d{=}0.5\), \(\lambda{=}0.1\)). Across all three tasks, no non-default configuration degrades performance by more than 2.31%, indicating that FM\(^2\) is not sensitive to precise hyperparameter tuning and that the default configuration generalizes reliably across diverse settings and tasks.

4.4.0.3 Caption Quality and Coverage.

Fig. 3 (c–d) examines CEL robustness on MIMH-SA. LLM-generated captions (87.25%) nearly match expert annotations (87.89%) while far exceeding templates (84.53%), confirming API generation as a cost-effective substitute for manual labeling. Even at 20% coverage, CEL yields +1.57% over the image-only baseline, with monotonically increasing but diminishing returns, showing that CEL is beneficial even under the uneven caption availability typical of real federated deployments.

4.4.0.4 Communication Cost.

Table 7 reports per-round communication and wall-clock overhead. Because the CMoE module is entirely local (never communicated), the additional communication of FM\(^2\)-Tiny over FedAvg is 38% in parameter count (718K vs.520K), a modest price for a 60.75 percentage-point accuracy gain. FedProto achieves the lowest communication (prototype-only exchange) but plateaus at 70.84%. FM\(^2\)-Large uses \(2\times\) FedAvg’s bandwidth yet reaches 86.68%, representing a fundamentally different accuracy-communication trade-off.

4.4.0.5 Convergence and Feature Analysis.

Figure 4 reports convergence and feature structure. On classification (Fig. 4a), FM\(^2\)-Tiny reaches 80% by round 40, a level no baseline achieves at round 100; FedAvg stagnates below 25% while FedProto and FedRep plateau near 70–75%. On VQA (Fig. 4b), FM\(^2\) converges fastest to 84.4%, above FedPer (83.3%) with tighter variance. t-SNE on MIMH-SA (Fig. 4c–d) shows that without HMA features cluster by modality; with HMA they reorganize by semantic class, with all five modalities interleaving within each cluster, validating HMA’s design objective and the out-of-modality generalization in Tables 1 and 3.

5 Conclusion↩︎

In this paper, we proposed FM\(^2\), a federated multimodal foundation model framework that addresses the under-explored challenge of Imaging Modality Heterogeneity in medical imaging. Through dual Mixture-of-Experts modules and Heterogeneous Modality Alignment with provable convergence guarantees, FM\(^2\) disentangles personalized class knowledge from shared domain representations. Extensions to Caption-Enhanced Learning and Federated Medical VQA demonstrate FM\(^2\)’s versatility as a multimodal foundation model. Extensive experiments on the MIMH benchmark (for classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art baselines, fast convergence, strong out-of-modality generalization, and practical robustness to caption availability and hyperparameter choices.

References↩︎