1 Introduction↩︎

Clinicians inherently integrate information from a variety of sources and modalities such as pathology reports, laboratory data, omics data, X-Ray, Magnetic Resonance Imaging (MRI), and Computed Tomographhy (CT) to form holistic and context-aware clinical judgements. This has motivated the recent development of models that make use of highly multimodal data to computationally mirror such integrative clinical reasoning [1]–[3]. In healthcare, multimodal data is inherently fragmented across medical centers due to privacy regulations and data governance constraints, posing significant challenges to centralized training paradigms. Federated Learning (FL) [4] addresses these challenges and provides a privacy-preserving approach to collectively training shared healthcare models without any exchange or distribution of patient data residing in medical centers. Recent efforts have therefore explored multimodal learning under the federated setting [5]–[7]. However, most of the existing literature on multimodal federated learning in healthcare assumes that all participating institutions have access to complete sets of modalities [8], [9]. In real-world settings, multimodal clinical data are frequently incomplete across centres owing to disparities in equipment availability, acquisition protocols, and data management resources. This challenge calls for novel methodologies for dealing with missing modalities. In centralized multimodal learning, the challenge of missing modalities is addressed through prompt-based [10], [11], generative [12]–[14], dropout-based [15] methods, or utilize specialized architectures [16]–[18] to model inter-modal dependencies. Such centralized approaches are impractical in FL scenarios, as their architectural designs do not consider the strict privacy, communication efficiency, and heterogeneity requirements inherent to practical FL deployments.

More recently, a growing body of FL work has begun to address the challenge of missing modalities, predominantly in non-medical domains [19]–[22], while a limited number of studies were proposed for the medical domain [23]–[26]. CAR-MFL [23] handles missing modalities by training with additional public multimodal data to compensate for incomplete client modalities. However, this strategy assumes the availability of representative public data, which is often unrealistic in healthcare due to privacy constraints and domain gaps between public and private datasets. FeatImp [24] performs feature-level imputation by training separate networks to learn modality-specific mappings to reconstruct missing features. This approach assumes stable cross-modal relationships across clients and synthesizes features solely from the observed modality, without explicitly modeling complementary information inherent to the missing modality. PmcmFL [21] substitutes class-level prototypes as features of missing modalities. Although prototype-based and closer to our formulation, it assigns identical representations to all samples within a class which limits instance-level diversity and fails to capture intra-class variability, which is critical in medical data.

To address these limitations, we propose ProMoE-FL, a prototype-conditioned Mixture-of-Experts framework for missing modality synthesis in federated learning. The method generates modality-specific features by conditioning on available modalities and a global prototype bank of missing modalities. This enables privacy-preserving and structured feature reconstruction without external data. A shared, routing-based Mixture-of-Experts (MoE) architecture allows experts to be reused across modalities, preventing combinatorial parameter growth. Furthermore, by accumulating client-specific prototypes that summarize local modality distributions, the global prototype bank serves as a client-aware prior in the latent space. This guides feature synthesis to account for distributional variability across clients, thereby improving robustness under the non-independently and identically distributed (i.e., non-IID) data characteristics prevalent in real-world federated settings. Our key contributions are the following:

• We propose ProMoE-FL, a prototype-conditioned feature generation framework for missing modality feature synthesis in federated multimodal learning.

• We introduce client-aware prototype attention to address cross-client heterogeneity and employ a MoE architecture for direction-aware synthesis within a shared parameterization.

• We conduct extensive experiments under heterogeneous settings that mirror real-world clinical data silos, demonstrating improved robustness. We further support our findings with qualitative analyses.

2 Method↩︎

We formulate our method for a typical multimodal federated setting with \(K\) clients, each with its private dataset \(D_k\) containing \(n_k\) samples. The \(n\)-th data sample in \(D_k\) is represented by the tuple \((\{X_m^{(n)}\}_{m=1}^{M_k},Y^{(n)})\), where \(Y^{(n)}\) denotes the label related to \(C\) classes and \(M_k\) is the number of modalities available at the \(k\)-th client. All clients share common architecture with the global model, which is defined as \(\{\boldsymbol{w} = \{f_e^{m}\}_{m \in \mathcal{M}}, \; f_c\}\), where \(\mathcal{M}\) denotes the complete set of modalities, \(f_e^{m}\) is the encoder for modality \(m\), and \(f_c\) is the classifier. We denote the latent feature representation extracted by the modality specific encoder as \(h_m^{(n)} = f_e^{m}(X_m^{(n)}), \quad m \in \mathcal{M}\). The objective of federated training is to minimize the data-weighted empirical risk across all the clients, which is expressed as: \[\boldsymbol{w}^* = \arg\min_{\boldsymbol{w}} \sum_{k=1}^{K} \frac{|D_k|}{|D|} \, \mathcal{L}_{task}^{(k)}(\boldsymbol{w}), \label{eq:global95optimization}\tag{1}\] with the local loss at client \(k\): \[\mathcal{L}_{task}^{(k)}(\boldsymbol{w}) = \frac{1}{|D_k|} \sum_{n=1}^{|D_k|} \mathcal{L}_{task} \Big( f_c\big( \bigoplus_{m \in \mathcal{M}} f_e^{m}(X_m^{(n)}) \big), Y^{(n)} \Big), \label{eq:local95optimization}\tag{2}\] where, \(\oplus\) denotes a feature fusion operator, implemented as concatenation of modality-specific latent representations, and \(D = \bigcup_{k=1}^{K} D_k\). In practical multi-center clinical deployments, multimodal systems frequently face missing modalities. A common approach to handle missing modality is zero-filling. For example, if the image modality is unavailable while text is present, the objective reduces to \(\mathcal{L}_{task}\big(f_c(0 \oplus h_T^{(n)}), Y^{(n)}\big)\). Naively optimizing it distorts the joint feature space and biases the global model toward frequently observed modalities, thereby under-representing sparse ones.

Fig. 1 presents the overall framework of ProMoE-FL. For unimodal clients, a feature synthesis network reconstructs representations of missing modalities. While our approach operates in the feature space similarly to [24], ProMoE-FL differs in both design and robustness. Learning modality mappings solely from multimodal clients can degrade performance under client heterogeneity, where cross-modal relationships vary across sites and available modalities provide incomplete priors. To mitigate this, we introduce a client-aware global prototype bank encoding federation-wide modality priors, conditioning the synthesis process on target-modality prototypes. Furthermore, to ensure scalability without combinatorial parameter growth, the synthesis module is implemented as a Mixture-of-Experts architecture that dynamically routes modality-specific transformations through expert subnetworks.

Prototype Construction and Alignment. In heterogeneous federated settings, modality embeddings often diverge across clients due to distribution shifts and inconsistent modality availability [27]. As a result, mappings learned only from multimodal clients may be insufficient for reliable cross-modal synthesis. Inspired by [20], [21], we introduce modality-specific, client-aware prototypes that act as global priors of the missing modality to complement the available modality and improve robustness to cross-client heterogeneity. For each modality \(m\), every client \(k\) maintains a set of \(C\) learnable prototypes \(\mathbf{P}_m^k = \{\mathbf{p}_1^{(m,k)}, \ldots, \mathbf{p}_C^{(m,k)}\}\) with \(\mathbf{p}_c^{(m,k)} \in \mathbb{R}^d\). For each sample with at least one positive label, the modality-specific target centroid \(\mathbf{c}_m^{(n)}\) is obtained by aggregating the corresponding class prototypes weighted by the multi-hot label vector and normalizing the result to unit norm. For each modality \(m\), we obtain a projected representation \(\mathbf{z}_m^{(n)} = f_{\mathrm{proj}}^{m}(\mathbf{h}_m^{(n)}) \in \mathbb{R}^d\) using a modality-specific projection head implemented as a Fully Connected (FC) layer, and it is aligned with its corresponding centroid via \(\mathcal{L}_{proto}^{(m)} = \frac{1}{|\mathcal{I}|} \sum_{n \in \mathcal{I}} \|\mathbf{z}_m^{(n)} - \mathbf{c}_m^{(n)}\|_2^2\), where \(\mathcal{I} = \{ n \mid \sum_{c=1}^{C} y_c^{(n)} > 0 \}\). As illustrated in Fig. 1 (b), we jointly optimize the client parameters \(\mathbf{w}^k\) together with modality-specific projection heads and prototypes, \(\{ f_{\mathrm{proj}}^{m}, \mathbf{P}_k^{m} \}_{m \in \mathcal{M}_k}\), where \(\mathcal{M}_k \subseteq \mathcal{M}\) denotes the set of modalities available on client \(k\). This is done via the following full training objective: \[\label{eq:local95loss} \mathcal{L} = \mathcal{L}_{\mathrm{task}} + \lambda \sum_{m \in \mathcal{M}_k} \mathcal{L}_{\mathrm{proto}}^{(m)},\tag{3}\] where \(\mathcal{L}_{\mathrm{task}}\) is the task-specific loss (Eq. 2 ), \(\mathcal{L}_{\mathrm{proto}}^{(m)}\) is the prototype alignment loss for modality \(m\), and \(\lambda\) controls the relative importance of the prototype loss. The prototypes are updated through gradient-based optimization each communication round, which encourages alignment with class representations while maintaining diversity, hence, improving robustness in heterogeneous settings.

Missing Modality Feature Synthesis. For unimodal clients, we propose a Prototype-Conditioned Decoder (PCD) \(\phi\), illustrated in Fig. 1 (c), to enable cross-modal feature synthesis. The PCD is implemented as a Transformer decoder where the source feature \(h_{i}\) of available modality \(i\) attends over the global prototype bank \(P_{j}\) of the target modality \(j\) received from the server to retrieve context necessary for synthesizing the target modality feature. Importantly, \(P_{j}\) is constructed via accumulation of prototypes from clients rather than aggregation (e.g., averaging). Consequently, the decoder can dynamically attend to the most relevant target prototypes during synthesis, enabling adaptive complementary knowledge retrieval across heterogeneous domains.

However, a single PCD may be insufficient to capture complex, non-linear relationships between heterogeneous modality pairs, especially in highly multimodal scenarios in the medical domain [9]. Thus, we propose a Mixture of Experts \(\Phi_{\mathrm{MoE}}\), illustrated in Fig. 1 (d) where we define \(E\) PCDs \(\phi_e(h_{i}, P_{j})\) as independent experts \(\{\phi_e\}_{e=1}^E\). In order to redirect the data flow via the right experts, a Modality-Aware Router \(\mathcal{R}\) computes a gating distribution conditioned on the instance context and the modality indices. Therefore, the network synthesizes the bottleneck feature of the missing modality \(j\) as: \[\hat{h}_j^{(n)} = \Phi_{\mathrm{MoE}}(h_i^{(n)}, P_j; i, j) = \frac{ \sum_{e=1}^{E} \mathcal{R}(h_i^{(n)}, P_j, i, j)_e \; \phi_e(h_i^{(n)}, P_j) }{ \left\| \sum_{e=1}^{E} \mathcal{R}(h_i^{(n)}, P_j, i, j)_e \; \phi_e(h_i^{(n)}, P_j) \right\|_2 }.\] This formulation allows \(\Phi_{\mathrm{MoE}}\) to adaptively select the optimal combination of prototype-conditioned experts for any missing modality scenario. Although the proposed framework is formulated for an arbitrary set of modalities \(\mathcal{M}\), for clarity of exposition we specialize the following derivations to the two-modality case, \(\mathcal{M} = \{I, T\}\) used in our experiments, namely Image (\(I\)) and Text (\(T\)), without loss of generality. For training unimodal clients, as shown in Fig. 1 (f), the task-calibrated loss is hence, defined as: \[\mathcal{L}_{task} = \mathcal{L}\!\left( fc\!\left( h_i^{(n)} \oplus \hat{h}_j^{(n)} \right), Y^{(n)} \right), \qquad \hat{h}_j^{(n)} = \Phi_{\mathrm{MoE}}(h_i^{(n)}, P_j; i, j)\] where, \(i,j \in \mathcal{M},\; i \neq j.\)

Federated Training in ProMoE-FL. At the start of each communication round, the server dispatches \(\Phi_{\mathrm{MoE}}\) , \(\boldsymbol{w}\), and prototype bank \(\mathbf{\{{P}}_m^{global}\}_{m \in \mathcal{M}}\) to all clients. Each client performs local training for \(N\) epochs, optimizing the training objective \(\mathcal{L}\) as defined in Eq. 3 . On multimodal clients, we construct a pool of paired features \(\mathcal{H}_k = \{(h_I^{(n)}, h_T^{(n)}) \mid (x_I^{(n)}, x_T^{(n)}) \in D_k\}\). As illustrated in 1 (e), this pool is then used to train the feature synthesis network \(\Phi_{MoE}^k\) using a symmetric reconstruction objective: \[\mathcal{L}(\Phi_{MoE}^k) = \frac{1}{|\mathcal{H}_k|} \sum_{(h_I^{(n)}, h_T^{(n)}) \in \mathcal{H}_k} \sum_{\substack{i,j \in \mathcal{M}_k\\ i \neq j}} \left\| \Phi_{MoE}^k(h_{i}^{(n)}, P_{j}; i, j) - h_{j}^{(n)} \right\|_2^2 .\] At the end of each communication round, each client transmits its updated model parameters \(\boldsymbol{w}^k\), the modality-specific projection heads, and the local prototype sets \(\mathbf{P}_m^k\) to the server. Multimodal clients additionally communicate \(\Phi_{MoE}^k\). The resulting communication overhead is negligible, as the prototype bank, projection, and modality-aware routing mechanism together account for only approximately \(0.3\%\) of the total model parameters. All the communicated parameters are aggregated via FedAvg [4] due to its simplicity and effectiveness as demonstrated by prior studies [23]–[25]. In contrast to the model parameters, the server accumulates all client prototypes into a global prototype bank: \(\mathbf{P}_m^{global} = \bigcup_{k=1}^{K} \mathbf{P}_m^k\) for each modality \(m \in \mathcal{M}.\)

3 Experiments and Results↩︎

Datasets and Setups. We perform a thorough experimentation on challenging datasets in both homogeneous and heterogeneous setups: MIMIC-CXR [28], [29], NIH Open-I [30], PadChest [31], and CheXpert [32]. We use the validation and test splits provided by MIMIC-CXR for the evaluation of the global model, following the protocol outlined in [23]. We experimented with a variety of client configurations, denoted as I:T:M, where I, T, and M represent the numbers of image-only, text-only, and multimodal clients, respectively. The homogenous setup comprises 10 clients, each with training data for 810 patients sampled from the same dataset MIMIC-CXR. In the heterogeneous setup, multimodal clients include 2 clients from NIH Open-I, while any additional multimodal clients are sampled from PadChest. All image-only and text-only clients are drawn from CheXpert and PadChest, respectively.
Implementation Details. We employ pretrained Resnet50 [33] and BERT-base [34] architecture as the image and text encoders, respectively. Their outputs are projected to 256-dimensional features and L2-normalized prior to concatenation. Optimization is performed using Adam [35] with a learning rate of \(1e^{-4}\). \(\lambda\) is tuned from \(\{0.1, 0.5, 1, 5, 10\}\). Each client trains locally for 3 epochs per communication round over 30 rounds in total. Performance is measured using macro AUC (Average area under the Receiver Operating Characteristic Curve). The results are averaged over three random seeds.
Baselines. We compare our method with two simple yet effective baselines: Zero-filling which replaces missing modalities with zero vectors and Uniform-filling which uniformly samples feature vectors (FedAvg [4] + Zero/Unif.). We compare with state-of-the-art (SOTA) methods: FeatImp [24] and PmcmFL [21], representing feature-level imputation and prototype-based approaches, respectively. One of the advantages of our ProMoE-FL is that it does not rely on public data like CAR-MFL [23]. However, ProMoE-FL can be easily adapted by incorporating a virtual client that uses public data during training. In this way, we can fairly compare with public-data-based SOTA, CAR-MFL [23].

Quantitative Results. Table ¿tbl:tab:perform? reports results across client configurations under both homogeneous and heterogeneous settings. In the homogeneous case, our method achieves the best performance in 6 configurations, with only a modest average gap of \(0.37\%\) in the remaining 3. Under heterogeneous distributions, which represent a more realistic federated scenario, ProMoE-FL consistently outperforms all baselines across every configuration. Notably, PmcmFL shows the largest degradation, likely due to its reliance on a single class-wise prototype for missing modalities, which limits adaptability under distributional shifts. Finally, in the heterogenous settings, we compare our method with CAR-MFL[23], SOTA for Public dataset based approach. The results demonstrate that our method achieves competitive performance. We also ablate the mixture-of-experts component in the 8:0:2 heterogeneous setting. A single PCD achieves an AUC of 78.98, whereas the MoE variant attains 79.68, indicating a performance gain from incorporating expert routing.

Qualitative Results. Figure 2 (a–b) shows t-SNE analyses of synthesized features from the test set of ProMoE-FL, FeatImp and the ground truth. FeatImp demonstrates mild centroid offsets and slightly more diffuse feature dispersion, indicating comparatively less precise alignment in the latent space. In contrast, ProMoE-FL shows a tighter integration between synthesized and ground-truth features, with a more precise alignment of their class centroids. These observations suggest that ProMoE-FL provides a more refined and structurally faithful semantic mapping. As shown in Figure 2 (c), expert utilization remains balanced across both synthesis directions, with gating values centered near 0.5. This suggests stable multi-expert collaboration rather than collapse to a single expert, supporting the use of weighted gating instead of sparse Top-\(k\) routing.
Clinical Relevancy. The UMAP visualizations in Figure 2 (d–e) highlight performance on rare pathological classes. While FeatImp captures the broader pathological structure, it exhibits a degree of mode collapse where synthesized centroids for different pathologies converge toward a single point. For conditions such as Enlarged Cardiomediastinum, Fracture, and Lung Lesion, ProMoE-FL maintains clear separation between class centroids while remaining closely aligned with the ground truth. This ability to prevent feature overlap is vital in a medical context, as it ensures that synthesized data from missing-modality patients retains the discriminative signals necessary for reliable diagnosis. Moreover, PadChest [31] contains Spanish radiology reports, whereas the other datasets are in English. To the best of our knowledge, this is the first work to consider linguistic domain heterogeneity in federated learning with missing modalities, reflecting realistic cross-lingual clinical settings.

4 Conclusion↩︎

In this work, we introduced ProMoE-FL, a prototype-conditioned Mixture-of-Experts framework for multimodal federated learning under missing modality constraints. By conditioning the synthesis on a client-aware global prototype bank of missing modality, ProMoE-FL enables robust cross-modal feature synthesis addressing data heterogeneity in multimodal federated setups without relying on public data. Extensive experiments across various client configurations show consistent improvements over state-of-the-art methods, particularly in heterogeneous (non-IID) settings, where conventional imputation approaches struggle to generalize. We demonstrate strong alignment between our synthesized and ground-truth feature distributions, preserving diagnostic structures, including rare pathologies such as lung lesions, enlarged cardiomediastinum, and fractures. By moving beyond simple feature imputation toward adaptive prototype-conditioned synthesis, ProMoE-FL provides a practical approach to addressing data heterogeneity in real-world clinical federated learning.

References↩︎