1 Introduction↩︎

The proliferation of Multimodal Large Language Models (MLLMs) has marked a significant milestone in artificial intelligence, demonstrating unprecedented capabilities in understanding and generating content that integrates vision and language [1], [2]. Driven by these capabilities, MLLMs are increasingly pivotal in high-stakes, real-world applications ranging from medical diagnosis and autonomous driving to content moderation in safety-critical networked deployments [3], [4]. However, deploying and maintaining MLLMs in privacy-sensitive domains necessitates training methodologies that do not rely on centralized data storage. Federated Learning (FL) has emerged as a leading paradigm for privacy-preserving machine learning, enabling collaborative model training on decentralized data sources without exposing raw user data [5]–[7]. Consequently, the federated fine-tuning of MLLMs represents a critical frontier for developing adaptable and secure autonomous AI systems that can be continuously improved in dynamic network environments [8].

A fundamental challenge arises when these decentralized data streams are non-stationary, evolving over time as new concepts, tasks, or data distributions emerge. In such scenarios, MLLMs must be updated sequentially to stay current, a process known as continual learning that is fraught with the peril of catastrophic forgetting, wherein the model’s performance on previously learned tasks deteriorates drastically upon learning new ones [9], [10]. A comprehensive taxonomy of continual learning strategies has been established [11], yet most approaches were designed for centralized settings and do not address the federated, multimodal regime. The problem is particularly acute in safety-critical applications: a model proficient at detecting known forms of hateful content [3] must be updated to recognize new evolving patterns, while a disaster-response classifier must retain prior event categories as new crisis types emerge [4]. Addressing catastrophic forgetting within a federated framework is therefore not merely a technical curiosity but a practical necessity for building robust and reliable autonomous MLLMs.

The intersection of continual learning and federated learning introduces a complex set of technical barriers. At its core, catastrophic forgetting stems from the stability-plasticity dilemma, where a model must be plastic enough to acquire new knowledge yet stable enough to retain old information [12]. Foundational continual learning methods developed for centralized settings, such as regularization-based approaches like Elastic Weight Consolidation (EWC) [13] and Synaptic Intelligence [14], or replay-based strategies that rehearse on stored exemplars [15], offer partial solutions. However, their direct application in FL is problematic: regularization methods often struggle to scale in the complex parameter spaces of MLLMs [16], while replay-based methods conflict with the core privacy principles of FL if raw data is stored or shared. Moreover, the inherent statistical heterogeneity (non-IID data) across clients in FL exacerbates forgetting, as divergent local updates can destructively interfere with one another during aggregation [17], [18].

Practical implementation challenges further compound these issues, particularly for MLLMs. A key difficulty lies in the multi-faceted nature of knowledge within these models: information is encoded not just within the vision encoder and language model separately but also in the intricate cross-modal alignment modules, which introduce unique vulnerability to modality-specific forgetting and adversarial drift [19]. A continual learning strategy must therefore be modality-aware, protecting critical parameters across all three components without stifling the model’s ability to learn new cross-modal relationships. Furthermore, client drift becomes substantially more pronounced in a continual learning context [20]: in a given round, some clients might be training on a new task while others remain on older data, producing gradient updates that are misaligned in their objectives. Adaptive server-side optimization methods [21], [22] reduce drift in static settings but are not designed for the evolving task objectives of continual learning. Client-side generative replay [23] can partially alleviate data-level forgetting, yet synchronizing it across tasks in a privacy-preserving federated regime remains an open challenge.

Recent advancements in Federated Continual Learning (FCL) have begun to tackle some of these issues. Methods such as FedWeIT [24] and Learning without Forgetting (LwF) [25] adapt knowledge distillation techniques to the federated setting, using the global model’s outputs on new data as soft labels to preserve old knowledge. Others directly apply regularization by computing parameter importance on the server, while FedProx-style methods [26] penalize deviations from the global model. More recent studies have expanded the landscape: AF-FCL selectively reuses prior knowledge under heterogeneous task drift [27], FedCBDR improves replay balance for federated class-incremental learning [28], cooperative multi-model training across heterogeneous devices addresses system heterogeneity from a signal-processing perspective [29], and Sec-MMFL studies privacy leakage and modality-specific protection in multimodal federated learning [30]. Although these methods mark clear progress, they are still not designed for continual federated adaptation of large MLLMs with modality-aware consolidation, privacy-preserving multimodal replay, and aggregation-time conflict suppression handled in one unified pipeline.

The limitations of existing methods motivate a new approach that synergistically integrates solutions to the core challenges of multimodal forgetting, federated data heterogeneity, and privacy exposure. Our key insight is that an effective autonomous optimization framework for networked MLLMs must operate at three distinct levels. At the parameter level, modality-aware regularization protects established knowledge through parameter importance estimation [13] applied to parameter-efficient adapter representations [31]. At the data level, a raw-data-free rehearsal mechanism actively combats forgetting, extending exemplar replay [32] to a local generative synthesis paradigm suitable for federated LLM adaptation [33]. At the aggregation level, a similarity-driven autonomous strategy mitigates interference from heterogeneous client updates, moving beyond static parameter isolation [34] toward gradient-informed coordination. By combining these three pillars, it becomes possible to balance stability and plasticity in the complex, distributed, and evolutive training environment of MLLMs deployed across networked AI infrastructures.

To address these challenges, we propose Federated Continual Multimodal Learning (FedCMM), a comprehensive framework designed to enable MLLMs to learn a sequence of tasks in a federated setting while mitigating catastrophic forgetting. The core innovation of FedCMM lies in its tripartite architecture that harmonizes parameter-level regularization, data-level rehearsal, and server-level aggregation. First, we introduce Modality-Aware Elastic Weight Consolidation (MA-EWC), a novel regularization strategy that computes and protects critical parameters independently for the vision encoder, the language model, and the cross-modal projector. Granular per-modality protection prevents destructive interference. Second, we design a Privacy-Preserving Federated Replay (PPFR) mechanism, where each client trains a lightweight, local generative replay module to synthesize raw-data-free embedding-level replay tuples from past tasks. During subsequent training, clients use these replay tuples for rehearsal, effectively refreshing old knowledge without sharing any raw data. Third, we develop Task-Similarity-aware Gradient Aggregation (TSGA), a server-side algorithm that weights client model updates based on their gradient cosine similarity, down-weighting updates that are likely to conflict with the current learning trajectory and thus reducing inter-client interference.

Our work offers several key contributions:

• We propose FedCMM, a federated continual multimodal learning framework designed to mitigate catastrophic forgetting during privacy-preserving sequential MLLM fine-tuning.

• We introduce three complementary components: MA-EWC for modality-specific parameter protection, privacy-preserving federated replay for synthetic rehearsal, and TSGA for similarity-aware aggregation.

• We conduct extensive experiments on two challenging continual federated benchmarks that show that FedCMM outperforms strong federated continual learning baselines.

The remainder of this paper is organized as follows. 2 reviews related work. 4 presents the technical details of our proposed FedCMM framework. 5 presents the experimental setup, main results, and analysis. Finally, 6 concludes the paper and discusses future work.

2 Related Work↩︎

2.1 Regularization and Replay in Continual Learning↩︎

Continual Learning (CL) aims to enable models to learn from a continuous stream of data without catastrophically forgetting previously acquired knowledge [9]. The core challenge is balancing the stability required to retain old knowledge with the plasticity needed to acquire new information [12]. Research in CL is extensive and can be broadly categorized into three main families. Regularization-based methods, such as EWC [13] and Synaptic Intelligence (SI) [14], introduce a penalty term into the loss function to constrain updates to parameters deemed important for past tasks. While effective, these methods can be computationally expensive and may underperform under significant task distribution shifts. In contrast, replay-based methods store a small buffer of exemplars from past tasks for rehearsal during the learning of new tasks [15]. Approaches like Gradient Episodic Memory (GEM) [15] and iCaRL [32] have demonstrated strong performance, but their memory requirements and, more critically, the privacy implications of storing raw data make them unsuitable for many real-world applications. A third category, parameter-isolation methods, allocates distinct subsets of model parameters for different tasks to prevent interference [34]. However, such methods often suffer from limited scalability and poor knowledge transfer between tasks. All these foundational CL strategies were designed for centralized training and cannot be directly applied in a federated setting due to privacy and communication constraints.

2.2 Optimization Under Heterogeneity in Federated Learning↩︎

FL provides a framework for training models on decentralized data while preserving user privacy [6]. The seminal Federated Averaging (FedAvg) algorithm [5] introduced a simple yet effective procedure of local client training followed by server-side model averaging. A primary challenge in FL is managing statistical heterogeneity (non-IID data) across clients, which can cause client drift and lead to poor convergence and performance [17]. A significant body of work has sought to address client drift. One line of research focuses on client-side regularization; FedProx [26], for instance, adds a proximal term to the local loss function to limit the divergence of local models from the global model. Another popular approach involves improving the server-side aggregation algorithm. For example, FedAdam and FedYogi [21] adapt concepts from adaptive optimizers to the server’s update rule to achieve faster and more stable convergence. Similarly, SCAFFOLD [22] uses control variates to correct for client drift at both the client and server levels. While these methods improve the robustness of FL on static, non-IID data distributions, they inherently assume that the underlying tasks are fixed and are not designed to handle the dynamic, non-stationary data streams encountered in continual learning scenarios.

2.3 Federated Continual Learning Under Task Drift↩︎

FCL has recently emerged to address the problem of learning a sequence of tasks in a federated environment. Most existing FCL methods adapt centralized CL techniques to the constraints of FL. For instance, LwF-style methods [25] and FedWeIT [24] are federated adaptations of knowledge distillation, where the global model from a previous task serves as a "teacher" to regularize the training of the new model, thereby preserving old knowledge without requiring old data. Similarly, EWC-style methods extend regularization-based consolidation to the federated setting [13]. Another line of work has explored federated replay mechanisms. Given the privacy constraints of FL, generative replay has become a promising direction [23]. Recent work has further expanded FCL toward harder settings, including accurate forgetting under heterogeneous clients [27], online data streams [20], and balanced replay for class-incremental federated learning [28]. However, these approaches have predominantly been evaluated on unimodal benchmarks and do not address the modality-specific interference, multimodal replay quality, and aggregation conflicts that arise in MLLM adaptation. Our work fills this gap by proposing a holistic framework specifically tailored for the multimodal setting.

3 Preliminaries↩︎

FL enables model training on decentralized data without requiring data centralization. Consider a federated network of \(K\) clients, indexed by \(k \in \{1, \dots, K\}\). Each client \(k\) possesses a local dataset \(\mathcal{D}_k = \{(\mathbf{x}_i, y_i)\}_{i=1}^{N_k}\), where \(N_k = |\mathcal{D}_k|\) is the number of local samples. The total number of samples in the network is \(N = \sum_{k=1}^K N_k\). The goal of FL is to collaboratively train a single global model with parameters \(\mathbf{w}\) by minimizing a global objective function \(L(\mathbf{w})\), which is a weighted average of the local client objectives \(L_k(\mathbf{w})\): \[\begin{align} \min_{\mathbf{w}} L(\mathbf{w}) = \sum_{k=1}^K \frac{N_k}{N} L_k(\mathbf{w}), \label{eq:fl95objective} \end{align}\tag{1}\] where \(L_k(\mathbf{w}) = \frac{1}{N_k} \sum_{(\mathbf{x}_i, y_i) \in \mathcal{D}_k} \ell(f(\mathbf{x}_i; \mathbf{w}), y_i)\) is the local loss on client \(k\)’s data, with \(f(\cdot; \mathbf{w})\) representing the model’s prediction and \(\ell(\cdot, \cdot)\) being a task-specific loss function. The canonical FL algorithm, FedAvg [5], iteratively performs the following steps: (1) a central server broadcasts the current global model \(\mathbf{w}\) to a subset of clients; (2) each selected client performs multiple local training steps on its data to obtain updated local parameters \(\mathbf{w}_k\); (3) the clients send their updated parameters back to the server, which aggregates them to produce the next global model \(\mathbf{w} \leftarrow \sum_{k} \frac{N_k}{N} \mathbf{w}_k\).

CL, also called lifelong learning, addresses the challenge of training a model on a sequence of tasks over time without access to the full data from previous tasks. Formally, the model is exposed to a sequence of \(T\) tasks, \(\mathcal{T} = \{\mathcal{T}_1, \mathcal{T}_2, \dots, \mathcal{T}_T\}\), where each task \(\mathcal{T}_j\) has an associated data distribution \(P^{(j)}(\mathbf{x}, y)\). The primary challenge in CL is catastrophic forgetting, where the model’s performance on earlier tasks \(\{\mathcal{T}_1, \dots, \mathcal{T}_{j-1}\}\) degrades significantly after being trained on the current task \(\mathcal{T}_j\). The objective of CL is to learn the parameters \(\mathbf{w}^{(T)}\) for the model after the final task \(\mathcal{T}_T\) such that the model performs well across all tasks seen so far. A successful CL method must balance plasticity, the ability to learn new knowledge from the current task, with stability, the ability to retain knowledge from past tasks. The federated continual multimodal learning setting integrates the challenges of FL and CL in the context of MLLMs. In this setting, a sequence of \(T\) distinct multimodal tasks arrives over time. The data for each task \(\mathcal{T}_j\) is distributed across the \(K\) federated clients, where each client \(k\) holds a local dataset \(\mathcal{D}_k^{(j)}\). Each data sample is a multimodal pair \((\mathbf{x}_i, y_i)\), where the input \(\mathbf{x}_i = (\mathbf{v}_i, \mathbf{t}_i)\) consists of a visual component \(\mathbf{v}_i \in \mathcal{V}\) (e.g., an image) and a textual component \(\mathbf{t}_i \in \mathcal{T}\) (e.g., a text prompt). The model \(f(\cdot; \mathbf{w})\) is an MLLM parameterized by \(\mathbf{w}\).

When the system is learning task \(\mathcal{T}_j\), it is assumed that data from previous tasks \(\{\mathcal{T}_1, \dots, \mathcal{T}_{j-1}\}\) is no longer directly available, and client data cannot be shared with the server or other clients due to privacy constraints. The global objective at the end of learning task \(\mathcal{T}_T\) is to find optimal model parameters \(\mathbf{w}^*\) that minimize the average loss across all tasks encountered: \[\begin{align} \mathbf{w}^* = \arg\min_{\mathbf{w}} \frac{1}{T} \sum_{j=1}^T L^{(j)}(\mathbf{w}), \label{eq:fcml95objective} \end{align}\tag{2}\] where \(L^{(j)}(\mathbf{w}) = \sum_{k=1}^K \frac{N_k^{(j)}}{N^{(j)}} L_k^{(j)}(\mathbf{w})\) is the global loss for task \(\mathcal{T}_j\). The core challenge is to minimize the objective in 2 under the dual constraints of (1) not having direct access to past task data and (2) adhering to the privacy-preserving communication protocol of federated learning.

4 Method↩︎

4.1 Method Overview↩︎

To address the challenge of catastrophic forgetting in federated continual learning for MLLMs, we propose FedCMM, a framework that integrates three synergistic components to balance knowledge stability and plasticity. As illustrated in 1, FedCMM embeds continual-learning safeguards directly into the federated training loop. Each client combines current-task data with synthetic replay samples, constrains important modality-specific parameters through MA-EWC, and uploads only its local model update. The server then evaluates update consistency through TSGA, suppresses conflicting directions, and broadcasts the aggregated global model for the next round and task. Formally, when learning the \(j\)-th task \(\mathcal{T}_j\), each participating client \(k\) aims to find local model parameters \(\mathbf{w}_k^{(j)}\) that minimize a composite loss function on its local data \(\mathcal{D}_k^{(j)}\) and a set of locally generated embedding-level replay tuples \(\mathcal{B}_k^{(j-1)}\) from the previous task. The local objective for client \(k\) is: \[\begin{align} {!}{\displaystyle \mathcal{L}_k^{(j)}(\mathbf{w}) = \mathcal{L}_{\text{task}}^{(j)}(\mathbf{w}) + \gamma\,\mathcal{L}_{\text{replay}}^{(j-1)}(\mathbf{w}) + \lambda\,\mathcal{L}_{\text{MA-EWC},k}^{(j-1)}(\mathbf{w}) } \label{eq:overall95local95loss} \end{align}\tag{3}\] where \(\mathcal{L}_{\text{task}}^{(j)}\) is the supervised loss on current-task real samples, \(\mathcal{L}_{\text{replay}}^{(j-1)}\) is the embedding-level rehearsal loss (see 6 ), \(\gamma\) weights the replay contribution, and \(\lambda\) balances the MA-EWC regularization. After local training, the server aggregates the resulting client updates using the TSGA mechanism to produce the new global model \(\mathbf{w}^{(j)}\). For clarity, we explicitly partition the trainable parameters as \[\begin{align} {!}{\displaystyle \mathbf{w} = \mathbf{w}_V \cup \mathbf{w}_L \cup \mathbf{w}_P,\quad \mathbf{w}_V \cap \mathbf{w}_L = \mathbf{w}_V \cap \mathbf{w}_P = \mathbf{w}_L \cap \mathbf{w}_P = \emptyset } \label{eq:parameter95partition} \end{align}\tag{4}\] where the three subsets correspond to the vision encoder, the language backbone, and the cross-modal projector. In the MLLM forward pass, the vision encoder extracts visual representations \(\mathbf{h}_V\), which are mapped by the cross-modal projector into the language embedding space as projected tokens \(\hat{\mathbf{h}}_V\). Specifically, the projected tokens are concatenated with text token embeddings \(\mathbf{h}_T\) and jointly processed by the language backbone to produce the output logits. Since we employ Low-Rank Adaptation (LoRA) [31] for parameter-efficient fine-tuning (PEFT), \(\mathbf{w}_V\), \(\mathbf{w}_L\), and \(\mathbf{w}_P\) refer specifically to the trainable LoRA adapter parameters within each module, while the dense pre-trained backbone weights are kept frozen. The task and replay contributions to 3 are: \[\begin{align} \mathcal{L}_{\text{task}}^{(j)}(\mathbf{w}) &= \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}_k^{(j)}} [\ell(f(\mathbf{x}; \mathbf{w}), y)], \tag{5} \\ \mathcal{L}_{\text{replay}}^{(j-1)}(\mathbf{w}) &= \mathbb{E}_{(\hat{\mathbf{h}}_V, \hat{\mathbf{h}}_T, \hat{y}) \sim \mathcal{B}_k^{(j-1)}} [\ell(f_{\mathrm{emb}}(\hat{\mathbf{h}}_V, \hat{\mathbf{h}}_T; \mathbf{w}), \hat{y})], \tag{6} \end{align}\] Here \(f(\mathbf{x};\mathbf{w})\) is the standard MLLM forward on raw image-text inputs, while \(f_{\mathrm{emb}}(\hat{\mathbf{h}}_V, \hat{\mathbf{h}}_T;\mathbf{w})\) injects the generated embeddings directly at the visual-token and text-prompt level, bypassing the input encoder; \(\gamma\) weights the replay contribution.

4.2 Modality-Aware Elastic Weight Consolidation↩︎

A primary challenge in applying continual learning techniques to MLLMs is the complex and modular nature of their architecture. Knowledge is distributed across a vision encoder, a language model, and a cross-modal projector that aligns the two modalities. Standard regularization methods like Elastic Weight Consolidation (EWC) [13] treat all model parameters uniformly. A uniform penalty is suboptimal because it overlooks that the importance of parameters can vary significantly across different modalities. Forgetting might occur in the visual understanding part while the linguistic part remains stable, or vice-versa.

To address this, we introduce Modality-Aware EWC (MA-EWC), which applies a more granular regularization by treating each core module of the MLLM independently. The total set of trainable parameters \(\mathbf{w}\) is partitioned into three disjoint subsets: vision parameters \(\mathbf{w}_V\), language parameters \(\mathbf{w}_L\), and cross-modal projector parameters \(\mathbf{w}_P\). For each subset, client \(k\) computes a separate diagonal Fisher Information Matrix (FIM) at the end of learning task \(\mathcal{T}_{j-1}\). The FIM serves as a proxy for parameter importance, with larger diagonal values indicating parameters that are more critical for performance on past tasks. The client-specific FIM for a parameter set \(\mathbf{w}_M\) (where \(M \in \{V, L, P\}\)) is approximated as: \[\begin{align} {!}{\displaystyle \mathbf{F}_{M,k}^{(j-1)} = \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}_k^{(j-1)}} \left[ \left( \nabla_{\mathbf{w}_M} \log p(y|\mathbf{x}; \mathbf{w}_M^{(j-1)*}) \right)^2 \right] }, \label{eq:fim} \end{align}\tag{7}\] At the element level, the diagonal approximation is accumulated sample-wise as \[\begin{align} {!}{\displaystyle F_{M,k,i}^{(j-1)} \approx \frac{1}{|\mathcal{D}_k^{(j-1)}|} \sum_{(\mathbf{x}, y) \in \mathcal{D}_k^{(j-1)}} \left(\frac{\partial \log p(y \mid \mathbf{x}; \mathbf{w}_M^{(j-1)*})}{\partial w_{M,i}}\right)^2 }, \label{eq:fim95element} \end{align}\tag{8}\] where \(\mathbf{w}_M^{(j-1)*}\) are the optimal parameters for module \(M\) after task \(\mathcal{T}_{j-1}\). In the federated setting, each client \(k\) independently computes and stores its own modular FIM \(\mathbf{F}_{M,k}^{(j-1)}\) over its local dataset \(\mathcal{D}_k^{(j-1)}\) and retains the corresponding reference parameters \(\mathbf{w}_{M,k}^{(j-1)*}\), consistent with the per-client notation used in 2. The MA-EWC regularization loss is then the sum of the penalties for each module: \[\begin{align} {!}{\displaystyle \mathcal{L}_{\text{MA-EWC},k}^{(j-1)}(\mathbf{w}) = \frac{1}{2} \sum_{M \in \{V, L, P\}} \sum_{i \in \mathbf{w}_M} F_{M,k,i}^{(j-1)} (w_i - w_i^{(j-1)*})^2 }, \label{eq:ma95ewc95loss} \end{align}\tag{9}\] which can be rewritten as the sum of modality-specific penalties: \[\begin{align} {!}{\displaystyle \mathcal{L}_{\text{MA-EWC},k}^{(j-1)}(\mathbf{w}) = \mathcal{L}_{V,k}^{(j-1)}(\mathbf{w}_V) + \mathcal{L}_{L,k}^{(j-1)}(\mathbf{w}_L) + \mathcal{L}_{P,k}^{(j-1)}(\mathbf{w}_P) }, \label{eq:maewc95decomposition} \end{align}\tag{10}\] where \[\begin{align} \mathcal{L}_{M,k}^{(j-1)}(\mathbf{w}_M) = \frac{1}{2}\sum_{i \in \mathbf{w}_M} F_{M,k,i}^{(j-1)} (w_i - w_i^{(j-1)*})^2. \label{eq:module95ewc} \end{align}\tag{11}\] where \(F_{M,k,i}^{(j-1)}\) is the \(i\)-th diagonal element of the client-specific FIM for module \(M\). By separating the FIM computation, MA-EWC can, for example, heavily penalize changes to critical vision parameters if the new task is visually similar to a past one, while allowing more flexibility in the language parameters if the new task introduces novel textual concepts. A modality-specific approach thus provides a more effective safeguard against the nuanced forms of forgetting.

4.3 Privacy-Preserving Federated Replay↩︎

Rehearsal, or replaying exemplars from past tasks, is one of the most effective strategies for mitigating catastrophic forgetting. However, in the federated setting, storing and replaying raw data from previous tasks is infeasible due to privacy constraints and storage limitations on client devices. To overcome this, we propose Privacy-Preserving Federated Replay, a mechanism that leverages client-side generative models to create synthetic embedding-level replay tuples while avoiding raw-data exchange. After a client \(k\) completes its training on task \(\mathcal{T}_j\), it uses its local dataset \(\mathcal{D}_k^{(j)}\) to train a lightweight, local replay generator \(G_k^{(j)}\). Importantly, PPFR does not attempt to generate raw image pixels or free-form text tokens. Instead, it operates in the frozen MLLM representation space and synthesizes multimodal replay tuples consisting of a visual-token embedding, a text-prompt prototype embedding, and a replay label. In our implementation, \(G_k^{(j)}\) is a compact latent-noise module driven by Gaussian codes \(\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\), a 64-dimensional latent space, and an event-conditioned code. A two-layer visual decoder maps the latent code to the MiniCPM visual-token embedding space, while a prompt-prototype table with a small residual MLP produces the corresponding textual embedding. The replay label is obtained from the local event label during fitting and stored as a teacher-smoothed soft target.

The generator is trained locally for 5 fitting epochs at the task boundary with a combination of embedding reconstruction, image-text alignment, teacher KL, and cross-entropy losses. When the next task \(\mathcal{T}_{j+1}\) is introduced, client \(k\) samples \(G_k^{(j)}\) to synthesize a small buffer of replay tuples \(\mathcal{B}_k^{(j)} = \{(\hat{\mathbf{h}}_{V,i}, \hat{\mathbf{h}}_{T,i}, \hat{y}_i)\}_{i=1}^{|\mathcal{B}|}\). Since \(G_k^{(j)}\), the generated replay buffer, and the Fisher statistics remain local, PPFR avoids raw-data exchange, although it is not intended to provide a formal differential-privacy guarantee. The generated buffer \(\mathcal{B}_k^{(j)}\) is then interleaved with the new task’s data \(\mathcal{D}_k^{(j+1)}\) for local training. Including these synthetic embedding-level samples in the training process, as reflected in 3 , allows the model to rehearse prior event knowledge while avoiding storage or transmission of raw multimodal posts. We formalize the replay synthesis process as \[\begin{align} {!}{\displaystyle \mathcal{B}_k^{(j)} = \left\{(\hat{\mathbf{h}}_{V,i}, \hat{\mathbf{h}}_{T,i}, \hat{y}_i)\;\middle|\;(\hat{\mathbf{h}}_{V,i}, \hat{\mathbf{h}}_{T,i}, \hat{y}_i) \sim G_k^{(j)}(\mathbf{z}_i, c_i),\;\mathbf{z}_i \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \right\}_{i=1}^{|\mathcal{B}|} }, \label{eq:replay95buffer} \end{align}\tag{12}\] where \(c_i\) is the sampled local event code used to condition replay generation. We define the replay-augmented dataset by \[\begin{align} \tilde{\mathcal{D}}_k^{(j+1)} = \mathcal{D}_k^{(j+1)} \cup \mathcal{B}_k^{(j)}. \label{eq:augmented95dataset} \end{align}\tag{13}\]

4.4 Task-Similarity-Aware Gradient Aggregation↩︎

In federated continual learning, client drift is exacerbated because the statistical heterogeneity across clients is compounded by temporal heterogeneity. At any given time, different clients might be training on different tasks or on data that represents different facets of the same task, leading to local updates that may conflict with each other. A naive federated averaging of these updates can lead to destructive interference, harming the global model’s performance on both old and new tasks. To address this, we introduce TSGA, a server-side mechanism to modulate the aggregation process. The core idea is to identify and down-weight client updates that are outliers, as they are likely to be either off-topic for the current learning objective or of low quality. After receiving the parameter updates \(\Delta \mathbf{w}_k = \mathbf{w}_k - \mathbf{w}_{\text{g}}\) from each participating client \(k\) in a given round, the server first computes the average update direction: \[\begin{align} \bar{\mathbf{g}} = \frac{1}{|\mathcal{S}|} \sum_{k \in \mathcal{S}} \frac{\Delta \mathbf{w}_k}{\|\Delta \mathbf{w}_k\|}, \label{eq:avg95grad} \end{align}\tag{14}\] where \(\mathcal{S}\) is the set of participating clients. The server then measures the alignment of each client’s update with this average direction using cosine similarity: \[\begin{align} s_k = \frac{\Delta \mathbf{w}_k \cdot \bar{\mathbf{g}}}{\|\Delta \mathbf{w}_k\| \|\bar{\mathbf{g}}\|}. \label{eq:cosine95sim} \end{align}\tag{15}\] Clients whose updates are poorly aligned with the consensus direction (i.e., have a low similarity score \(s_k\)) are likely to be sources of interference. We introduce a similarity threshold \(\tau\). Any client with \(s_k < \tau\) is considered an outlier and is excluded from the current round. The filtered participant set is \[\begin{align} \mathcal{S}' = \{k \in \mathcal{S} \mid s_k \ge \tau\}, \label{eq:filtered95clients} \end{align}\tag{16}\] and, if this set is empty, the round falls back to \(\mathcal{S}'=\mathcal{S}\) to preserve a well-defined aggregation step even under highly conflicting two-client rounds. The filtered set is then used with normalized aggregation weights \[\begin{align} \alpha_k = \frac{N_k}{\sum_{i \in \mathcal{S}'} N_i}, \quad k \in \mathcal{S}', \label{eq:aggregation95weight} \end{align}\tag{17}\] and the final global model is updated using a weighted average over only the clients in \(\mathcal{S}'\): \[\begin{align} \mathbf{w}_{\text{g}} \leftarrow \mathbf{w}_{\text{g}} + \sum_{k \in \mathcal{S}'} \alpha_k \Delta \mathbf{w}_k. \label{eq:tsga95aggregation} \end{align}\tag{18}\] By filtering out potentially disruptive updates, TSGA ensures a more stable and coherent learning trajectory for the global model, improving its ability to consolidate knowledge from diverse clients in a sequential learning environment. The complete FedCMM workflow is summarized in 2 and 3. The first algorithm captures the task-level server coordination, and the second details the client-side rehearsal and local update procedure.

5 Experiments↩︎

Experiments were conducted on two federated continual social-event benchmarks in which every task is an image-text classification problem. The evaluation follows chronological event streams so that the model must preserve earlier event knowledge while adapting to later multimodal distributions.

5.1 Datasets and Evaluation Metrics↩︎

Datasets and Metrics. We use PHEME [35] and CrisisMMD [4], two social-media event datasets that support image-text event understanding. PHEME contains rumor and breaking-news discussions collected from social media, with 6,273 text posts and 2,089 associated images across seven events. CrisisMMD contains crisis-related tweets from natural disasters, with 16,097 text instances and 18,126 images annotated for emergency-response categories. Following chronological event-incremental evaluation, events are ordered by time and grouped into three tasks with a \([2,2,3]\) class split for each dataset, yielding \(T=3\) tasks per stream. All datasets are split into train/validation/test partitions with a 70/10/20 ratio. Unless otherwise noted, client data is partitioned across 10 clients with a Dirichlet label-skew distribution using concentration \(\alpha=0.5\); we additionally report the severe non-IID setting \(\alpha=0.1\). Let \(A_{i,j}\) be the accuracy on task \(\mathcal{T}_j\) after training up to task \(\mathcal{T}_i\). We adopt three standard continual learning metrics: i) Average Accuracy (Acc), computed as \(\frac{1}{T} \sum_{j=1}^T A_{T,j}\); ii) Backward Transfer (BWT), computed as \(\frac{1}{T-1} \sum_{j=1}^{T-1} (A_{T,j} - A_{j,j})\) to quantify forgetting; and iii) Forward Transfer (FWT), computed as \(\frac{1}{T-1} \sum_{j=2}^{T} A_{j-1,j}\). Higher Acc and FWT are better, and BWT is better when closer to zero.

5.2 Baseline Methods and Implementation Details↩︎

We compare FedCMM against a comprehensive set of baselines: Joint Training (upper bound with all task data), Fine-tuning (sequential lower bound), FedAvg-CL [5], FedProx-CL [26], LwF-FL [25], EWC-FL [13], FedWeIT [24], and GEM-FL [15]. GEM-FL stores raw replay exemplars and is therefore treated as a privacy-violating rehearsal reference. We also include AF-FCL [27], a generative-replay method based on accurate forgetting, and TEKNet-FL [36], a federated adaptation of the temporal event knowledge learner. All experiments were implemented using PyTorch and Hugging Face Transformers. The base MLLM was openbmb/MiniCPM-V-2.6-int4. We employed PEFT with LoRA [31] using rank \(r=8\) and scaling factor 8. Each dataset is trained as a 3-task stream, with 50 federated rounds per task, 10 total clients, and 5 sampled clients per round unless otherwise stated. Local training uses paged AdamW with learning rate \(2 \times 10^{-5}\), cosine decay, per-device batch size 1, and 16 gradient-accumulation steps. Each selected client performs one local epoch per federated round. PPFR fits only the local replay generator for 5 epochs at each task boundary, using a 64-dimensional Gaussian latent code and an embedding-level replay budget of \(|\mathcal{B}|=100\). The main hyperparameters are \(\lambda=500\) for MA-EWC and \(\tau=0.5\) for TSGA, selected by the sensitivity analysis. Experiments were conducted on NVIDIA L60 GPUs.

5.3 Main Results↩︎

1 reports the comparison under moderate and severe heterogeneity. On PHEME with \(\alpha=0.5\), FedCMM reaches 86.3% Acc, improving over TEKNet-FL by 1.7% and over AF-FCL by 2.2%. The gain is not limited to the final event group: BWT improves from -10.2% for TEKNet-FL to -8.7%, indicating less damage to earlier events. Under the severe \(\alpha=0.1\) split, all methods lose accuracy, but the ranking is not a simple shifted copy of the moderate case: GEM-FL has slightly higher Acc than AF-FCL, whereas AF-FCL has stronger FWT. FedCMM retains a 1.1% Acc lead and keeps BWT 0.4% closer to zero than TEKNet-FL. On CrisisMMD, FedCMM obtains 84.4% and 79.6% under \(\alpha=0.5\) and \(\alpha=0.1\), respectively, below the non-continual Joint Training upper bound but above the strongest continual baselines. Several close comparisons are dataset-dependent; for example, AF-FCL is slightly stronger than TEKNet-FL in severe CrisisMMD Acc, while TEKNet-FL gives better FWT on severe CrisisMMD and moderate PHEME, reflecting a trade-off between forward plasticity and forgetting resistance. GEM-FL is competitive because raw exemplars provide direct rehearsal, yet it stores original samples and is not a privacy-preserving federated option. The gaps are therefore not uniform across datasets or metrics: CrisisMMD benefits more from visually grounded evidence, while PHEME depends more on event-specific textual shifts and stance cues. Across both streams, the strongest improvements appear on BWT, suggesting that the proposed modules mainly reduce forgetting while preserving enough plasticity for later crisis and rumor events.

5.4 Ablation Studies↩︎

To validate each component’s contribution, we conducted ablation studies on PHEME under \(\alpha=0.5\). 2 removes each component individually, and 3 analyzes the MA-EWC design.

Contribution of each component. As shown in 2, FedAvg-CL reaches 73.8% Acc with a BWT of -22.7%, indicating severe forgetting under event-incremental PHEME. Adding all three components increases Acc to 86.3% and improves BWT to -8.7%. The component effects are not identical across metrics. Removing PPFR causes the largest retention loss, reducing Acc to 82.4% and worsening BWT to -13.6%, which indicates that embedding replay is the main anchor for old-event decision boundaries. Removing MA-EWC gives 83.7% Acc and weaker FWT, suggesting that replay alone cannot fully stabilize modality-specific adapters. Removing TSGA gives 84.5% Acc; its BWT is slightly better than removing MA-EWC, but the full method still improves both retention and transfer. Notably, the nonparallel pattern across metrics is expected because replay, consolidation, and aggregation filtering affect different failure modes. The ablation pattern also shows that no single component fully explains the final performance: replay improves retention, MA-EWC constrains module drift, and TSGA reduces server-side interference.

Analysis of MA-EWC. 3 shows that the Fisher penalty must respect the MLLM module structure. Removing EWC entirely gives 82.8% Acc and -12.4% BWT. A uniform EWC penalty improves this to 84.3%, but still trails the full modality-aware design. Single-branch protection is unstable and metric-dependent: vision-only protection reaches 83.1% Acc but gives relatively high FWT, while language-only protection gives higher Acc and BWT but lower FWT. Notably, visual stabilization appears to help early adaptation to new events, whereas language-side protection better preserves accumulated rumor cues. The full model separately protects the vision encoder, language backbone, and projector, reaching 86.3% and the best BWT of -8.7%. In turn, the evidence supports the claim that forgetting in social-event MLLM adaptation is distributed across visual, textual, and cross-modal alignment modules. The separate Fisher design is therefore not only a stronger regularizer, but also a better match to how multimodal adapters store task-specific evidence.

5.5 Parameter Sensitivity and Robustness↩︎

Sensitivity to hyperparameters. 4 shows sensitivity analyses for four key hyperparameters. The curves are not perfectly symmetric because the two benchmarks fail in different ways: PHEME is dominated by cross-event semantic shift, while CrisisMMD is more affected by noisy image-text alignment and class imbalance. For MA-EWC strength \(\lambda\), both datasets peak at \(\lambda=500\), but the decline after the peak is sharper on CrisisMMD. Increasing \(\lambda\) from 500 to 2000 reduces CrisisMMD from 84.4% to 80.3%, whereas PHEME decreases from 86.3% to 84.2%. In particular, over-constraining adapters can suppress crisis-event adaptation even when it protects old tasks. For the TSGA threshold \(\tau\), 0.5 gives the best result, but the neighboring points are uneven: \(\tau=0.3\) is more competitive on CrisisMMD (83.1%), whereas PHEME is less sensitive at \(\tau=0.7\). The replay buffer trend is also nonuniform. Moving from no replay to \(|\mathcal{B}|=100\) improves PHEME by 3.9% and CrisisMMD by 4.6%, but the intermediate gains differ across datasets, and increasing to 150 does not help. Consequently, low-quality synthetic tuples can begin to dilute current-task supervision. Finally, sampling \(C=5\) clients per round is best, while the \(C=3\) and \(C=10\) gaps differ across datasets. In aggregate, the pattern across all four parameters indicates that the default configuration is not an isolated lucky point; it lies in a stable region where consolidation, replay, and aggregation filtering remain balanced.

Robustness to data heterogeneity. 5 confirms that FedCMM remains more stable as the client label distribution becomes skewed. On PHEME, FedAvg-CL decreases from 80.6% at \(\alpha=1.0\) to 69.4% at \(\alpha=0.1\), while FedCMM decreases from 89.7% to 81.8%. Notably, the gap is not constant: it is 9.1% under the mild split, 12.5% at \(\alpha=0.5\), and 12.4% under severe heterogeneity. On CrisisMMD, FedAvg-CL falls from 74.0% to 65.6%, whereas FedCMM falls from 86.1% to 79.6%. A related asymmetry appears in the degradation pattern: the moderate-to-severe drop is larger than the mild-to-moderate drop, especially for FedAvg-CL on CrisisMMD. Disaster events appear to have more heterogeneous visual evidence across clients. Importantly, the proposed method does not eliminate non-IID sensitivity; rather, it reduces the additional forgetting caused by skewed client updates. In turn, the results support the role of TSGA as a stabilizer rather than as a substitute for better client coverage. The remaining drop at \(\alpha=0.1\) also shows that the evaluation is not saturated, leaving visible room for stronger personalization or adaptive client sampling.

Robustness to task ordering. 4 evaluates desynchronized client progression on PHEME, where 30% of clients lag one task behind at each boundary. Notably, the setting is harsher than ordinary label skew because stale clients optimize an earlier event distribution while synchronized clients have already moved to the next group. FedAvg-CL drops from 73.8% to 65.7%, and EWC-FL drops from 78.6% to 71.4%. FedCMM drops from 86.3% to 82.4%, losing less than 4%. A smaller degradation is consistent with the interaction between TSGA and PPFR: stale updates tend to have lower agreement with the round consensus, and replay keeps the retained clients from over-specializing to only the latest event. Beyond a simple heterogeneity test, the asynchronous setting specifically probes whether aggregation can tolerate clients being at different points in the continual stream. Moreover, the property is important for social-event deployments, where clients may receive delayed event evidence or skip event groups entirely.

5.6 Local-Epoch Sensitivity and Efficiency Analysis↩︎

Effect of local epoch number \(E\). 6 shows average accuracy as \(E\) increases from 1 to 4. All methods peak at \(E=1\), confirming that aggressive local training amplifies client drift in temporally skewed social-event streams. On PHEME, FedAvg-CL drops from 73.8% to 71.2%, EWC-FL drops from 78.6% to 75.0%, and FedCMM drops from 86.3% to 83.5%. On CrisisMMD, the absolute decline is larger for the baselines but not by a fixed margin: FedAvg-CL loses 10.8%, EWC-FL loses 9.3%, and FedCMM loses 5.45%. The intermediate points also differ; for example, FedAvg-CL on PHEME is relatively flat from \(E=2\) to \(E=3\) before dropping at \(E=4\), whereas all methods on CrisisMMD show a steeper and more consistent decline. In particular, the nonparallel degradation shows that FedCMM is not merely an upward-shifted baseline. Replay and aggregation filtering reduce the damage caused by longer client-side optimization, but they do not make large local epochs harmless. For this reason, the default setting uses one local epoch for the MLLM update, while PPFR’s five fitting epochs apply only to the small replay generator at task boundaries. The result also suggests that increasing local computation is not a reliable substitute for communication in federated continual MLLM adaptation.

Convergence efficiency. 5 reports the number of federated rounds needed to first reach 50% and 75% accuracy, and the final accuracy at round 50, under \(\alpha=0.5\). At the 50% milestone, FedCMM converges in 8 rounds on PHEME and 10 rounds on CrisisMMD, achieving speedups of approximately 2.25\(\times\) over FedAvg-CL (18 and 23 rounds) and 1.6\(\times\) over EWC-FL (13 and 15 rounds). At the more demanding 75% target, FedCMM requires 13 and 19 rounds, while EWC-FL needs 28 and 33 rounds, roughly twice as many; FedAvg-CL cannot reach 75% within the 50-round budget on either benchmark, saturating below that threshold. At the final checkpoint (round 50), FedCMM achieves 86.3% on PHEME and 84.4% on CrisisMMD, exceeding EWC-FL by 7.7% and 7.3%, respectively.

5.7 Discussion↩︎

Superior performance of FedCMM across both benchmarks stems from three complementary mechanisms: i) Modality-aware regularization via MA-EWC applies module-specific Fisher penalties to the vision encoder, language backbone, and cross-modal projector independently, preventing the uniform parameter shrinkage that causes a single-branch EWC to under-protect whichever modality updates most aggressively in the current task; ii) Privacy-preserving replay via PPFR restores sample-space supervision without exposing raw client data, and the empirical gap between PPFR and standard exemplar replay (GEM-FL) confirms that synthetic embeddings preserve enough decision geometry to achieve comparable retention; and iii) Similarity-aware aggregation via TSGA filters out client updates that are directionally misaligned with the consensus, reducing the amplification of task-ordering drift that ordinary FedAvg cannot suppress. Nevertheless, several limitations warrant attention: optimal \(\lambda\) and \(\tau\) values are currently determined by grid search rather than adaptive scheduling; the Fisher diagonal approximation may underestimate parameter importance in cross-modal projector layers where interference is most nonlinear; and scaling to \(T\!\geq\!6\) tasks has not been validated. Future work will pursue learnable hyperparameter schedules [10], more expressive Fisher surrogates for low-rank adapters, and extended evaluation on the seven-task PHEME\(\rightarrow\)CrisisMMD combined stream [11], [36].

6 Conclusion↩︎

In this paper, we have addressed catastrophic forgetting in the federated fine-tuning of Multimodal Large Language Models on evolving data streams. Our proposed FedCMM framework synergistically integrates modality-aware parameter regularization, raw-data-free embedding-level replay, and similarity-aware gradient aggregation, enabling MLLMs to learn sequentially without sacrificing prior knowledge. Extensive experiments show that FedCMM consistently achieves the best average accuracy and BWT, while remaining competitive on FWT. The framework’s robustness to data heterogeneity and asynchronous task ordering further highlights its practical utility for real-world networked AI deployments. Future work will focus on improving efficiency and autonomous hyperparameter adaptation, advancing toward evolutive optimization in large-scale distributed AI systems.

References↩︎