1 Introduction↩︎

Multimodal representation models have become the foundation of modern retrieval systems by learning a shared embedding space across modalities such as images, text, and audio. Models such as CLIP [1] and AudioCLIP [2] enable cross-modal retrieval and transfer learning by aligning heterogeneous inputs into a common representation. While these models are typically trained on large static datasets, many real-world applications require continual adaptation as new concepts, domains, or data distributions emerge over time. A practical multimodal retrieval system should therefore be able to incorporate new information without destroying the cross-modal relationships learned from previous data.

Continual adaptation is particularly challenging for multimodal representation learning because forgetting affects not only individual modality encoders but also the geometry of the shared embedding space [3], [4]. Small changes introduced while learning new tasks can alter the relative positions of representations across multiple modalities, degrading retrieval performance in different and often asymmetric ways. For example, adapting to improve audio–text retrieval may simultaneously weaken image–text or image–audio alignment. As a result, continual multimodal retrieval requires preserving a globally consistent embedding space while remaining sufficiently flexible to learn from new data.

Existing continual learning methods address catastrophic forgetting through mechanisms such as regularization, knowledge distillation, or replay [5]–[7]. Although these approaches improve the balance between stability and plasticity, they still produce one final checkpoint. Since these approaches impose different constraints, their final checkpoints can preserve different parts of the multimodal embedding geometry. Selecting a single checkpoint therefore forces all retrieval directions to inherit one stability–plasticity trade-off. This motivates the central question of whether checkpoints obtained from different continual-learning strategies can be composed after training so that the resulting model learns a tensor-specific balance between retention and adaptation.

To address this question, we propose AlphaWiSE, a post-hoc cweight-space interpolation designed to compose two frozen continual-learning checkpoints into a single retrieval model through learned tensor-level interpolation. The normal sequential checkpoint is produced by sequential fine-tuning, and the companion continual-learning endpoint is produced by EWC, LwF, or iCaRL. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. This tensor-wise parameterization is more expressive than a fixed global interpolation coefficient while keeping coefficient fitting low-dimensional. In our AudioCLIP ViT-B/32 backbone, AlphaWiSE optimizes 499 coefficient logits: 152 for image-encoder tensors, 149 for text-encoder tensors, 195 for audio-encoder tensors, and three learned scalar logit-scale parameters, one for each modality pair. The two source checkpoints, each with approximately 182M parameters, remain frozen throughout coefficient fitting.

We evaluate AlphaWiSE in an AudioCLIP-based continual retrieval setting using the AudioSet dataset under a constrained-memory regime with 840 exemplars [8]. This setting tests whether post-hoc weight-space fusion can improve multimodal continual learning when only an exemplar set is available across sequential phases [9]–[11]. Performance is measured on audio–image–text retrieval across continual-learning phases, with retrieval quality reported using R@1 and mAP [2].

Our contributions are threefold:

• We formulate continual multimodal checkpoint selection as post-hoc weight-space interpolation between a sequential fine-tuning checkpoint and a companion continual-learning endpoint.

• We introduce tensor-wise interpolation, which fits one scalar coefficient per aligned parameter tensor on a smaller exemplar memory while the two source checkpoints remain frozen.

• We demonstrate that the materialized interpolated checkpoint consistently outperforms both individual continual-learning baselines and standard weight-interpolation methods on continual multimodal retrieval, while preserving the architecture, parameter count, and inference cost of a single backbone.

2 Related Work↩︎

Continual learning. Continual learning studies how to adapt models to sequentially arriving data while mitigating catastrophic forgetting [10], [12]–[23], liu2023online?. Existing methods can be broadly categorized into regularization-based and replay-based approaches. Regularization-based methods [6], [24]–[28], including Elastic Weight Consolidation (EWC) [29] and Learning without Forgetting (LwF) [9], preserve previously acquired knowledge by constraining parameter updates or distilling predictions from earlier models. Replay-based methods [7], [10], [14], [30]–[35], including iCaRL [7] and experience replay, retain or revisit samples from previous tasks to stabilize sequential optimization. Despite their different optimization strategies, each method returns a single checkpoint, committing the deployed model to one particular stability–plasticity trade-off.

Continual multimodal representation learning. Recent advances in multimodal representation learning have led to shared-embedding models such as CLIP and AudioCLIP [2], which align heterogeneous modalities within a common embedding space for cross-modal retrieval. Continual adaptation is particularly challenging in these models because forgetting affects not only unimodal representations but also the geometry of cross-modal alignment. Unlike classification, where forgetting primarily changes decision boundaries, retrieval depends on preserving globally consistent embedding neighborhoods across multiple modalities. Consequently, small representation shifts can significantly alter nearest-neighbor rankings and degrade retrieval performance.

Recent methods, therefore, extend continual learning directly to multimodal or vision-language representation models. C-CLIP combines parameter-efficient adaptation with contrastive knowledge consolidation for continual vision-language learning [36], [37], while Dynamic Adapter Routing (DAR) introduces prototype-guided adapter routing for continual multimodal retrieval [38], [39]. Other approaches similarly employ adapters, prompts, low-rank modules, or task-specific projections to reduce interference during continual adaptation [40]–[45]. Despite their architectural differences, these methods improve continual learning by introducing additional trainable components or modifying the optimization process.

Our work takes a complementary perspective. Instead of proposing another continual-learning algorithm, we assume multiple continual-learning solutions have already been obtained and investigate how to compose them into a stronger representation. AlphaWiSE performs post-hoc fusion of frozen continual-learning checkpoints, producing a single deployable model without adapters, routing mechanisms, or additional inference-time computation.

Weight-space interpolation and model merging. Weight-space interpolation has recently emerged as an effective approach for combining neural networks without increasing inference cost. Stochastic Weight Averaging (SWA) demonstrated that averaging checkpoints along an optimization trajectory improves generalization by converging to wider optima [46]–[48]. Building on this idea, Model Soups, Fisher-weighted averaging, and task arithmetic showed that independently fine-tuned models can often be merged directly in parameter space while maintaining or improving downstream performance [49]–[51]. WiSE-FT further demonstrated that interpolating a pretrained model with its fine-tuned counterpart improves robustness while preserving zero-shot capability [52].

Existing interpolation methods primarily target transfer learning or robustness by combining a pretrained model with a fine-tuned model using one or a few global interpolation coefficients. In contrast, AlphaWiSE addresses continual multimodal retrieval by learning a tensor-level interpolation rule between frozen continual-learning checkpoints using a small exemplar memory. The coefficient vector follows an ordered set of checkpoint keys, and the scalar associated with each key is applied to every entry of the corresponding aligned parameter tensor. This granularity keeps the post-hoc optimization compact while allowing different layers and modality-specific components to draw different proportions from the source checkpoints, producing a single fused model with unchanged inference cost.

3 Methods↩︎

3.0.0.1 Continual multimodal retrieval.

We consider a sequence of \(K\) training phases, where the data available at phase \(k\) consist of aligned audio–image–text samples \[\mathcal{D}_k = \{(a_i, x_i, t_i)\}_{i=1}^{N_k}.\] The model consists of modality-specific audio, image, and text encoders which project each modality into a shared embedding space. The image and text encoders are initialized from CLIP, while the audio encoder is a dedicated ResNeXt-based network trained jointly to project audio into the same embedding space [2]. The model is evaluated over a set of directed retrieval tasks \(\mathcal{P}\), such as audio-to-text, image-to-audio, and image-to-text retrieval [2]. For each direction \((m,n) \in \mathcal{P}\), an observation from modality \(m\) is used as the query, and candidates from modality \(n\) are ranked according to their embedding similarity [53], [54]. This setting differs from static multimodal retrieval in three respects: (i) the training data arrive sequentially across phases rather than being available jointly; (ii) after each phase, the model must incorporate the newly observed data while retaining cross-modal alignment acquired during earlier phases [3], [53]; and (iii) access to previous-phase data is restricted to a bounded exemplar memory \(\mathcal{M}\), preventing full replay of the training history [7], [10], [11].

Our method builds on the weight-space interpolation principle used in WiSE-FT [52]. AlphaWiSE is a post-hoc weight-space fusion procedure for two compatible checkpoints from the same architecture. During coefficient fitting, the checkpoint weights are fixed and only the interpolation coefficients are updated. Our starting observation is that no single continual learning method is uniformly best in multimodal retrieval: one strategy may better preserve certain modality relationships, while another may provide stronger adaptation to new phases. This suggests that useful information is distributed across distinct continual solutions rather than concentrated in a single final checkpoint. We denote the less constrained endpoint by \(\theta^{\mathrm{un}}\), such as a standard sequential fine-tuning checkpoint, and the more stability-preserving endpoint by \(\theta^{\mathrm{reg}}\), such as an EWC, LwF, or iCaRL checkpoint.

Let \((\kappa_1,\ldots,\kappa_P)\) denote the ordered checkpoint keys selected for interpolation. For each key \(\kappa_p\), let \(\theta_p^{\mathrm{un}}\) and \(\theta_p^{\mathrm{reg}}\) denote the corresponding endpoint tensors. The tensors under the same key must have identical shapes and represent the same model component. AlphaWiSE learns one scalar interpolation coefficient shared by all entries of each aligned parameter tensor.

\[\tilde{\theta}_p(\boldsymbol{\beta}) = \alpha_p\theta_p^{\mathrm{un}} + (1-\alpha_p)\theta_p^{\mathrm{reg}}, \qquad \alpha_p=\sigma(\beta_p), \label{eq:alphawise95fusion}\tag{1}\]

where \(\beta_p\in\mathbb{R}\) is unconstrained and the sigmoid keeps \(\alpha_p\in(0,1)\). We initialize \(\beta_p=0\) for all tensors, so every coefficient starts from \(\alpha_p=0.5\).

The coefficients are fitted on the exemplar memory \(\mathcal{M}\). For a minibatch \(\mathcal{B}=\{(a_i,x_i,t_i)\}_{i=1}^{B}\), the fused model produces audio, image, and text embeddings in the shared retrieval space. For a directed modality pair \((m,n)\), let \(s_{ij}^{m,n}(\boldsymbol{\beta})\) denote the temperature-scaled similarity between the \(m\)-embedding of example \(i\) and the \(n\)-embedding of example \(j\) under the fused checkpoint \(\tilde{\theta}(\boldsymbol{\beta})\). The directed InfoNCE loss is

\[\mathcal{L}_{m\rightarrow n}(\mathcal{B};\boldsymbol{\beta}) = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp(s_{ii}^{m,n}(\boldsymbol{\beta}))}{\sum_{j=1}^{B}\exp(s_{ij}^{m,n}(\boldsymbol{\beta}))}. \label{eq:directed95infonce}\tag{2}\]

Given a set \(\mathcal{P}\) of retrieval directions used for coefficient fitting, AlphaWiSE minimizes the empirical retrieval loss

\[\min_{\boldsymbol{\beta}\in\mathbb{R}^{P}} \mathbb{E}_{\mathcal{B}\sim\mathcal{M}} \left[ \mathcal{L}_{\mathrm{ret}}(\mathcal{B};\boldsymbol{\beta}) \right], \qquad \mathcal{L}_{\mathrm{ret}}(\mathcal{B};\boldsymbol{\beta}) = \frac{1}{|\mathcal{P}|} \sum_{(m,n)\in\mathcal{P}} \mathcal{L}_{m\rightarrow n}(\mathcal{B};\boldsymbol{\beta}). \label{eq:alphawise95objective}\tag{3}\]

The set \(\mathcal{P}\) can contain one direction for a pair-specific objective or multiple directions for the joint objective studied in Section 4.3. Gradients are computed through the fused model where \(\boldsymbol{\beta}\) is updated while \(\theta^{\mathrm{un}}\) and \(\theta^{\mathrm{reg}}\) stay fixed. The coefficient-gradient interpretation is given in Appendix 6.4.

4 Experiments↩︎

We evaluate AlphaWiSE in the main AudioCLIP-based continual retrieval setting using the AudioSet dataset under a constrained-memory regime with 840 exemplars [8]. This setting is designed to test whether post-hoc weight-space fusion can improve multimodal continual learning when only an exemplar set is available over a course of phases [9]–[11]. Performance is measured on audio-image-text retrieval across the continual learning phases, with retrieval quality reported using R@1 and mAP [2].

Our evaluation focuses on whether AlphaWiSE can improve retention and transfer relative to individual continual-learning trajectories, including standard fine-tuning, EWC, iCaRL, and LwF [7], [9], [29]. In this setting, each baseline represents a different stability–plasticity tradeoff: standard fine-tuning provides stronger adaptation to new data, while regularization- and distillation-based methods are designed to preserve prior behavior. AlphaWiSE uses the 840-exemplar memory to learn per-tensor interpolation coefficients between frozen checkpoints produced by these strategies, enabling the fused model to reuse complementary properties of different training trajectories without increasing inference-time model capacity.

This evaluation is intended to answer whether AlphaWiSE improves continual multimodal retrieval in the low-memory regime, and whether learned checkpoint fusion can provide a better balance between preserving prior cross-modal alignment and adapting to later phases than selecting any single continual-learning trajectory.

4.1 Implementation Details↩︎

4.1.0.1 Dataset

We conduct our experiments on AudioSet, a large-scale dataset of human-annotated audio events collected from YouTube videos. AudioSet contains 10-second clips annotated with 527 sound-event labels drawn from a hierarchical ontology. The classes cover a broad range of acoustic concepts, including human sounds, musical instruments, animals, vehicles, natural environments, and mechanical sounds. Individual clips may contain multiple labels, and the dataset is highly imbalanced, with substantially different numbers of examples across classes [8]. To construct the continual-learning benchmark, we partition the classes used in our experiments into eight disjoint phases. The model is trained sequentially on these phases, introducing a new subset of sound-event classes at each phase without revisiting the complete training data from earlier phases. This class-incremental organization allows us to evaluate both adaptation to newly introduced concepts and retention of previously learned audio-image-text alignment.

4.1.0.2 Backbone architecture.

For our implementation, we utilize the AudioCLIP backbone with some minor changes. The image and text branches are inherited from CLIP ViT-B/32 [1], with embedding dimension \(512\), patch size \(32\), transformer width \(768\), and \(12\) transformer layers, while the audio branch uses an ESResNeXt-FBSP convolutional encoder [2], [55]. The model is trained and evaluated using three pairwise retrieval objectives: audio–image, audio–text, and image–text. Additionally, because only the ESResNeXt-FBSP audio encoder contains BatchNorm layers, we replace each BatchNorm module in this branch with GroupNorm, while preserving its learned affine weight and bias parameters; the CLIP image and text encoders already use LayerNorm and are unaffected. This replacement removes running mean and variance buffers, making normalization independent of batch composition and avoiding inconsistencies that can arise when interpolated affine parameters are paired with BatchNorm statistics estimated under different continual-learning checkpoints [56]–[60].

4.1.0.3 Source checkpoints.

At each phase, AlphaWiSE merges two checkpoints with identical architectures and parameter structures. The less constrained checkpoint, denoted by \(\theta^{\mathrm{un}}\), is obtained through standard sequential fine-tuning and serves as the more plastic endpoint. It is optimized using SGD with learning rate \(5\times10^{-5}\), momentum \(0.9\), weight decay \(5\times10^{-4}\), and Nesterov momentum. The learning rate follows an exponential decay schedule with \(\gamma=0.96\) per epoch. AlphaWiSE uses the epoch-\(30\) checkpoint as the unconstrained endpoint [2].

The stability-preserving checkpoint, denoted by \(\theta^{\mathrm{reg}}\), is obtained using a continual-learning method such as EWC, LwF, or iCaRL. These methods preserve previous knowledge through parameter regularization, knowledge distillation, or exemplar replay, respectively, and therefore provide a more constrained endpoint than standard sequential fine-tuning. For the EWC instantiation, the model is optimized using AdamW, with a separately tuned learning rate for the logit-scale parameters. The diagonal Fisher information is estimated from \(64\) minibatches of the preceding phase, lower-bounded by \(\epsilon=10^{-3}\), and upper-bounded by \(10^{4}\). The corresponding objective is \[\mathcal{L}_{\mathrm{EWC}}(\theta) = \mathcal{L}_{\mathrm{CLIP}}(\theta) + \frac{\lambda_{\mathrm{ewc}}}{2} \sum_n F_n \left( \theta_n-\theta_n^{(k-1)} \right)^2,\] where \(F_n\) is the estimated diagonal Fisher importance of parameter \(n\), \(\theta^{(k-1)}\) denotes the parameters retained from the preceding phase, and \(\lambda_{\mathrm{ewc}}=0.8\) [29].

For LwF, a frozen copy of the preceding-phase checkpoint provides temperature-softened targets for the audio–image, audio–text, and image–text objectives. Let \(\mathcal{R}_{\mathrm{dist}}\) denote the distillation directions used during training. The loss is \[\mathcal{L}_{\mathrm{LwF}} = \mathcal{L}_{\mathrm{CLIP}} + \lambda_{\mathrm{LwF}}T^2 \frac{1}{|\mathcal{R}_{\mathrm{dist}}|} \sum_{r\in\mathcal{R}_{\mathrm{dist}}} \operatorname{CE}\!\left(q_r^{\mathrm{old}}(T),q_r(T)\right),\] where \(T=2\), \(\lambda_{\mathrm{LwF}}=0.1\), and \(\mathcal{R}_{\mathrm{dist}}\) is matched to the directed retrieval losses used for coefficient fitting [9]. For iCaRL, we use the same optimizer and combine distillation with replay from a fixed memory of \(840\) herding-selected exemplars [7]. Its objective is \[\mathcal{L}_{\mathrm{iCaRL}} = \mathcal{L}_{\mathrm{CLIP}} + \lambda_{\mathrm{iCaRL}} \operatorname{BCE}\!\left(\sigma(z^{\mathrm{old}}),z\right),\] with \(\lambda_{\mathrm{iCaRL}}=1.0\). Both \(\theta^{\mathrm{un}}\) and \(\theta^{\mathrm{reg}}\) remain frozen during coefficient optimization, and AlphaWiSE updates only the interpolation variables.

4.2 Continual-retrieval results↩︎

Table 1 reports the main 840-exemplar continual retrieval results across audio-to-text, image-to-audio, and image-to-text retrieval. Overall, AlphaWiSE improves over individual continual-learning baselines on most average retrieval metrics, indicating that post-hoc interpolation can recover a stronger multimodal representation than selecting a single continual-learning trajectory.

2.5pt

The strongest gains appear when AlphaWiSE combines the normal sequential checkpoint with a stability-preserving checkpoint. For audio-to-text retrieval, AlphaWiSE (N+EWC) achieves the best average performance, improving over EWC from 0.2900 to 0.3037 R@1 and from 0.3792 to 0.3946 mAP. AlphaWiSE (N+LwF), however, obtains the best last-phase audio-to-text R@1 and mAP. For image-to-audio retrieval, AlphaWiSE (N+EWC) obtains the strongest average and last-phase R@1, while AlphaWiSE (N+LwF) obtains the strongest average and last-phase mAP. These results indicate that the strongest source pairing depends on both the retrieval metric and the reporting stage.

For image-to-text retrieval, the best average performance is obtained by WiseFT(N + iCARL) and AlphaWiSE (N+iCaRL), followed by AlphaWiSE (N+LwF). This indicates that the most effective checkpoint pairing can depend on the retrieval direction. While EWC provides strong audio-related retention, other trajectories can better preserve or recover image-text alignment. This supports the central motivation of AlphaWiSE: different continual-learning methods encode different stability–plasticity tradeoffs, and these tradeoffs are not uniformly optimal across modality pairs. Rather than treating each baseline checkpoint as a final solution, AlphaWiSE uses them as reusable components for learned weight-space composition.

4.3 Cross-objective effects across retrieval directions↩︎

5pt

Table 2 studies whether the interpolation coefficients learned from one modality-pair objective affect other retrieval directions. In this ablation, AlphaWiSE is trained using one retrieval objective at a time, such as A\(\rightarrow\)T, I\(\rightarrow\)A, or I\(\rightarrow\)T, and the resulting fused model is evaluated on all retrieval tasks.

Joint optimization gives the best A\(\rightarrow\)T and I\(\rightarrow\)A results. In contrast, optimizing only A\(\rightarrow\)T gives the best I\(\rightarrow\)T result, reaching 0.2386 R@1 and 0.3223 mAP. Thus, the coefficient objective affects both the directly optimized direction and off-diagonal directions.

The results show that optimizing the coefficients on a single objective affects both the directly optimized direction and the off-objective directions. Training on A\(\rightarrow\)T gives the best I\(\rightarrow\)T result in this ablation but does not improve I\(\rightarrow\)A. Training on I\(\rightarrow\)A gives strong I\(\rightarrow\)A performance and remains competitive on A\(\rightarrow\)T. Overall, joint optimization gives the strongest audio-related performance, achieving the best A\(\rightarrow\)T and I\(\rightarrow\)A results in Table 2.

This pattern suggests that objectives involving the image modality may provide broader cross-modal benefits in this setting. Optimizing I\(\rightarrow\)A directly improves image–audio retrieval and remains competitive on A\(\rightarrow\)T, whereas optimizing A\(\rightarrow\)T primarily benefits audio–text and image–text retrieval. This behavior is consistent with prior work showing that audio can be aligned with pretrained image–text representation spaces and subsequently support retrieval across all three modalities [2], [61]. More directly, ImageBind demonstrates that aligning multiple modalities through images can induce emergent alignment between modality pairs that are not observed together during training [62]. Although AlphaWiSE differs substantially from ImageBind in both training regime and scale, our ablation suggests a related phenomenon: when the image modality is included in the interpolation objective, the learned interpolation can improve retrieval directions beyond the modality pair directly optimized.

4.4 Qualitative embedding analysis↩︎

Figures 4 and 5 provide qualitative t-SNE visualizations of the learned embedding space.

In the image-to-text visualization, AlphaWiSE produces tighter or more separated neighborhoods for several selected classes than the corresponding baseline, suggesting improved organization of the shared representation space for those examples. In the image-to-audio visualization, AlphaWiSE similarly shows more coherent cross-modal clustering for several selected neighborhoods, consistent with the strong I\(\rightarrow\)A results in Table 1. These qualitative results support the interpretation that AlphaWiSE improves retrieval by shifting the fused model toward a region of weight space that better preserves class-level cross-modal alignment.

4.5 Analysis of learned interpolation↩︎

Since AlphaWiSE learns one interpolation coefficient per parameter group, the converged \(\alpha\) values reveal where in the network the merge draws on the fine-tuned model and where it preserves the stable continual model. Figure 6 disaggregates the learned coefficients along two axes: relative block depth within each encoder, and parameter type (attention weights, MLP/convolutional weights, normalization scales and biases, and remaining bias vectors).

Two observations emerge. First, parameter type, not depth, is the dominant axis of variation: within every panel, the separation between types (up to \(\Delta\alpha \approx 0.8\)) far exceeds any trend across blocks, supporting the choice of per-named model parameter rather than per-layer or per-model coefficients. Second, normalization and bias parameters absorb a disproportionate share of adaptation whenever the weights remain conservative. For example as exemplified in Figure 6, when the stable model is trained with EWC, the image- and text-tower attention and MLP weights converge to \(\alpha \approx 0.1\), i.e.they are taken almost entirely from the stable model, while LayerNorm scales and biases vary with a range of \(\Delta\alpha \approx 0.3\) .

This pattern is also seen with the AlphaWiSE models with iCARL and Lwf. This mirrors prior evidence that norms and biases constitute a cheap, high-leverage channel for domain adaptation [63], [64]. Additionally, the results in Table 1 support a view of AlphaWiSE as a mechanism for post-hoc trajectory composition. Standard fine-tuning, EWC, LwF, and replay-based methods each produce checkpoints with different retrieval strengths. EWC is particularly useful for audio-related retrieval, while other pairings are more effective for image-text retrieval. AlphaWiSE exploits this complementarity by learning per-tensor interpolation coefficients on a small exemplar memory.

The cross-objective ablation further suggests that the interpolation coefficients are sensitive to the optimization objective. Joint optimization gives the best A\(\rightarrow\)T and I\(\rightarrow\)A results, whereas A\(\rightarrow\)T-only optimization gives the best I\(\rightarrow\)T result. This indicates that different modality-pair objectives favor different per-tensor mixtures, and it motivates future work on retrieval-conditioned or objective-aware interpolation strategies.

Overall, the results indicate that AlphaWiSE is most useful when the fused checkpoints provide complementary solutions. The method does not require a single continual-learning baseline to dominate across all retrieval directions. Instead, it benefits from the fact that different baselines preserve different parts of the multimodal embedding geometry.

5 Conclusion↩︎

We presented AlphaWiSE, a parameter-efficient framework for continual multimodal representation learning that adaptively fuses frozen continual-learning checkpoints through per-tensor weight interpolation. Rather than committing to a single continual-learning trajectory, AlphaWiSE treats multiple continual-learning solutions as complementary building blocks and learns how to combine them using a small exemplar memory, producing a single fused model with the same architecture and inference cost as the original backbone. Experiments on continual audio–image–text retrieval demonstrate consistent improvements over strong continual-learning baselines, showing that adaptive checkpoint composition provides a better balance between adaptation and retention than individual continual-learning strategies alone. More broadly, our results suggest that continual-learning trajectories should not be viewed as competing alternatives from which a single model must be selected, but rather as complementary sources of knowledge that can be composed to obtain stronger multimodal representations. We hope this perspective motivates future work on scalable checkpoint composition, interpolation across multiple continual-learning trajectories, and more general model fusion techniques for continually adapting multimodal foundation models.

Acknowledgements↩︎

This research is supported by the National Artificial Intelligence Research Resource Pilot Awards NAIRR250199, NAIRR260019, and NAIRR260077, the AMD University Program’s AI & HPC Cluster, NVIDIA Academic Grant Program, and Lambda’s Research Grant. Computational resources are also provided by Delta and DeltaAI at the National Center for Supercomputing Applications through ACCESS allocations CIS250012, CIS250816, and CIS251188.

6 Pair-Specific Interpretations↩︎

6.1 Normal + EWC↩︎

Let \(\theta^{N}\) denote the checkpoint obtained by normal sequential training, and let \(\theta^{E}\) denote the checkpoint obtained by EWC. At a given phase, normal training approximately optimizes the current-phase objective \[J_N(\theta) = \mathcal{L}_{\mathrm{new}}(\theta),\] whereas EWC approximately optimizes \[J_E(\theta) = \mathcal{L}_{\mathrm{new}}(\theta) + \lambda_E \Omega_E(\theta),\] where \(\Omega_E\) penalizes movement in parameters that are important for previous phases. In this pairing, \(\theta^{E}\) is the more regularized checkpoint, while \(\theta^{N}\) is the less constrained checkpoint.

AlphaWiSE performs interpolation at the level of named parameter tensors. Let \(p\) index a named parameter tensor. For Normal+EWC, AlphaWiSE defines the fused tensor as \[\tilde{\theta}_{p} = \alpha_{p}\theta_{p}^{N} + (1-\alpha_{p})\theta_{p}^{E}.\] Equivalently, \[\tilde{\theta}_{p} = \theta_{p}^{E} + \alpha_{p} \left( \theta_{p}^{N} - \theta_{p}^{E} \right).\]

Since \(\alpha_p=\sigma(\beta_p)\in(0,1)\) for every finite \(\beta_p\), the implemented parameterization does not attain either endpoint exactly at a finite coefficient logit. Instead,

\[\lim_{\beta_p\to-\infty}\tilde{\theta}_p=\theta_p^{E}, \qquad \lim_{\beta_p\to+\infty}\tilde{\theta}_p=\theta_p^{N}.\]

Algebraically extending \(\alpha_p\) to the closed interval \([0,1]\) would recover the two endpoints at \(\alpha_p=0\) and \(\alpha_p=1\).

Thus, \(\alpha_p\) controls movement from the regularized EWC solution toward the more plastic normal sequential solution.

For the exemplar retrieval loss, the coefficient gradient is

\[\frac{\partial \mathcal{L}_{\mathrm{ret}}}{\partial \alpha_p} = \left\langle \nabla_{\tilde{\theta}_p}\mathcal{L}_{\mathrm{ret}}, \theta_p^{N}-\theta_p^{E} \right\rangle_F, \qquad \langle A,B\rangle_F = \sum_{\mathbf{i}} A_{\mathbf{i}}B_{\mathbf{i}}.\]

Under gradient descent, \(\alpha_p\) increases when the inner product above is negative, meaning that movement from the EWC checkpoint toward the normal checkpoint is a local descent direction. It decreases when the inner product is positive. In this sense, AlphaWiSE learns a per-tensor effective EWC preference after continual training where smaller values of \(\alpha_p\) retain more of the EWC-stabilized solution, while larger values move toward the more plastic normal checkpoint.

A per-tensor EWC bound requires the quadratic penalty to decompose across the same tensor partition. Vectorizing each parameter tensor, suppose \[\Omega_E(\theta) = \sum_{p=1}^{P}\Omega_{E,p}(\theta_p), \qquad \Omega_{E,p}(\theta_p) = \frac{1}{2} (\theta_p-\theta_{\mathrm{old},p})^{\top} F_p (\theta_p-\theta_{\mathrm{old},p}), \quad F_p\succeq 0.\] Then each \(\Omega_{E,p}\) is convex. For the per-tensor interpolation above, \[\Omega_E\bigl(\tilde{\theta}(\boldsymbol{\alpha})\bigr) \leq \sum_{p=1}^{P} \left[ \alpha_p\Omega_{E,p}(\theta_p^{N}) + (1-\alpha_p)\Omega_{E,p}(\theta_p^{E}) \right].\] This inequality applies when the EWC matrix has no cross-tensor blocks with respect to the chosen tensor partition. If cross-tensor couplings are present, the bound does not follow from convexity because the per-tensor mixture is not a single convex combination of the two complete checkpoints.

6.2 Normal + LwF↩︎

Let \(\theta^{N}\) denote the normal sequential checkpoint, and let \(\theta^{\mathrm{LwF}}\) denote the checkpoint obtained by Learning without Forgetting (LwF). Normal training approximately optimizes \[J_N(\theta) = \mathcal{L}_{\mathrm{new}}(\theta),\] whereas LwF approximately optimizes \[J_{\mathrm{LwF}}(\theta) = \mathcal{L}_{\mathrm{new}}(\theta) + \lambda_{\mathrm{LwF}} D_{\mathrm{old}}(\theta),\] where \(D_{\mathrm{old}}\) is a distillation loss that encourages the current model to preserve the predictions or similarities of an earlier model on inputs available during the current phase. In this pairing, \(\theta^{\mathrm{LwF}}\) is the more regularized checkpoint, while \(\theta^{N}\) is the less constrained checkpoint.

For Normal+LwF, AlphaWiSE defines the fused parameter tensor as \[\tilde{\theta}_{p} = \alpha_{p}\theta_{p}^{N} + (1-\alpha_{p})\theta_{p}^{\mathrm{LwF}}.\] Equivalently, \[\tilde{\theta}_{p} = \theta_{p}^{\mathrm{LwF}} + \alpha_{p} \left( \theta_{p}^{N} - \theta_{p}^{\mathrm{LwF}} \right).\] Under this convention, \(\alpha_p=0\) recovers the LwF checkpoint for parameter tensor \(p\), while \(\alpha_p=1\) recovers the normal checkpoint.

The coefficient gradient is \[\frac{\partial \mathcal{L}_{\mathrm{ex}}}{\partial \alpha_{p}} = \left( \nabla_{\tilde{\theta}_{p}} \mathcal{L}_{\mathrm{ex}}(\tilde{\theta}) \right)^{\top} \left( \theta_{p}^{N} - \theta_{p}^{\mathrm{LwF}} \right).\] Under gradient descent, \(\alpha_p\) increases when the inner product above is negative, meaning that movement from the LwF checkpoint toward the normal checkpoint is a local descent direction. It decreases when the inner product is positive.

This gives a direct interpretation of Normal+LwF AlphaWiSE: rather than choosing between pure plasticity and distillation-based preservation, the method learns how much of the distillation-biased checkpoint to retain for each named parameter tensor. Smaller values of \(\alpha_p\) preserve more of the LwF solution, while larger values move toward the normal sequential checkpoint.

6.3 Cross-objective transfer in AlphaWiSE↩︎

6.3.0.1 Cross-objective transfer in coefficient-optimization space.

Table 2 studies whether optimizing the coefficients for one retrieval objective can affect another retrieval direction. Since AlphaWiSE updates \(\boldsymbol{\beta}\) rather than \(\boldsymbol{\alpha}\) directly, define

\[L_i(\boldsymbol{\beta}) = \mathcal{L}_i\!\left( \tilde{\theta}(\boldsymbol{\beta}) \right), \qquad \tilde{\theta}_p(\boldsymbol{\beta}) = \theta_p^{\mathrm{reg}} + \sigma(\beta_p) \left( \theta_p^{\mathrm{un}}-\theta_p^{\mathrm{reg}} \right).\]

for retrieval objective \(i\). A gradient-descent step on objective \(i\) is \[\boldsymbol{\beta}' = \boldsymbol{\beta} - \eta\nabla_{\boldsymbol{\beta}}L_i(\boldsymbol{\beta}).\] A first-order Taylor expansion of objective \(j\) gives \[L_j(\boldsymbol{\beta}') \approx L_j(\boldsymbol{\beta}) - \eta \left\langle \nabla_{\boldsymbol{\beta}}L_j(\boldsymbol{\beta}), \nabla_{\boldsymbol{\beta}}L_i(\boldsymbol{\beta}) \right\rangle.\] For a sufficiently small step size, objective \(j\) decreases locally when the two gradients have a positive inner product. This is a local first-order interpretation, which means that Table 2 does not directly measure gradient alignment or establish a causal transfer mechanism.

6.4 Coefficient-gradient interpretation↩︎

The coefficient gradient provides a local interpretation of the AlphaWiSE update. For tensor \(p\), \[\frac{\partial\mathcal{L}_{\mathrm{ret}}}{\partial\beta_p} = \alpha_p(1-\alpha_p) \left\langle \nabla_{\tilde{\theta}_p}\mathcal{L}_{\mathrm{ret}}, \theta_p^{\mathrm{un}}-\theta_p^{\mathrm{reg}} \right\rangle, \label{eq:alphawise95beta95gradient}\tag{4}\] where the inner product is over all entries of tensor \(p\). Thus, each coefficient is updated according to the local directional derivative of the exemplar retrieval loss along the difference between the two endpoint tensors.

References↩︎