1 Introduction↩︎

Multi-modal contrastive learning aims to jointly process and align data from diverse modalities such as images, text, audio, and video. This paradigm has demonstrated high performance across a wide range of tasks, including image captioning [1]–[3], visual question answering [4]–[6], zero-shot classification [7]–[9], and cross-modal retrieval [10]–[12]. These successes underscore the benefits of integrating heterogeneous modalities, which lead to improved generalization and more robust semantic representations. However, it remains unclear to what extent memorization contributes to the observed improvements, motivating the need to better understand the role of memorization in multi-modal contrastive learning.

Previous studies have shown that in both supervised learning (SL) [13], [14] and self-supervised learning (SSL) [15], model’s memorization of training data points is essential for generalization. In the vision domain, it has been observed that models tend to memorize outliers in the training set, which correspond to mislabeled samples in SL [13], [16] and atypical examples in SSL [17]. A similar study of memorization in multi-modal contrastive learning is lacking.

Multi-modal learning introduces unique challenges not seen in uni-modal settings [18]–[20], such as inconsistencies between modalities, varying noise levels, and the lack of explicit modality-alignment. These factors limit the direct transfer of insights from uni-modal studies to multi-modal learning. Importantly, most existing definitions of memorization are designed for specific modalities and tasks, for example, based on label prediction in SL [13] or augmentation matching in SSL [17] for the vision domain. Such definitions do not generalize well to the multi-modal setting: Measuring memorization within individual modalities or among a limited subset of all the modalities used to train a given multi-modal model, fails to capture multi-modal memorization faithfully. Therefore, a new definition tailored to multi-modal contrastive learning is needed.

There are only two studies on measuring memorization for the bi-modal image-text models like CLIP [7], namely déjà vu memorization [21] and CLIPMem [22]. The déjà vu method measures memorization by masking one modality and testing whether the model can recover the other, while CLIPMem measures memorization by comparing alignment between model pairs trained with or without a given sample (i.e., an image-text pair). However, these methods are not directly applicable to multi-modal contrastive learning which involves additional modalities, such as video and audio.

To address these limitations, we propose MultiMem: a general-purpose metric for measuring memorization in multi-modal contrastive learning, which is designed for any number and type of modalities involved. Specifically, MultiMem builds on the leave-one-out framework [13], [17], [23] for measuring memorization and compares the outputs of a pair of multi-modal models trained with and without a given multi-modal sample on this particular sample.

Through our extensive empirical study on multiple contrastively trained multi-modal models such as AudioCLIP [24] (including Audio, Image, and Text modalities), AVT-CLIP (a custom-built tri-modal model with Audio, Video, and Text modalities, introduced in Section 4.1.0.1), and AVIT-CLIP (a custom-built quad-modal model with Audio, Video, Image, and Text modalities, introduced in Section 4.1.0.1), we observe that: (1) Global memorization, measured across all modalities, behaves differently from memorization observed within any subset of modalities and to capture the full extend of a model’s memorization, we need to assess memorization jointly across all its modalities. (2) The most memorized samples are not simply mis-captioned, as reported for the most memorized samples in text-image models [22], but semantically misaligned across all modalities: the information provided by different modalities not only contradict each other but is mostly semantically not related. (3) Multi-modal models increasingly memorize cross-modal patterns, rather than primarily text as observed in bi-modal models [22]. This suggests that multi-modal models’ memorization behavior is closer aligned with the one of SSL models [17] which were shown to memorize pattern, rather than SL models which memorize labels [13].

Based on the above observations, we explore different strategies to mitigate memorization and improve generalization in multi-modal models. In-training: we actively measure memorization during multi-modal training to identify the top-memorized samples. At a given stage of training, we re-group the top-memorized samples into new batches and continuously apply noise-based augmentations (only) to them in subsequent training steps. Post-training: we identify highly memorized samples post-training and then fine-tune the model on the remaining training samples. Our experiments show that both approaches substantially reduce memorization by up to 20%, while increasing model performance up to 8% for retrieval, 10% for zero-shot, and 4% for downstream classification tasks. In summary:

• We propose MultiMem, a metric that measures memorization by comparing the outputs of a pair of multi-modal models trained with and without a given multi-modal sample on this particular sample

• Our analysis with MultiMemdemonstrates the key differences between multi-modal models with more than two modalities and previously studied bi-modal or uni-modal models, including distinct memorization patterns, a shift from label-driven to pattern-driven memorization behavior, and a stronger influence of cross-modal semantic inconsistency on memorization.

• Based on our findings, we propose two approaches to mitigate memorization and improve generalization in multi-modal models: either during training or after training. Our extensive experiments show that both methods effectively reduce the model’s memorization level and lead to substantial improvements in performance.

2 Background and Related Work↩︎

2.0.0.1 Multi-Modal Contrastive Learning.

Contrastive Language-Image Pretraining (CLIP) [7] proposed a joint training framework for image and text modalities, where text and image representations are aligned using the InfoNCE loss [25]. CLIP enables strong zero-shot image classification by using language prompts as class representations, and supports accurate image-text retrieval through a shared representation space. VideoCLIP [26] extends CLIP to the video domain by substituting the original image encoder with a transformer-based video encoder. The video is first uniformly sampled into a fixed number of frames, which are individually processed by a frozen convolutional neural network (CNN) with a trainable multi-layer perceptron (MLP) to obtain frame-level representations. These frame-level representations are then encoded as tokens by a transformer and aggregated by applying average pooling to obtain the representation for the entire video. This modification enables the model to process temporal-visual information while retaining a contrastive training framework. VideoCLIP shows strong performance in zero-shot video understanding tasks, such as action recognition and video-text retrieval, without task-specific supervision. AudioCLIP [24] extends CLIP to the audio domain by introducing an additional audio encoder based on an ESResNeXt convolutional network [27]. The audio input is first converted into a log-mel spectrogram and then projected into a shared representation space, where it is aligned with image and text representations. The model is trained using the same contrastive objective as CLIP, encouraging alignment between semantically related audio, image, and text samples, while pushing apart mismatched pairs. As the model incorporates one more modality than CLIP, the training loss is computed as the sum of the pairwise InfoNCE losses (i.e.,Audio-Image, Audio-Text, and Image-Text). AudioCLIP performs well in zero-shot audio classification and cross-modal retrieval tasks, outperforming previous methods that rely on task-specific supervision.

Memorization in SL & SSL. Although memorization has been associated with potential risks of sensitive information leakage, multiple studies [13], [17], [28] in supervised learning (SL) and self-supervised learning (SSL) suggest that memorization also plays an important role in the generalization of models. They also show that both SL and SSL models tend to memorize atypical samples during training, while the nature of these atypical samples varies. For example, in SL, memorization often correlates with mislabeled or noisy samples, while in SSL, it tends to occur on images with rare or distinctive visual patterns [15], [29].

A common definition of label memorization in SL is the leave-one-out style definition by [14]: \[\label{equ:aligment95multi}\begin{align} \boldsymbol{Mem}\left( \mathcal{A}, S, i\right) = \underset{f\leftarrow \mathcal{A}(S)}{\boldsymbol{Pr}}\left[ f\left( x_i\right) = y_i\right] \\ - \underset{g\leftarrow \mathcal{A}(S^{\setminus i})}{\boldsymbol{Pr}}\left[ g\left( x_i\right) = y_i\right] \text{,} \end{align}\qquad{(1)}\] where \(\mathcal{A}\) is a training algorithm (here for models \(f\) and \(g\)) and \(S^{\setminus i}\) represents train set \(S\) without the data point \((x_i,y_i)\). In this definition, a data point is considered memorized if the model’s prediction of the point’s ground truth training label changes significantly based on whether the point was or was not used to train the model.

For SSL, where no labels are available, [17] proposed a new metric: \[\label{eq:memdef} \begin{align} \boldsymbol{SSLMem}(S,Aug,i) = \underset{ \substack{g \sim \mathcal{A}(S^ {\setminus {i}})\\ {x_i}',{x_i}''\sim \text{Aug}({x_i})}}{\mathbb{E}} \left[ d\left( g\left({x_i}' \right), g\left({x_i}'' \right)\right)\right]\\ - \underset{ \substack{ f \sim \mathcal{A} (S)\\ {x_i}',{x_i}''\sim \text{Aug}({x_i})}} {\mathbb{E}} \left[ d\left( f\left({x_i}' \right), f\left({x_i}'' \right)\right)\right] \text{,} \end{align}\tag{1}\] where \(d(\cdot , \cdot)\) represents a distance function, commonly the \(\ell_2\) distance, and \(\text{Aug}(\cdot)\) is the augmentation sets used during training. Here, the expectation captures the alignment of the respective models as the expected Euclidean distance between their representations of two augmentations of the same sample. Then, SSLMem is computed as the alignment difference between the model pairs trained with and without that sample. Both metrics are limited to uni-modal models, making them unsuitable for evaluating memorization effects in multi-modal settings.

Memorization in Bi-modal Contrastive Models. [21] proposed a retrieval-based evaluation protocol to detect déjà vu memorization in vision-language models. Their method assesses whether a model trained on one data subset can retrieve objects that appeared only in a different, held-out subset, thereby indicating unintended memorization across disjoint training sets. However, these approaches only offer qualitative evaluations of whether memorization occurs, without providing a quantitative measurement of how strongly a model memorizes specific data points across all modalities.

The only existing method for quantifying memorization across the text and vision modalities is CLIPMem [22]. It computes the difference in alignment scores between a pair of CLIP-style models, trained with and without this data point. Formally, CLIPMem is defined as: \[\text{\boldsymbol{CLIPMem\@}}(I, T) = \mathcal{A}_{\text{align}}(f, I, T) - \mathcal{A}_{\text{align}}(g, I, T) \text{,}\quad\] where \(f\) and \(g\) are CLIP models trained on datasets with and without a data point \((I,T)\), consisting of an Image and Text (caption). The alignment score \(\mathcal{A}_{\text{align}}\) is computed as the cosine similarity between image and text representations, corrected by subtracting similarities to unrelated text and image samples \((i,t)\) that were not used in training \(f\) or \(g\).

\[\begin{align} \mathcal{A}_{\text{align}}(f, I, T) = \underset{(I',T') \sim \text{Aug}(I,T)}{\mathbb{E}} \left[\text{sim}(f_{\text{img}}(I'), f_{\text{txt}}(T'))\right] \\ - \underset{t}{\mathbb{E}} \left[ \text{sim}(f_{\text{img}}(I), f_{\text{txt}}(t)) \right] - \underset{i}{\mathbb{E}} \left[ \text{sim}(f_{\text{img}}(i), f_{\text{txt}}(T)) \right] \text{,} \end{align}\] where \((i,t)\) is a set of randomly chosen image and text testing samples that were not used in training \(f\) or \(g\). CLIPMem indicates that samples where caption and corresponding image do not align well (so-called mis-captioned samples) are most memorized. While CLIPMem is limited to measuring bi-modal memorization between Image and Text modalities, our MultiMem metric generalizes it to capture memorization among many more diverse modalities.

3 Our MultiMem Metric↩︎

The memorization metrics discussed in the previous section are tailored to uni-modal or bi-modal models and do not account for global interactions among all modalities in a multi-modal setting. We demonstrate empirically in 1 that these approaches are insufficient for faithfully capturing memorization in multi-modal models. Therefore, we introduce our MultiMem metric, which is specifically designed to quantify global memorization across all modalities.

We build MultiMem on the leave-one-out framework [13], [17], [22] to measure memorization with respect to a pair of models, \(f\) and \(g\). Here, model \(f\) is trained on the full training set \(S\), while model \(g\) is trained on the subset \(S^{\setminus i}\) where the multi-modal sample \(x_i\) was removed.

Given the complex interactions among modalities in multi-modal contrastive learning, we first require a proxy that quantifies the quality of the model’s representations of an input data point \(x_i\) which can be compared between models. Since the learning objective in multi-modal contrastive learning is to make the representations returned by the model on the different modalities of the same sample as consistent as possible, and the representations of unrelated examples as far as possible, we consider cross‑modal consistency (\(CMC\)) on the sample itself vs. on unrelated samples as a proxy. We calculate the cross-modal consistency as follows: Given a model with \(n\) different modalities, we define the representation space \(\Phi\) as: \[\label{equ:phi} \begin{align} \Phi_{x_i} = \begin{bmatrix} \hat{\phi}_1\\ \hat{\phi}_2 \\ \vdots\\ \hat{\phi}_n \end{bmatrix} \in \mathbb{R}^{n \times d}\text{,} \end{align}\tag{2}\] where \(\phi_j \in \mathbb{R}^d\) is the representation of \(x_i\) in the \(j\)-th modality, and \(\hat{\phi}_j\) denotes its \(\ell_2\)-normalized version.

Then we define cross-modal consistency \(CMC(i, H)\) on data point \(x_i\) (with respect to a held-out set \(H\)) as: \[\label{uesjhqoc} \begin{align} CMC(i, H) = \frac{1}{2}\underset{(\cdot)'\sim \text{Aug}}{\mathbb{E}}\mathbf{1}_{n}^{\top} \left( {\Phi_{x_{i}^{'}}}{\Phi^{\top}_{x_{i}^{'}}} \right) \mathbf{1}_{n} \\ - \frac{1}{2}\underset{^{\;\;\;h \in H}_{(\cdot)'\sim \text{Aug}}}{\mathbb{E}}\mathbf{1}_{n}^{\top} \left( {\Phi_{x_{i}^{'}}}{\Phi^{\top}_{{h}'}} \right) \mathbf{1}_{n}\text{,} \end{align}\tag{3}\] where \(\mathbf{1}_n\) denotes an all-ones vector of length \(n\) and \((\cdot)'\sim \text{Aug}\) indicates that we compute an expectation over representations computed on different random augmentations of the samples, increasing stability of the metric. The held-out set are randomly picked from non-trained samples in the validation set of training datasets.

The first term of the score captures the similarities across all modality pairs within the representation space of \(x_i\). The second term measures the distribution differences across all modality pairs between the representation space of \(x_i\) and held-out samples \(h \in H\). Subtracting these terms yields a score that is high when the modalities of \(x_i\) are strongly aligned with each other, and weakly aligned with unrelated examples, modeling the model’s intended objective.

Finally, we define the MultiMem of data point \(x_i\) as the \(CMC\) difference between model \(f\) and model \(g\): \[\label{equ:clipmem95multi} \begin{align} \text{\boldsymbol{MultiMem\@}}(i, H, f) = \underset{f \sim S}{CMC}(i, H) - \underset{g \sim S ^ {\setminus i}}{CMC}(i, H)\text{.} \end{align}\tag{4}\]

Unlike prior approaches that focus on pairwise similarities between modalities, this resulting metric leverages the entire distribution of representations across all modalities. This provides a principled way to evaluate global memorization in multi-modal models, capturing interactions beyond single modality pairs.

4 Evaluating Multi-Modal Memorization↩︎

We first describe the setup for our experiments and then analyze memorization in multi-modal contrastive learning using our MultiMem.

4.1 Experimental Setup↩︎

4.1.0.1 Models and Datasets.

We run our experiments on the following models: OpenCLIP [30], AudioCLIP [24], VideoCLIP [26], and our custom-built AVT-CLIP(Audio + Video + Text) and AVIT-CLIP(Audio + Video + Image + Text). The detailed encoder architectures and training datasets used are shown in 1 while other training details and hyperparameters are presented in 5 in 7.2.

4pt

4.1.0.2 Dataset Splitting.

Following [17], we divide each training dataset into three subsets: (1) a candidate set \(S_C\), which is used only for training model \(f\) and whose memorization we want to measure; (2) an independent set \(S_I\), which is used only for training model \(g\); and (3) a shared set \(S_S\), which is used for training both \(f\) and \(g\). \(S_C\) and \(S_I\) have an equal number of training samples to ensure that \(f\) and \(g\) are trained with the same number of data points. Detailed splits are provided in 4 in 7.2.0.2. We report the average MultiMem score on \(S_C\) as the memorization for model \(f\). Next, we discuss evaluation metrics.

4.1.0.3 Evaluation Metrics.

In prior work, retrieval tasks typically refer to settings where one modality is used to retrieve another, which we refer to as the uni-to-uni retrieval setting. To enable the measurement of models’ performance on more than two modalities, we introduce a multi-to-uni retrieval task for evaluation. In this task, representations from multiple source modalities are combined to retrieve data points from a target modality. Specifically, we compute a retrieval score by summing the pairwise cosine similarities between the target modality representation and each source modality representation. A higher retrieval score shows better semantic alignment between target and input modalities. A retrieval is considered successful if the ground-truth target sample appears among the top \(\boldsymbol{N}\) retrieved samples with the highest retrieval scores; otherwise, it is considered a failure. The overall retrieval success rate on the test set is used to evaluate the model’s performance. In the following experiments, we set \(\boldsymbol{N} = 5\) for both uni-to-uni and multi-to-uni retrieval tasks. The final retrieval accuracy is reported using \(\text{TOP}@\text{5}\) (\(T@5\)). Note that retrieval tasks are deterministic with no randomness in inference, so we only report the results once instead of an average with standard deviation. We further rely on linear probing and zero-shot classification tasks to assess model performance under our mitigations. For the zero-shot classification, we follow the settings introduced in the CLIP paper [7], where classification is performed by computing the similarity between label embeddings and the representations of the target modality.

4.2 Measuring Multi-modal Memorization↩︎

4.2.0.1 Memorization Distribution.

We study the memorization distribution of training samples on three multi-modal models: AudioCLIP, AVT-CLIP, and AVIT-CLIP. For AudioCLIP, we quantify the memorization level with three metrics: (1) tri-modal MultiMem with all three modalities, (2) the CLIPMem(based on Image-Text pair), and (3) bi-modal MultiMem restricted to the remaining modality pairs (Audio-Image and Audio-Text). We present our results in 1. Overall, MultiMem in 1 (a) is able to measure the highest level of memorization on \(S_C\) when compared with the baselines on the same AudioCLIP model. Additionally, it separates the scores for \(S_C\) and \(S_S\) significantly better, indicating a more sensitive measurement of memorization. We observe the same trends for AVT-CLIP and AVIT-CLIP, as we show in 7, 8, and 10 in 7.3.0.2. These findings suggest that in multi-modal models with more than two modalities, a memorization metric that jointly considers all modalities is necessary for a more accurate assessment of memorization.

4.2.0.2 Robustness to Held-out Set.

To examine the sensitivity of our MultiMem metric to the choice of held-out set \(H\), we evaluate the metric using several selection strategies for \(H\): (1) randomly sampled data points, (2) data points chosen to achieve balanced class representation, and (3) samples drawn from an entirely different dataset within the same domain. Furthermore, we investigate the robustness of the metric to the size of \(H\) by varying the number of data points included in the set. The results in 2 show that our MultiMem is robust to the choice of \(H\) in both size and composition.

4.2.0.3 Highly Memorized Samples.

We examine the highly memorized samples in VideoCLIP and AVT-CLIP to gain deeper insights into the underlying causes of strong memorization. For VideoCLIP, we find that, similar to the results reported in the work of [22], videos with misaligned captions and visual content (i.e., mis-captioned videos) tend to experience a higher level of memorization. However, in AVT-CLIP, we find that highly memorized samples are typically those with semantic misalignment across multiple modalities. These findings highlight the importance of evaluating cross-modal consistency rather than focusing only on text quality. Examples of the most memorized samples for VideoCLIP and AVT-CLIP are shown in 11 in 7.3.0.3 and a complete list of the top 10 most memorized samples (including their audio and video content and full captions) is provided in the supplementary material.

4.2.0.4 Memorization and Generalization.

To investigate the relationship between generalization and memorization in multi-modal models, we introduce Gaussian noise with different strengths into the representations of all modalities during training as augmentations to control the model’s memorization level, as proven successful by previous works [22], [35], [36]. The models’ generalization is evaluated by the \(T@5\) retrieval task introduced in Section 4.1.0.3. More specifically, we implement the experiments on VideoCLIP (bi-modal), AudioCLIP (tri-modal) and AVIT-CLIP(quad-modal) by injecting Gaussian noise with mean \(\mu=0\) and standard deviations \(\sigma =[0.025, 0.050, 0.075, 0.100, 0.125, 0.150, 0.175, 0.200]\).

In 2 (a), when the noise strength is lower than 0.1, we observe that a higher Video-Text MultiMem results in a worse downstream performance, and vice versa. This aligns with the results reported by [22] for bi-modal memorization in CLIP models. Moreover, our results in 2 (b) and 3 (c) show that AudioCLIP’s generalization (trained on three modalities) is negatively correlated with the tri-modal MultiMem. In contrast, there is no consistent correlation between generalization and memorization when applying CLIPMem or the bi-modal version of MultiMem, highlighting again the need to take all modalities into account when assessing multi-modal memorization. We also observe similar trends in AVIT-CLIP for 3 (d). Together, these results suggest that partial-modal memorization is insufficient to explain how memorization contributes to generalization in multi-modal models and highlight again the benefits of a truly multi-modal metric.

Finally, when the noise strength exceeds 0.1, the performance of both AudioCLIP and VideoCLIP begins to decline, while global memorization increases. This occurs because excessive noise destroy the semantic consistency between modalities. As a result, learning across modalities becomes more difficult, leading to increased global memorization and weakening the model’s generalization.

4.2.0.5 Impact of Augmentations.

We further study how applying augmentations (injecting noise) to different modality configurations (e.g., single modality, a subset of all modalities, and all modalities) during training influences the generalization of multi-modal models. We adopt SSLMem from [17] to measure memorization for single modality and use MultiMem for multi-modal memorization. The results in 2 on AudioCLIP suggest that applying augmentations to a single modality slightly increases the SSLMem of all modalities and enhances the model’s global generalization. However, it also leads to increased memorization in modality pairs that do not involve the augmented modality, which in turn degrades the performance of retrieval tasks based on that modality pair (as indicated by the underlined values in the tables). Moreover, applying augmentations to a subset of the modalities reduces both global and pairwise memorization, resulting in improved performance over all retrieval tasks. Notably, applying augmentations to all modalities always yields the best performance and the lowest memorization (as reflected by the bolded values in the tables), which provides a foundation for the MultiMem-based memorization mitigation strategies that we introduce in Section 5.

4.2.0.6 Memorization with ImageBind.

We train an additional AVIT model using the ImageBind-loss [37]. Unlike our setup, which performs pairwise alignment of all modalities, the ImageBind-loss takes the image representation from a pretrained image encoder as a reference and aligns all other modalities exclusively to this representation. During training, the image encoder remains frozen. Our results show that training with the ImageBind loss leads to increased global memorization (0.571 versus 0.488 achieved with our original AVIT) and reduced performance in retrieval task (36.5% versus 48.2% our AVIT model). These results suggest that training with a pairwise alignment across all modalities is beneficial as it reduces memorization and improves generalization. In 7.4, we present further details and an additional results on how the integration of more modalities during training improves generalization.

4.2.0.7 Balancing Per-Modality Memorization.

Given the differences in how each modality contributes to the overall memorization (see for example 2 for AudioCLIP), we further show that enforcing a balanced memorization distribution among modalities during training leads to a lower overall memorization and improved model performance. We apply \(\ell_1\)-normalized weights of \([0.332, 0.324, 0.344]\), which are negatively correlated to their memorization contribution \([0.222, 0.227, 0.214]\), to the three modality pairs (I-T, A–T, A–I) for AudioCLIP training. The new trained model achieves a lower MultiMem of 0.290 (compare to 0.332 of baseline model), a 3.2% improvement in retrieval task performance and a more balanced bi-modal MultiMem level of \([0.192, 0.195, 0.193]\) for three modality pairs (I-T, A–T, A–I). We present the full setup in 7.4.

4.2.0.8 Memorization Behavior in Multi-modal, SSL and SL.

We apply UnitMem [15] to analyze where inside the models memorization happens. We compare AudioCLIP (tri-modal), and compare it with CLIP (bi-modal), SL, and SSL models (uni-modal). All these models are trained with UrbanSound8K dataset + spectrogram images. A higher average UnitMem at a given layer indicates a greater contribution of that layer to the model’s global memorization.

We observe that, compared to CLIP, the vision encoder in AudioCLIP aligns more with SSL, rather than falling between SSL and SL as CLIP. This suggests that the introduction of the audio modality partially reduces the dominance of the text modality (e.g., with labels or captions) during training. Thereby, the memorization behaviors of the vision encoder shifts from being label-driven (in SL) or caption-driven (in CLIP), to pattern-driven as in SSL [17]. We hypothesize that this is because the learning signals in multi-modal models no longer come from a single supervision source (e.g., labels or captions). Instead, the model focuses more on the shared or similar semantic patterns across modalities such as audio, vision, and text. Therefore, the pattern underlying this shift is the strong semantic interdependence across modalities in multi-modal models.

5 Our Mitigation Strategies↩︎

Our findings from the previous section highlight that mitigating cross-modal memorization over all modalities can improve the generalization of multi-modal models. Based on this motivation, we design two methods built on MultiMem to mitigate high memorization, either during or post training.

5.0.0.1 In-Training Mitigation.

Instead of applying noise to the entire dataset, which harms overall performance and incurs additional computational overhead, we selectively add Gaussian noise (\(\mu=0\), \(\sigma=0.1\)) as augmentation only to the representations of the most memorized samples during training to mitigate memorization. Specifically, at every 10-epoch interval (i.e., epochs 10, 20, ..., 90), we use MultiMem to measure memorization for all training samples and select the top 5% most memorized samples. The 5% ratio is selected based on our experiments in 13 in 7.4.0.3, which show that this ratio yields the highest model performance and the lowest memorization level. We then aggregate these samples into new mini-batches and apply noise-based augmentations to their representations during the following training steps while keeping training unaltered for all other mini-batches.

In our experiment, we train AudioCLIP on the UrbanSound8K + Spectrogram dataset for 100 epochs with our mitigation in place and report performance on the retrieval task in 5 (a), the classification task on UrbanSound8K in 5 (b), and the zero-shot classification task on AudioSet [34] in 5 (c).

The results yield two main findings. First, applying our mitigation strategy at any training stage effectively reduces global memorization and improves overall performance across all three tasks. Second, model performance in all three tasks follows a trend of initial improvement, followed by a plateau, then further improvement, with a slight decline observed at epoch 90. We attribute this pattern to the timing of memorization mitigation. Applying mitigation too early in training may fail to accurately capture the most memorized samples, as the model is far from convergence. As a result, highly memorized examples that are not identified in the early stages may continue to increase the model’s memorization in later epochs, thereby harming generalization. Applying memorization mitigation too close to the end of training may lead to insufficient decoupling of modality correlation for highly memorized samples. This is the reason why the performance drop is observed at epoch 90 compared to epoch 80. These results indicate that applying noise-based augmentation near the end of training (e.g., at 80%) to a selected ratio of most memorized samples according to MultiMem can effectively reduce memorization and enhance generalization. In 7.4.0.5, we provide further insights into this strategy. (1) We compare the mitigation effects between using noise-based augmentation and gradient clipping (which is widely used in Differential Privacy area) for most memorized samples. (2) We compare it with an alternative approach that directly removes a fixed proportion of the most memorized samples during training. The results show that our strategy yields better performance improvements. (1) We also examine the effect of repeatedly applying this strategy at multiple stages of training. The findings indicate that repeated application leads to additional gains compared to single use. However, the improvement becomes marginal when applied more than twice.

5.0.0.2 Post-Training Mitigation.

Post-training, we first train the model and then use MultiMem to identify the most memorized samples. Then, we remove these samples from training and fine-tune the model for some additional steps on the remaining data. In our experiments, we train an AudioCLIP model on the UrbanSound8K dataset for 100 epochs, then, we use MultiMem to identify the most memorized samples and remove the top [50, 100, 150, 200, 300, 500, 700] of them. Finally, we fine-tune the model with the remaining dataset for an additional 25 epochs. For comparison, we additionally employ three alternative strategies for selecting samples to remove as baselines: using CLIPMem, computing the sum of cosine similarities across modality pairs, and randomly selecting samples.

We show the results in 5 (lower row) for retrieval, classification on UrbanSound8K, and zero-shot tasks accuracy on AudioSet. We observe that regardless of the method used to mitigate the model’s memorization (apart from random), the model’s performance improves to varying degrees when fewer than 500 samples are removed on all three tasks. Among them, MultiMem yields the best performance improvement, followed by the cosine similarity and CLIPMem. In contrast, the baseline of removing random samples does not show a clear trend of performance improvement. This highlights the effectiveness of MultiMem in post-training memorization mitigation and improving generalization.

5.0.0.3 Insights into Mitigation.

We further compared the average modality distance between the representations of each modality for the most memorized samples, as well as the average modality gap [18] across all samples in the training set, before and after mitigation. Results in 6 show that: (1) The model struggles to align semantic misaligned samples through generalization, thereby memorizes those datapoints so that results in an extremely low average sample modality distance. This shows that semantic misalignment is the key driver of memorization in multi-modal models. (2) Both of our mitigation methods increase the average modality distance of the most memorized samples thereby reducing the memorization level of the model. However, in-training mitigation performs better than post-training mitigation, and therefore yields a greater improvement in the generalization of the model. At the same time, in-training mitigation preserves the original average modality gap better. This is consistent with the conclusion in [18], that preserving the original modality gap tends to yield better model performance. These results show that our mitigation methods address the fundamental cause of memorization in multi-modal models, i.e., controlling the average modality distance for semantic misaligned samples.

6 Conclusions↩︎

We propose MultiMem, a novel metric for measuring and characterizing memorization in multi-modal contrastive learning with arbitrary modality configurations. Our results show clear differences compared to commonly studied bi-modal models like CLIP. We find that training with multiple modalities not only mitigates the single-modality dominance in global memorization observed in bi-modal models, but also makes the contrastive models’ memorization behavior more similar to that of self-supervised learning. Specifically, text modality, which has been viewed in previous work [22] as similar to labels, is no longer the only reason for high memorization when it does not align with other modalities. Inconsistency in semantic information between multiple modality pairs is instead the leading cause of high memorization. Finally, we show that our proposed MultiMem metric can be used to inform mitigations against memorization, either during training or after training, that improve generalization more efficiently than other metrics.

Acknowledgments↩︎

This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Project number 550224287. Franziska Boenisch received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (grant agreement No 101220235). Responsibility for the content of this publication lies with the authors.

7 Appendix↩︎

7.1 Hardware Usage & Calculation Overhead↩︎

Two devices are used for all experiments mentioned in this paper: a cloud server with four A100 GPUs and a local workstation with Intel 13700k CPU, Nvidia 4090 graphics card, a total of 10Tb storage space and 64GB of RAM.

Applying In-training mitigation strategy once during training will bring 0.70% (2.6 min out of 372 min) overhead in total time usage, without requiring any additional memory. Applying Post-training mitigation strategy will bring 0.55% (2.1 min out of 372 min) overhead in total time usage for memorization measurement, without requiring any additional memory.

Measuring MultiMem for all samples in candidate set \(S_C\) averagely brings 0.48% overhead in total time usage, without requiring any additional memory. Since the computational complexities of our MultiMem on both the number of modalities and the number of training samples are \(\text{O}(n^2)\), which matching the complexity of multi-modal contrastive learning. Therefore, the relative overhead remains essentially unchanged as modalities or data scale up.

7.2 Extended Experiment Setup↩︎

7.2.0.1 Glossary.

For the reader’s convenience, we provide a glossary with all important notation used in the main paper in 3.

7.2.0.2 Dataset Split.

4 provides a detailed summary of the splitting \(S_C\), \(S_I\), and \(S_S\) discussed in main paper.

7.2.0.3 Training Setup.

5 provide a detailed summary of the training configurations for all models used in our experiments. As stated in the main paper, all settings for existing models follow the default configurations of their respective original implementations.

7.2.0.4 AVT-CLIP and AVIT-CLIP.

For AVT-CLIP, the video and text modalities are first pre-trained for 20 epochs, after which the audio modality is introduced. The model is then trained jointly on all three modalities for an additional 100 epochs. For AVIT-CLIP, the image and text modalities are first pre-trained for 20 epochs, after which the video and audio modalities are introduced and trained jointly on all four modalities for an additional 200 epochs.

7.2.0.5 Experimental Setup for UnitMem.

In this experiment, we use the original UrbanSound8K dataset as the source of both audio inputs and text labels. Following the evaluation setup of AudioCLIP, we also use spectrograms of the audio signals as image inputs. In the SL setting, we employ a ViT encoder followed by a two-layer MLP classifier, trained with spectrogram images as inputs and text labels as supervision. For SSL settings, we employ the ViT-based DINO framework and train the ViT encoder on only spectrogram images. Both the image augmentation sets and hyperparameters follow the default configurations of their respective original implementations.

7.3 Extended Experiment Results↩︎

7.3.0.1 Measuring Memorization in AVT-CLIP and AVIT-CLIP.

For AVT-CLIP, we measure the memorization with two metrics: 1) tri-modal MultiMem with all three modalities and 2) bi-modal MultiMem of all modality pairs (A-V, A-T, and V-T). The results are shown in 7, which aligns with the results for AudioCLIP in the main paper. For AVIT-CLIP, we also measure the memorization with two metrics: 1) quad-modal MultiMem with all four modalities and 2) tri-modal MultiMem for VideoCLIP and AVT-CLIP(i.e., AVT MultiMem and AIT MultiMem). The results are shown in 8. We can find that when AVT-CLIP or AudioCLIP is extended to a quad-modal setting, the previously used tri-modal MultiMem becomes insufficient in capturing both the distribution of memorized samples and the accurate measurement of memorization levels. This again highlights the necessity of using all modalities to measure memorization.

7.3.0.2 Measuring Memorization AVIT-CLIP with synthesized images.

We use Stable Diffusion v1.5 to generate images from the captions of the MSRVTT dataset to construct a dataset with true four modalities: (1) audio, (2) video, (3) image, and (4) text. We adopt this setup based on the assumption that using newly generated images with styles different from the video frames can mitigate the potential unnecessary memorization caused by overly similar visual styles between images and video frames. Note that we synthesize the images for the MSRVTT dataset because there are no natural 4-modality datasets. A few examples of generated images are presented in 9.

The results are shown in 10, we find the memorization distribution fully follow the same trend observed in 8.

7.3.0.3 Top 10 most memorized samples for VideoCLIP and AVT-CLIP.

Unlike images, videos contain dynamic visual and audio signals, which cannot be effectively displayed in the paper. To address this, We provide two examples here and others are shown in supplementary materials. As shown in 11 (a), video3405 is a completely dark-screen video, providing almost no visual semantic information relevant to its caption: “a man talking about hydroponic fluid pressure.” As a result, it ranks first in VideoCLIP with a high MultiMem of 0.529. However, its audio clearly mentions “hydroponic fluid pressure”, which aligns well with the caption. Therefore, its MultiMem score in AVT-CLIP is only 0.561, ranking 36th. Another example is video6659 shown in 11 (b), where the visual content features two animated characters, while the audio contains a conversation between two male-like voices. The title, “a character hunting for love”, is poorly related to both the video and audio. All three modalities are semantically misaligned. As a result, this sample receives a high MultiMem score of 0.712 in AVT-CLIP, ranking second among all samples.

7.4 Additional Experiments↩︎

7.4.0.1 ImageBind Experiment Setup

We follow the original design of ImageBind that leverage the image embedding as the unified anchor space. All other modalities are aligned to this space through image-to-modality contrastive supervision. We keep the audio, video, and text encoders unchanged as random initialization, but replace the image encoder with our own pretrained image encoder. The encoder produces a normalized image representation that serves as the reference for all contrastive objectives. This design removes all non-image pairwise losses (e.g., audio–video, audio–text, and video-text), ensuring that the image modality forms the central representation space. We use Adam optimizer with a learning rate of 1e-4 and train for 200 epochs. The batch size and dataset split follow the implementation of AVIT-CLIP.

7.4.0.2 Enhancing Generalization by Introducing new Modalities.

Our results in 6 show that, when extending VideoCLIP to AVT-CLIP, the accuracy of V-T retrieval task increases. This is because adding a new modality to VideoCLIP reduces the text modality’s dominance in memorization, yielding a more balanced memorization distribution across modalities. So that increasing model’s generalization. This is consistent with our claims in Section 4.2.0.7. Similar trend could be observed in extending AVT-CLIP or AudioCLIP to AVIT-CLIP.

Note that the AVIT-CLIP always has the best performance in all retrieval tasks (as reflected by the boldface values in the 6).

7.4.0.3 Effects of Augmenting Different Ratios of Most Memorized Samples.

In 13, we test the effect of applying noise-based augmentation to different proportions of the most memorized samples during training. The results show that when noise-based augmentation is applied to less than 5% of the most memorized samples, global memorization decreases rapidly, and overall performance improves significantly. However, once the proportion exceeds 5%, both memorization and performance begin to stabilize as more samples are augmented. This indicates that decoupling the cross-modal associations of non-highly memorized samples is not effective in mitigating global memorization or improving performance. In contrast, applying augmentation only to the most memorized samples identified by MultiMem achieves the best results with significantly lower computational cost.

7.4.0.4 UnitMem results on COCO.

In this experiment, we use the original COCO dataset as the source of both image inputs and text captions. Speeches generated from COCO captions by open-source Tortoise TTS are used as audio modality. For SL, the model is trained using a multi-label classification setting with the cross-entropy loss with COCO dataset. All other settings for SSL, CLIP and AudioCLIP are similar to those in the experiments with UrbanSound8K dataset. We adopt this setup based on the assumption that captions and the audio generated from them share higher cross-modal semantic alignment. This helps mitigate the potential impact of weak semantic alignment among spectrograms, audio, and textual labels in the UrbanSound8K dataset. Note that the synthesized audio for COCO only used as extended evidence to verify the trend observed on the UrbanSound8K dataset in main paper.

The results are shown in 12. Despite semantic alignment across modalities being strengthened, we observe that the behavior of AudioCLIP’s vision encoder remains more similar to SSL than to SL. This fully aligns with the result of that implemented on UrbanSound8K dataset in main paper.

7.4.0.5 Mitigating Memorization During Training.

In addition to the experiments presented in Section “In-training Mitigation”, we implement three additional experiments to further validate the effectiveness of this approach.

In the first experiment, we compared the different effects of noise-based augmentation and gradient clipping during the in-training mitigation process. The results in 14 shows that gradient clipping needs a much larger learning rate than noise-injection to mitigate the memorization level of the model. Besides, the best performance of gradient clipping (41.7% +4.8% to 36.9% baseline) is worse than noise-injection(43.3% +6.7% to 36.9% baseline)

In the second experiment, we still use MultiMem to measure the memorization of all training samples at every 10-epoch interval. However, instead of regrouping the top 5% most memorized samples into new batches with noise-based augmentation, we directly remove them from the training set. As shown in 15, the performance of in-training removal first increases and then drops. This performance drop may be due to similar reasons observed in in-training regrouping,i.e., the learning rate becomes too low in the later stages of training, and the remaining number of epochs is insufficient to fully eliminate the negative impact introduced by previously highly memorized samples. Moreover, we observe that the maximum performance gain achieved by in-training removal is smaller than that of in-training regrouping, further demonstrating the effectiveness of our proposed strategy.

In the third experiment, we further examine that whether applying the in-training regrouping strategy multiple times will provide additional memorization mitigation and performance improvement or not. Specifically, we apply the in-training regrouping strategy twice: once at the end of the warm-up phase (epoch 10) and once near the end of training (epoch 80). The results in 7 show that applying the in-training regrouping strategy multiple times can further mitigate model memorization and yield better generalization. However, using it three times can hardly achieve a significant performance improvement compared to using it twice, so using it twice at the beginning and end of the training is the best choice considering that it does not bring many extra calculations.

7.4.1 Regularization vs. Memorization Mitigation↩︎

We conducted new experiments where we add the same amount of noise to different types of samples. 1) most memorized samples and least memorized samples identified by MultiMem, 2) samples with highest gradients or loss, 3) randomly selected samples. We report the performance gains on retrieval, linear probing, and zero-shot tasks, and MultiMem. If the improvements were only due to generic regularization rather than memorization mitigation, we would observe similar results over all setups. However, the results in the 8 show discrepancies: 1) adding noise to random samples does not change performance significantly. 2) Adding noise to least memorized samples slightly degrades performance. 3) Our MultiMem approach has much higher performance gains compared to loss-based and gradient-based methods and also achieves the best memorization mitigation. This shows that our reported improvements indeed stem from memorization removal and not general regularization.

7.4.2 Model Size and Dataset Size.↩︎

We added additional experiments on CLIP models of varying sizes trained on the same data, as well as models of the same size trained on different amount of the training data. The results in ¿tbl:tab:size? show that larger models exhibit higher memorization levels when trained on the same data, and also reducing the amount of training data leads to increased memorization for models of the same size. Both trends are consistent with prior work (e.g., [Carlini et al., ICLR 2022], [Tirumala et al., NeurIPS 2022], [27]), as larger model capacity provides more room to retain instance-level details, while sparser data coverage reduces generalization opportunities and increases reliance on memorizing individual samples.

width=

7.4.3 Impact of Noise Injection and Augmentation↩︎

In main paper, we use random Gaussian noise as augmentation because it allows precise control over augmentation strength via the noise level, enabling systematic quantitative experiments. We further added experiments on AudioCLIP using GaussianBlur (sigma=(0.1, 2.0)) with random color jitter (transforms.ColorJitter(0.4, 0.4, 0.4, 0.1)) as augmentation for image, synonym substitution (p=0.2) for caption and time shifting (+ 10%) for audio. The results shown in 9 suggest that mitigation guided by modality-specific augmentations remains effective, achieving performance comparable to that of noise injection.

References↩︎