1 Introduction↩︎

Multimodal emotion and intent recognition [1]–[3] is pivotal for automated human-computer interaction, as it seeks to interpret and identify human emotions and intentions through various modalities such as speech content, voice tones, and facial expressions. However, real-world applications often face the challenge of missing modalities due to incomplete data collection or sensor malfunctions, including speech recognition errors, user silence, or visual occlusion, as shown in Fig. 1. These issues significantly complicate multimodal recognition.

To tackle missing modalities, several existing studies [4]–[9] focus on leveraging available modalities to reconstruct the missing information, thus mitigating the adverse effects on model performance caused by modality loss. Approaches like those in [6], [10] utilize modality translation to create a unified representation, but this requires separate reconstruction models for each missing modality scenario, leading to increased costs and inconvenience. To address this, [7] proposed a unified framework capable of handling all missing modalities within a single model. This framework, known as the Missing Modality Imagination Network (MMIN), employs cascade residual auto-encoders [5] and cycle consistency learning to generate the missing modalities. Building upon MMIN, [11] enhances model performance by introducing modality-invariant features to bridge modality gaps.

Despite advancements, significant challenges persist. Current methods often attempt to recover the multimodal joint representation by jointly training feature extraction and classification networks. However, this approach can lead to over-coupling, causing a sharp decline in performance when modalities are missing. Even with completed modalities, the performance may fall short compared to simple zero padding, which is unsatisfactory. Moreover, these methods often lack a precise generation process, making it difficult to produce missing modality features that closely align with authentic multimodal feature distribution, leading to minimal model improvements.

In this paper, we propose an Attention-based Diffusion model for Missing Modalities feature Completion framework (ADMC), as illustrated in Fig. 2. To prevent over-coupling between features and the classification network, we independently train feature extraction networks for each modality through single-modality classification, preserving their unique characteristics. Subsequently, we introduce an Attention-based Diffusion Network (ADN) to generate missing modality features that closely adhere to the authentic multimodal feature distribution. ADN operates in the latent space of full-modality features, training a denoising autoencoder with a self-attention mechanism to capture inter-modal dependencies. In the presence of missing modalities, the module completes them through a sequence of denoising steps. Finally, to ensure optimal performance of available modalities even when some are missing, we train separate multimodal feature fusion networks (MF) for each missing modality scenario. MF employs a self-attention mechanism to calculate the weights for each modality and aggregates them accordingly. Interestingly, we find that even in full-modality scenarios, using ADN for cross-modal generation can provide supplemental information, enhancing the final recognition. Thus, our method excels in both missing and full-modality scenarios, achieving state-of-the-art results on the IEMOCAP and MIntRec benchmarks.

Our contributions can be summarized as follows:

• We introduce a novel ADMC framework for missing modality completion that integrates independently trained feature extraction networks and an attention-based diffusion model.

• We propose an Attention-based Diffusion Network (ADN) for capturing inter-modal dependencies and generating accurate missing modality features.

• Our method excels in both missing and full-modality scenarios, showcasing strong practical applicability.

• We evaluated our approach on the IEMOCAP and MIntRec datasets, achieving state-of-the-art results.

2 RELATED WORK↩︎

2.1 Incomplete Multimodal Learning↩︎

To address the issue of missing modalities, simple strategies like filling missing modality values with zeros [12] or averages [13]. However, these methods lack supervision information, leading to suboptimal performance. Low-rank imputation methods attempt to recover missing data by restoring the low-rank nature of complete multimodal data. For instance, Fan et al. [14] minimized the tensor tubal rank to handle various missing modalities. Liang et al. [15] incorporated non-linear functions to capture complex correlations in tensor rank minimization. However, these methods may overlook the complementarity between heterogeneous modalities, resulting in suboptimal results.

Given the powerful feature representation capabilities of deep learning methods, they have shown remarkable performance in reconstructing missing modalities. Early approaches include Autoencoders (AE) [16], Generative Adversarial Networks (GAN) [17], and Variational Autoencoders (VAE) [18]. Recently, Tran et al. [5] proposed a Cascade Residual Autoencoder (CRA) to reconstruct missing modality data. Zuo et al. [11] and Zhao et al. [7] employed CRA and cycle consistency learning to learn joint multimodal representations for addressing missing modality problems, gaining significant attention due to their superior performance. Yuan et al. [19] and zeng et al. [8] utilized Transformer architectures to integrate cross-modal information and capture inter-modal dependencies, achieving outstanding results in missing modality completion tasks. Additionally, Lian et al. [20] introduced Graph Neural Networks (GNN) to capture speaker and temporal dependencies, effectively enabling the reconstruction of missing modalities. These methods have significantly advanced research in the field of incomplete multimodal learning.

2.2 Diffusion Model↩︎

Diffusion models were first applied in the field of image processing. Ho et al. [21] significantly improved the quality of generated samples by simplifying the training objective (directly predicting noise) and optimizing model parameterization. Compared to other generative models, such as GANs and VAEs, these improvements demonstrated a more stable training process. Building on this foundation, Song et al. [22] proposed a unified framework for diffusion models by integrating score matching with stochastic differential equations (SDE), addressing the inefficiency of traditional diffusion models. These advancements have established a solid theoretical foundation and practical methods for high-efficiency diffusion models.

With the further development of diffusion models, their conditional generation capabilities have been progressively explored and optimized. Dhariwal et al. [23] and Ho et al. [24] leveraged the gradients of pre-trained classifiers or directly trained generative models to achieve conditional control, laying the groundwork for conditional diffusion. Conditional diffusion models can also guide the generation process by integrating textual descriptions or other modalities. For example, Zhong et al. [25] combined pre-trained language models with diffusion models to generate high-quality images that align closely with textual descriptions. Similarly, Saharia et al. [26] applied diffusion models to conditional image generation, demonstrating remarkable performance across various image transformation tasks. Furthermore, Liu et al. [27] implemented diffusion processes in latent space for text-to-audio cross-modal generation, significantly reducing computational complexity while preserving generation quality. These advancements have further expanded the application scenarios of diffusion models.

3 METHODOLOGY↩︎

Our method aims to classify the category \(y\) for each video segment, as shown in Fig. 2. We use \(A\), \(V\), and \(T\) to represent the raw acoustic, visual, and textual datas of the video segment, respectively. During the training phase, the model is provided with the full-modalities as input, while in the testing phase, it accepts either partial or full-modalities. The method consists of three stages: First, independently train feature extraction networks for each modality to obtain visual \(F_v\), acoustic \(F_a\), and textual \(F_t\) features (Fig. 2 (a)). Next, fix these feature extraction networks (\(Enc_v\) for visual, \(Enc_a\) for acoustic, and \(Enc_t\) for textual) and train an Attention-based Diffusion Network (ADN) to generate missing modalities features (Fig. 2 (b)). Finally, with the parameters of Enc and ADN fixed, we train Multimodal Fusion classification networks (MF) based on the full-modalities, which outputs the final classification results (Fig. 2 (c)).

3.1 Modality-specific Feature Extraction↩︎

We follow the feature extraction approach described in [7], [11], [28], extracting visual \(F_v\), audio \(F_a\), and text \(F_t\) utterance-level representations from the raw datas. These features are extracted using networks trained on single-modality classification, with early stopping applied to prevent overfitting, as shown in Fig. 2 (a). The parameters of these feature extraction networks are fixed during subsequent tasks. For ease of processing, these features are stacked to form a multimodal feature representation \(F^0_{tva}\), which serves as the input for the following modules.

3.2 Attention-based Diffusion Networks↩︎

The Attention-based Diffusion Networks (ADN) consists of two processes: forward and reverse [27].

In the forward process, multimodal features \(F^0_{tva}\) are gradually corrupted by adding Gaussian noise \(\epsilon\). As the diffusion step \(n \in [1, \dots, N]\) increases, the original information is progressively lost, approaching a noise distribution, as defined in Eq. 1 . Additionally, a multi-head self-attention network[29] is employed to capture inter-modal dependencies and predict the added noise \(\epsilon'\) at each time step, as defined in Eq. 2 . The ADN network is trained by minimizing the mean squared error (MSE) between \(\epsilon\) and \(\epsilon'\), as shown in Fig. 2 (b). \[\label{q951} q(F^n_{tva} | F^0_{tva}) = \mathcal{N}(F^n_{tva}; \sqrt{\bar{\alpha}_n} F^0_{tva}, (1 - \bar{\alpha}_n)\epsilon),\tag{1}\] \[\label{q952} \epsilon' = \text{Attention}(Q^l, K^l, V^l) \quad \text{for } l = 1, 2, \dots, L,\tag{2}\] Where \(F^n_{tva}\) represents the multimodal feature after the \(n\)-th forward diffusion step, and \(\bar{\alpha}_n\) is the cumulative noise scheduling parameter. \(Q^l\), \(K^l\), and \(V^l\) correspond to the query, key, and value vectors of \(F^n_{tva}\), respectively, where \(L\) indicates the number of layers in the self-attention network.

In the reverse process, we incorporate the available modalities part of the latent representation \(F_{o}\) into the generated latent representation \(F_{tva}\). Starting from step \(n\) in the reverse diffusion process, after each inference step (Eq. 3 ), we update \(F^{n-1}_{tva}\) using Eq. 4 . By using this approach, we can generate the missing modality features while preserving the ground-truth data of the available modalities. \[\begin{align} \label{q953} p_\theta(F^{n-1}_{tva} | F^n_{tva}) &= \mathcal{N}\bigg(F^{n-1}_{tva}; \frac{1}{\sqrt{\alpha_n}} \left( F^n_{tva} - \right. \\ &\;\left. \frac{\beta_n}{\sqrt{1 - \bar{\alpha}_n}} \epsilon_\theta(F^n_{tva}, n) \right), \sigma^2_n \mathbf{I} \bigg), \end{align}\tag{3}\] \[\label{q954} F^{n-1}_{tva} = m \odot F^{ n-1}_{o} + (1 - m) \odot F^{n-1}_{tva},\tag{4}\] Where \(F^{n-1}_{tva}\) represents the latent representation after the \((N-n+1)\)-th reverse diffusion step, while \(F^{n-1}_{o}\) denotes the latent representation of the available modality part after the \((n-1)\)-th forward diffusion step. The optimal \(n\) is determined experimentally. The mask \(m\) indicates missing and available modalities, and \(\beta_n\) is the noise scheduling parameter.

3.3 Modality Completion and Enhancement↩︎

We validate our method under both missing modality and full-modality scenarios, as shown in Fig. 2 (c). In the missing modality scenario, we replace the missing features with Gaussian noise \(\epsilon\) and use the trained ADN to generate the missing modality features. This process ensures the completeness of multimodal data, enhancing the model’s robustness across various missing modality scenarios.

In the full-modality scenario, we propose a MultiModal Enhancement Recognition (MMER) method. The input consists of full-modality features, we use the ADN for cross-modal generation (e.g., using \(F_a\) and \(F_v\) to generate \(F'_t\)). These generated features \(F'\) encode key information from other modalities and can provide additional complementary cues to the original modality. To integrate this information, we concatenate \(F\) with \(F'\), resulting in the enhanced modality representation \(F_{\text{enhd}}\). As shown in Eq. 5 , where \(F\) denotes the original modality feature, and \(F'\) represents the generated cross-modal feature obtained from the ADN. The function \(\text{cat}(\cdot)\) denotes the concatenation operation, performed along the channel dimension.

\[\label{q955} F_{\text{enhd}} = \text{cat}(F, F').\tag{5}\]

3.4 Multimodal Feature Fusion Network↩︎

The output feature \(F_M \in \mathbb{R}^{M \times D}\) obtained from completion or enhancement operation is processed by a multi-head self-attention based multimodal fusion network (MF). Specifically, MF first employs the multi-head attention mechanism to model inter-modal dependencies and learn attention weights for each modality. Then, it performs a weighted aggregation of modality features followed by mean pooling along the modality dimension to obtain the fused representation \(u\), as shown in Eq. 6 . Finally, \(u\) is fed into a fully connected classification network to produce the final prediction.

\[u = \text{Mean}\!\left(\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\right), \label{q956}\tag{6}\] where \(Q = F_M W^Q\), \(K = F_M W^K\), and \(V = F_M W^V\). \(W^Q\), \(W^K\) and \(W^V\) are learnable parameters, \(d_k\) denotes the dimension of the key vectors, and Mean(\(\cdot\)) represents the mean pooling operation along the modality dimension.

4 Experiments Setup↩︎

4.1 Dataset↩︎

We evaluated our proposed method on two benchmark multimodal datasets: IEMOCAP [30] and MIntRec [28]. IEMOCAP is an Interactive Emotional dataset with four emotional labels. MIntRec is a multimodal intent recognition dataset collected from life TV dramas, featuring twenty intent labels.

In the IEMOCAP dataset, the frame-level features are 130-dim OpenSMILE features (IS13 ComParE) [31], 342-dim Denseface features from a pretrained DenseNet [32], and 1024-dim BERT-large embeddings [33]. In the MIntRec dataset, the frame-level features are 768-dim wav2vec 2.0 features [34], 256-dim Faster R-CNN features from a pretrained ResNet-50 [28], and 768-dim BERT embeddings [33]. Similar to [7], [11], for \(Enc_a\), we use an LSTM [35] network to capture temporal information from sequential frame-level acoustic features and apply max-pooling to obtain utterance-level acoustic embeddings. For \(Enc_v\), we use the same approach as \(Enc_a\) to obtain utterance-level visual embeddings. For \(Enc_t\), we employ a TextCNN [36] to obtain utterance-level textual embeddings from sequential word-level features.

4.2 Implementation details↩︎

The hidden size for encoders \(Enc_a\) and \(Enc_v\) is set to 256. \(Enc_a\) consists of 3 convolutional blocks with kernel sizes of 3, 4, and 5, each producing an output size of 256. In the ADN module, the self-attention network has 4 encoder layers with 8 attention heads and a hidden size of 1024. The diffusion model uses noise scheduling parameters from \(\beta_s = 0.0003\) to \(\beta_e = 0.06\) over \(N = 1000\) steps. In the MF module, the self-attention network has 2 layers with 4 attention heads, a hidden dimension of 1024, and a dropout rate of 0.5. The classifier consists of 2 fully connected layers of size {128, 4}. We use the Adam optimizer with a dynamic learning rate, starting at 0.0001, and apply the Lambda LR scheduling method for learning rate updates. We select the best model on the validation set and report its performance on the testing set. All models are implemented with Pytorch deep learning toolkit and run on an RTX 4090 GPU.

4.3 Baseline↩︎

The method is compared with recent approaches addressing missing modalities: 1) CRA [5]: is an extension of AE [16]. It combines a series of residual autoencoders into a cascaded architecture for missing data imputation. 2) MCTN [6]: Utilizes cyclic translation between modalities to obtain joint representations. 3) MMIN [7]: Learns joint representations via cross-modality imagination using cascade residual auto-encoder and cycle consistency learning. 4) IF-MMIN [11]: Enhances MMIN by introducing modality-invariant features to reduce modality gaps. 5) IMDer [37]: A score-based diffusion framework that models the distribution of missing modalities and reconstructs them by denoising random noise under semantic conditions. 6) MPLMM [38]: Learning of both intra- and inter-modal information through generative and missing-signal prompts, while significantly reducing the number of trainable parameters. 7) P-RMF [39]: Combines quantified modality intrinsic uncertainty to learn stable multimodal joint representations, enhancing the model’s robustness to noise and missing data.

5 Experimental results and analysis↩︎

5.1 Comparative Study↩︎

To comprehensively evaluate the effectiveness of our method, we compare it with previous state-of-the-art approaches and report weighted accuracy (WA) and unweighted accuracy (UA), as shown in Tables 3 and  4. On the IEMOCAP dataset, our results significantly outperform previous approaches across all modality-missing scenarios, with an average performance improvement of 1.9%. Similarly, on the MIntRec dataset, our method sustained these advantages, achieving an average performance improvement of 9.4%. These results demonstrate the robustness and generalization of our method in missing modality completion tasks.

Specifically, compared to autoencoder-based methods (AE, CRA), translation-based methods (MCTN, MMIN) typically yield better performance. The modality-to-modality transformations in these methods help capture inter-modality dependencies and generate features for missing modalities. However, IF-MIN and P-RMF consistently underperform ADMC in single-modality input scenarios. This performance gap arises primarily from their joint training strategies, which lead to strong coupling between the feature extraction networks of different modalities. When certain modalities are missing, this coupling causes a significant drop in model performance. Although IMDer also uses diffusion models to reconstruct missing modalities, its performance is limited by the lack of an attention-based cross-modal mechanism. In contrast, ADMC utilizes independently trained feature extraction networks (ITFN), which effectively reduce inter-modality coupling. This approach allows each available modality to fully express its representational power, even when some modalities are missing. Additionally, ADMC employs Attention-based Diffusion Networks (ADN) to capture inter-modality dependencies and generate missing modality features that closely align with the authentic multimodal feature distribution, thereby significantly enhancing the overall performance of the model.

5.2 Ablation Study↩︎

To validate the contributions of the independently trained feature extraction networks (ITFN) and ADN modules, we conducted ablation experiments, as shown in Tables 3 and  4.

1) w/o ITFN: Removing the independently trained feature extraction networks and using the jointly trained extractor, while missing modality features are generated by the ADN, leads to a performance drop under missing-modality conditions (e.g., using only audio or only visual modality). This indicates that joint training leads to over-coupled feature extraction across modalities, making the model less robust when certain modalities are missing. 2) w/o ADN: Retaining the independent feature extractors but replacing missing modality features with zero vectors instead of reconstructing them via ADN causes performance degradation across all scenarios, showing that without reconstructing missing modalities, multimodal information cannot be fully utilized. With ADN, the model can generate complementary features based on the available modalities, thereby improving robustness under missing-modality conditions. 3) w/ U-Net: Replacing the self-attention module in ADN with a standard U-Net[40] yields lower performance. Although U-Net can generate features close to the target distribution, it lacks explicit modeling of cross-modal dependencies, resulting in reconstructed features that are less consistent with the global multimodal semantics and thus lower overall performance. In contrast, ADN captures cross-modal relationships through self-attention and produces more discriminative complementary features.

5.3 Visualization Analysis↩︎

To intuitively analyze the impact of different values of \(n\) on the quality of generated features during reverse generation, we visualized the differences between ground-truth multimodal embeddings and the corresponding generated embeddings. Specifically, we randomly selected 600 real samples and their generated counterparts from each modality, and applied the T-SNE toolkit [41] for dimensionality reduction. As shown in Figure 3, we observe that: 1) Features from different modalities exhibit clear clustering patterns in the embedding space, indicating that the ground-truth embeddings effectively preserve modality-specific information. 2) Extremely small or large values of \(n\) negatively impact the quality of the generated embeddings, leading to blurred distribution boundaries and overlaps between modalities. 3) When \(n\) = 50, the generated embeddings closely align with the distribution of the ground-truth embeddings, demonstrating the effectiveness of our method in generating missing modality features. 4) On the MIntRec dataset, the generated features show a higher degree of dispersion, primarily due to the limited sample size, which restricts the model’s ability to capture the underlying data distribution.

5.4 Effects of MMER↩︎

To evaluate the effectiveness of the MMER module, we conducted comparative experiments under full-modality scenarios, as shown in Table 5. The experiment compared three methods: unrestricted feature extraction networks (Jointly_training), fixed feature extraction networks (Use_ITFN), and using generated features to enhance original modality representations (MMER). The results indicate that the Jointly_Training method outperforms the Use_ITFN method by dynamically adjusting the feature representations of each modality, enabling the model to achieve optimal performance. The MMER method achieves the best performance by enhancing original modality representations using generated features.

Existing methods for addressing missing modality problems often perform poorly in full-modality scenarios. In contrast, our method not only focuses on tackling missing modality issues but also demonstrates excellent performance in full-modality tasks.

5.5 Experimental of Different Missing Rate↩︎

To evaluate the robustness of multimodal models under inter-modality missingness, we using a probabilistic approach to randomly drop one or two modalities at varying missing ratios \(p \in {0.1, 0.2, \ldots, 0.5}\). The performance trends of different models under increasing missing ratios are shown in Fig. 4.

As the missing ratio increases, the F1 scores of all baseline methods consistently decline, indicating that inter-modality missingness disrupts semantic integrity and weakens the effectiveness of multimodal feature fusion. In contrast, our method achieves the best performance across all missing ratios and suffers the smallest performance degradation. This advantage stems, on the one hand, from the ADN module, which generates missing modality features that closely match the real feature distribution and thus enhance multimodal fusion, and on the other hand from the decoupled feature extraction mechanism in our framework, which effectively mitigates the impact of missing modalities on overall model performance.

5.6 Time Complexity and Inference Cost↩︎

Introducing a diffusion model inevitably increases inference time. We therefore analyze the computational complexity and practical inference cost of the proposed ADN module. ADN is a 4-layer Transformer-encoder-based self-attention network. For an input with \(L\) modalities and feature dimension \(D\), the per-step inference complexity is \(\mathcal{O}(L^2 D + L D^2)\). In our setting, the number of modalities is very small (\(L = 3\)), so the overall complexity is substantially lower than that of standard long-sequence Transformers.

We further evaluate the actual inference latency on an RTX 4090 GPU. With 1,000 diffusion denoising steps, the additional inference time introduced by ADN is approximately 50 ms per sample, which is acceptable for our target scenarios. Moreover, existing acceleration techniques for diffusion models (e.g., DDIM, reduced-step samplers, and step-skipping strategies) can reduce the number of steps to 50–100 with negligible performance degradation, indicating clear potential for further efficiency improvements.

6 CONCLUSION↩︎

In this paper, we propose an Attention-based Diffusion Model for Missing modalities feature Completion (ADMC), which integrates independently trained feature extraction networks and an Attention-based Diffusion Network (ADN). Our approach preserves the distinct characteristics of each modality while generating missing modality features that closely adhere to the authentic multimodal feature distribution. Extensive experimental results demonstrate that our method achieves state-of-the-art performance. In future work, we will continue to explore the applications of attention-based diffusion networks in a broader range of tasks.

7 Safe and Responsible Innovation Statement↩︎

This work supports more robust multimodal emotion and intent recognition under missing-modality conditions, which may improve accessibility and reliability in human-computer interaction. However, such systems may also introduce risks related to privacy, bias, and misuse. The datasets used in this study are public benchmarks, and we do not collect new personal data. We acknowledge that emotion and intent recognition models can inherit demographic, cultural, and contextual biases from training data, potentially leading to unfair outcomes. Our method is intended as a decision-support tool rather than a sole basis for high-stakes decisions. Responsible deployment should include human oversight, bias evaluation across groups, privacy protection, and clear usage boundaries.

References↩︎