1 Introduction↩︎

Ambivalence and hesitancy are complex affective states that may be expressed through inconsistent linguistic, acoustic, facial, and contextual information. Their automatic recognition is therefore relevant to affective computing, which studies human affect from face [1], audio [2], text [3], and bodily modalities [4], with applications in human–computer interaction, healthcare, education, and assistive systems [5]. The AH Video Recognition task of the 11th ABAW Challenge addresses this problem at the video-level using the BAH corpus [6]. In digital behavior-change scenarios, such states may reflect uncertainty, resistance, unstable motivation, or potential disengagement [7].

Previous studies have shown that transcripts provide the strongest unimodal signal for this task, whereas facial and acoustic information can further improve performance when fused effectively [6], [8], [9]. This finding motivates an asymmetric fusion strategy in which text serves as the primary semantic representation, while audio, face, and scene modalities provide complementary information. This design is consistent with recent studies on modality-aware and uncertainty-aware multimodal fusion [10], as well as with broader advances in multimodal affective computing [11], [12].

The 10th ABAW Challenge also highlighted a practical limitation of current top-performing systems [13]–[15]: the three highest-ranked solutions used ensembles or multiple prediction branches, and the winning method combined approximately twenty models [13]. Although such systems may achieve high benchmark performance, they require substantially greater computational resources, inference time, memory, and implementation effort. In this work, we propose a single text-centered multimodal approach that combines text, audio, face, and scene modalities without relying on a large ensemble. The proposed approach aims to preserve complementary multimodal information while offering a more compact and practically deployable solution for video-level AH recognition.

2 Related Work↩︎

2.1 Ambivalence and Hesitancy Recognition↩︎

The BAH benchmark introduced in [6] established the main experimental setting for video-level AH recognition. The authors evaluated unimodal and multimodal systems using face, audio, and text modalities. The examined fusion methods included feature concatenation, co-attention, Transformer-based fusion, and cross-attention. The results showed that text was the strongest individual modality. The official baseline also included a zero-shot M-LLM based on Video-LLaVA [16].

Submissions to the challenge confirmed this finding. Hallmen et al. [9] combined visual features extracted using a ViT [17], acoustic features obtained using Wav2Vec2 [18], and textual features produced by BERT [19]. They used LSTMs [20] and an MLP [21] for temporal modeling and multimodal fusion. Savchenko et al. [8] extracted facial, acoustic, and textual features with EmotiEffLib [22], Wav2Vec2 [18], and RoBERTa [23], respectively, and evaluated both early and late fusion strategies. Both studies identified text as the strongest unimodal source, while improvements from multimodal fusion depended on the effective preservation of complementary information [10].

2.2 Top Systems of 10th ABAW Challenge↩︎

The three highest-ranked solutions in the AH task of the 10th ABAW Challenge relied on model ensembles. The third-ranked system [15] combined scene, facial, acoustic, and textual features. Its final results on the Private Test subset was obtained using an ensemble of five prototype-augmented multimodal fusion models. The second-ranked ConflictAwareAH system [14] combined video, audio, and text modalities with pair-wise cross-modal conflict features and text-guided late fusion. Its final submission also used an ensemble of five models. The winning BROTHER system [13] employed a considerably larger ensemble of approximately twenty heterogeneous models and combined their predictions using a voting strategy.

Although these results demonstrate the effectiveness of model ensembles, the use of five to twenty models substantially increases computational cost, inference latency, memory requirements, and implementation complexity. These factors complicate the practical deployment and reproduction of such systems and motivate our focus on a single text-centered multimodal fusion model.

3 Proposed Approach↩︎

The pipeline of the proposed approach is shown in Figure 1. The details are described below.

3.1 Text Model↩︎

Textual features are extracted using a RoBERTa-base model pre-trained on the GoEmotions dataset¹. During training, the embedding layer and the first four Transformer encoder layers are frozen.

Let an input text sequence of length \(T^{text}\) be represented by a matrix of token embeddings: \[X^{text} = [x^{text}_1, x^{text}_2, \ldots, x^{text}_{T^{text}}] \in \mathbb{R}^{T^{text} \times 768}.\]

Each token \(x^{text}_t\) is mapped to a contextual representation by the encoder layers, producing the hidden-state matrix: \[H^{text} = \text{Encoder}(X^{text}) = [h^{text}_1, h^{text}_2, \ldots, h^{text}_{T^{text}}] \in \mathbb{R}^{T^{text} \times 768}\]

The pooled representation corresponding to the classification token is used for the final prediction. This sentence-level representation is passed to an MLP classification head consisting of a dropout layer, a FCL, a \(\tanh\) activation function, a second dropout layer, and a final FCL with two output units.

3.2 Audio Model↩︎

Before training the audio model, each video is converted into a waveform sampled at 16 kHz and processed as a full audio, without voice-activity filtering. The acoustic features are extracted from the tenth Transformer layer of a frozen Wav2Vec2 model [18]². The resulting audio sequence is represented as \[X^\text{audio} = [x^\text{audio}_1, x^\text{audio}_2, \ldots, x^\text{audio}_{T^\text{audio}}] \in \mathbb{R}^{T^\text{audio} \times 1024},\] where \(x^\text{audio}_t\) denotes the acoustic embedding corresponding to the \(t\)-th temporal interval. The extracted features are first projected into a compact hidden space and then processed by a bi-directional Mamba [24] block, followed by dropout and a residual connection. The resulting temporal acoustic features are denoted by \(H^\text{audio}\in\mathbb{R}^{T^\text{audio}\times256}\).

Because AH behavior may occur only during short intervals of a recording, frame-level annotations are used as auxiliary temporal supervision to help the model localize such events. These annotations are converted into token-level soft targets. For the set \(\mathcal{F}_t\) of video frames temporally aligned with the \(t\)-th audio token, the target is defined as \[\widetilde{y}_t = \frac{1}{|\mathcal{F}_t|} \sum_{j\in\mathcal{F}_t} y_j.\]

A token-level head consisting of a single FCL produces a token-level logit \(z_t\) and the corresponding probability \(p_t=\sigma(z_t)\). For file-level classification, masked mean and max pooling over \(H^\text{audio}\) are concatenated with the mean, maximum, top-\(K\) mean, and threshold statistics of the token-level probabilities, producing a \(518\)-dimensional descriptor. This descriptor is processed by an MLP consisting of two FCLs. LN and dropout are applied before the first layer, while a GELU activation function and dropout are applied between the two FCLs. The final FCL contains two output units. The audio model is trained using a combination of file-level and token-level supervision: \[\mathcal{L} = \mathcal{L}_{\mathrm{file-level}} + 0.5 \mathcal{L}_{\mathrm{token-level}} + 0.02 \mathcal{L}_{\mathrm{smooth}} + 0.1 \mathcal{L}_{\mathrm{event}}.\] where \(\mathcal{L}_{\mathrm{file-level}}\) denotes the main cross-entropy loss for the file-level AH label. The auxiliary term \(\mathcal{L}_{\mathrm{token-level}}\) is a token-level binary cross-entropy loss whose soft targets are obtained from the 24 fps frame-level annotations. The smoothness loss penalizes isolated peaks in the token-level probabilities, while the event loss applies binary cross-entropy to the maximum token logit, encouraging the model to detect short AH intervals within long recordings. The auxiliary-loss weights were selected based on performance of the Development subset.

3.3 Face Model↩︎

Face regions are detected using a YOLO-based face detector³. The detected regions are resized to \(224 \times 224\) pixels and processed by the frame-level facial emotion encoder Emo-AffectNet [25], which produces a \(512\)-dimensional feature vector for each face. A video containing \(T^{\text{face}}\) retained face regions is represented as \[X^\text{face} = [x^\text{face}_1, x^\text{face}_2, \ldots, x^\text{face}_{T^\text{face}}] \in \mathbb{R}^{T^\text{face} \times 512},\] where \(x^\text{face}_t\) denotes the frame-level embedding. A maximum of \(T^\text{face}=500\) frames is used. Longer sequences are uniformly subsampled, while shorter sequences are remained unchanged.

The temporal dynamics of facial AH expressions are modeled with a Transformer-based encoder [26]. The Emo-AffectNet embeddings are projected into a \(128\)-dimensional hidden space, combined with positional encodings, and processed by a single Transformer layer with eight attention heads. The temporally pooled representation is then passed to an MLP classification head consisting of a FCL, LN, a GELU activation function, dropout, and a final FCL with two units. The output of the penultimate FCL is used as the facial classification features, and the output of the final layer represents the facial logits.

To regularize the model, we apply flow matching [27] in both the temporal space and the logit spaces. Let \(Z\) denote the temporal features produced by the Transformer-based encoder from the input sequence \(X^\text{face}\). For feature-space flow matching, a noise tensor \(Z_0\) is interpolated with \(Z\) using \(t \sim \mathcal{U}(0,1)\): \[\begin{align} Z_t = (1-t) Z_0 + t Z, \mathcal{L}_{\mathrm{FM}}^{\mathrm{feat}} &= \left\| f_\theta(Z_t, Z, t) - (Z - Z_0) \right\|_2^2 . \end{align}\]

For logit-space flow matching, the initial logits \(\hat{y}_0\) are interpolated with the one-hot target vector \(q\): \[\begin{align} y_t = (1-t)\hat{y}_0 + t q, \mathcal{L}_{\mathrm{FM}}^{\mathrm{logit}} &= \left\| g_\psi(y_t, z, t) - (q-\hat{y}_0) \right\|_2^2 . \end{align}\]

The final training loss is defined as \[\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda_{\mathrm{FM}} \left( \mathcal{L}_{\mathrm{FM}}^{\mathrm{feat}} + \mathcal{L}_{\mathrm{FM}}^{\mathrm{logit}} \right),\] where \(\mathcal{L}_{\mathrm{CE}}\) is the class-weighted cross-entropy loss for two-class classification. Flow matching is performed using four integration steps, with \(\lambda_{\mathrm{FM}}=0.1\).

3.4 Scene Model↩︎

Scene information is extracted from the full video frames rather than from cropped face regions. We use a VideoMAE-v2-base visual encoder [28] as the scene feature extractor. Sixteen frames are uniformly sampled from the entire video to preserve information about the background, body posture, and interaction context. For each video, the encoder produces a sequence of embeddings:

\[X^\text{scene} = [x^\text{scene}_1, x^\text{scene}_2, \ldots, x^\text{scene}_{T^\text{scene}}] \in \mathbb{R}^{T^\text{scene} \times 768},\] where \(x^\text{scene}_t\) denotes the visual representation of the \(t\)-th sampled temporal position.

The extracted sequence of scene features is projected into a latent space and processed by a Mamba-based encoder [24]. In the main configuration, the temporal encoder has a hidden dimension of \(128\) and consists of two Mamba layers with a state dimension of \(64\), a one-dimensional convolution kernel of size \(3\), and an expansion factor of \(1\). Given the projected sequence \(H^\text{scene}=[h^\text{scene}_1,h^\text{scene}_2,\ldots,h^\text{scene}_{T^\text{scene}}]\), each Mamba layer applies LN, sequence mixing, dropout, and a residual connection: \[H{^\text{scene}}^{(l+1)} = H{^\text{scene}}^{(l)} + \text{Dropout} \left( \text{Mamba}\left(\text{LN}(H{^\text{scene}}^{(l)})\right) \right).\]

The output sequence is aggregated using masked mean pooling to obtain a single scene representation, which is passed to the same MLP classification head as that used in the face model. The output of the penultimate FCL is used as the scene classification features, while the temporal hidden states is used as the temporal scene features.

3.5 Multimodal Fusion Model↩︎

Let \(\mathcal{M}\subseteq\{\mathrm{audio},\mathrm{text},\mathrm{face},\mathrm{scene}\}\) denote the set of active modalities. For each modality \(m\), we use a vector representation and, when available, the corresponding unimodal logits \(\ell_m\). Temporal representations are converted into compact vectors using summary statistics. The best configuration uses: \[s_m=[\mu_m,\sigma_m,\mu_m^\Delta,\sigma_m^\Delta],\] where \(\mu_m\) and \(\sigma_m\) denote the mean and standard deviation of the temporal features, while \(\mu_m^\Delta\) and \(\sigma_m^\Delta\) denote the mean and standard deviation of their first-order temporal differences. The input to the fusion model is denoted by \(u_m\). For example, when the unimodal logits are concatenated with the temporal statistics, \(u_m=[s_m;\ell_m]\).

The Text Residual Fusion model uses text as the anchor modality, because it provides the strongest unimodal performance in our experiments. After modality-specific projections, the textual representation \(b=h_{\mathrm{text}}\) is adjusted using gated residuals from the remaining modalities. For each non-text modality \(m\), the gate is computed from the concatenated textual and modality-specific representations: \[\begin{align} g_m = \sigma\left(\text{MLP} [b;h_m]\right), z = \text{LN}\left(b + \sum_{m\neq \text{text}} g_m\,d_m(h_m)\right), \end{align}\] where \(d_m(\cdot)\) is a residual MLP, \(g_m\) is a learned gate, and \(\odot\) denotes element-wise multiplication. The gate controls the contribution of modality \(m\) to the textual representation. The fused vector \(z\) is then passed to a two-layer MLP classifier. This design preserves the text as the primary source of information while allowing the audio, face, and scene modalities to provide input-dependent residual adjustments. The pipeline of the proposed model is shown in Figure 2.

4 Experiments↩︎

4.1 Research Corpus↩︎

The BAH corpus is the benchmark dataset for the AH task of the 11th ABAW Challenge. It was introduced for the multimodal AH recognition in realistic digital behavior change scenarios [6]. Participants responded to a predefined set of questions intended to elicit neutral, positive, negative, willing, resistant, ambivalent, and hesitant responses during online interactions guided by virtual avatar [6].

The corpus contains \(1427\) videos from \(300\) participants, with a total duration of \(10.60\) hours. It includes video-level and frame-level annotations, temporal boundaries of AH episodes, aligned face regions, timestamped speech transcripts, and participant metadata [6]. Following the challenge protocol, AH are treated jointly as a binary classification task. The dataset is divided at the participant level into the Train, Development, Public Test, and Private Test subsets. Performance is evaluated at the video-level using MF1 [6].

4.2 Experimental Results↩︎

For all unimodal and multimodal experiments, we used Optuna [29] to optimize the training procedure and model architecture. The final configurations (e.g., learning rate, dropout, hidden dimensions, batch size, and other parameters) were selected on the Development subset and evaluated on the Public Test subset.

As shown in Table 1, text is the strongest unimodal modality, followed by audio. Both fusion models outperform all unimodal configurations on the Development and Public Test subsets. Text Residual Fusion achieves the highest Development MF1 of 76.13%, the highest average MF1 across the Development and the Public Test subsets of 75.14%, and the best Private Test MF1 of 78.24%. On the Private Test subset, it outperforms the text model by 4.03%.

5 Conclusion↩︎

This work presented a single text-centered multimodal approach that combines textual, acoustic, facial, and, scene features for video-level AH recognition. The proposed Text Residual Fusion model uses text as the anchor modality and adds complementary information through gated residual adjustments. The proposed fusion model achieves the best performance on the Private Test subset and the most consistent results across the evaluation subsets.

References↩︎