1 Introduction↩︎

Ambivalence and hesitancy (AH) — the co-occurrence of competing inclinations and the reluctance to commit to a behavioural choice — arise in high-stakes communication contexts with direct clinical and social relevance: motivational interviews aimed at health behaviour change [1], negotiation, and social-skill coaching [2]. Recognising AH from video is fundamentally different from identifying discrete emotions such as happiness or disgust. Discrete emotions tend to produce correlated responses across face, voice, and speech [3]; AH, by contrast, frequently materialises as cross-modal incongruence: a speaker’s words may frame a thought positively while their prosody signals hesitation, or their facial expression remains neutral as their lexical choices convey ambivalence [1]. The discriminative signal lies in the disagreement between modalities, not in any single channel.

The BAH dataset [1], developed for the ABAW AH challenge series, operationalises these difficulties under controlled experimental conditions. Naturalistic clinical interview recordings from 300 speakers are partitioned with strict speaker disjointness across 778 training, 124 validation, and 525 test clips. Each clip carries a binary AH label alongside a per-clip annotator-agreement certainty score; low certainty indicates genuine perceptual ambiguity, not labelling error. Three consequences shape the learning problem: (i) the training set is small relative to the feature dimensionality of large pre-trained backbones; (ii) label noise is structurally present on ambiguous samples; and (iii) any system that encodes speaker-specific visual appearance or voice timbre will degrade on the test split, where all speakers are unseen.

Prior work on BAH can be grouped into three threads, each addressing a subset of these challenges. Multimodal fusion methods improve over unimodal baselines substantially, but simple concatenation implicitly assumes modality correlation and does not exploit the cross-modal disagreement that defines AH. Conflict-aware fusion, introduced by Bekhouche et al. [4], addresses this directly by appending pairwise absolute-difference terms \(|\mathbf{e}_i - \mathbf{e}_j|\) to the concatenated embeddings, explicitly encoding cross-modal incongruence; this yields the strongest published frozen-backbone result on BAH (val Macro F1 = 0.746). Fine-tuned large-model approaches such as Team LEYA [5] achieve the highest published validation F1 (0.854) through end-to-end optimisation of VideoMAE, Wav2Vec 2.0, and EmotionDistilRoBERTa, but their validation-to-test gap (0.854 vs.) signals overfitting to the small 124-video validation set. Domain adaptation methods reduce subject-identity sensitivity in expression recognition [6]–[8], but they require subject-specific data at adaptation time — unavailable under the speaker-disjoint BAH protocol. None of these threads simultaneously handles explicit cross-modal incongruence modelling, speaker-identity removal without requiring adaptation data, and principled handling of annotation uncertainty.

We present CF-Net to address all three challenges within a single lightweight architecture. CF-Net encodes visual, audio, and transcript streams with frozen pre-trained backbones (SigLIP2, HuBERT, DistilBERT), applies per-speaker normalisation to the visual and audio outputs to remove the constant speaker-identity offset, and fuses the normalised embeddings through a ConflictFusion module that explicitly encodes cross-modal incongruence via pairwise absolute-difference features. Training is guided by certainty-weighted focal loss and an auxiliary certainty-regression head that jointly leverage per-video annotator-agreement scores; manifold mixup and modality dropout further regularise against the small training set.

In summary, our contributions are as follows:

• Introduce CF-Net with a ConflictFusion module that explicitly represents cross-modal incongruence via pairwise absolute-difference features over visual, audio, and text embeddings, trained with certainty-weighted focal loss, manifold mixup, and modality dropout.

• Propose per-speaker feature normalisation applied to visual and audio backbone outputs before temporal encoding, removing the constant speaker-identity offset while preserving within-speaker variation; ablation shows a \(+\)​0.013 gain in validation Macro F1 (2), and the positive val-to-test transfer (\(+\)​0.021) is consistent with reduced identity overfitting.

• Incorporate an auxiliary certainty-regression head and certainty-weighted focal loss that jointly leverage per-video annotator-agreement scores to reduce overconfidence on ambiguous samples, contributing cumulatively \(+\)​0.013 in validation F1 (2).

• CF-Net achieves Macro F1 of 0.7155 on the BAH validation set and 0.7364 (AP = 0.7392) on the 11th ABAW private test, competitive with the best results from the 10th ABAW challenge (BROTHER: 0.747 [9]; ConflictAwareAH: 0.715 [4]), with all backbone weights frozen and training under 15 minutes on a single GPU.

2 Related Work↩︎

2.1 Ambivalence and Hesitancy Recognition↩︎

The ABAW AH challenge series [1] provides the formal benchmarks on which we evaluate. González-González et al. [1] introduced the BAH dataset and established a TF-IDF + LinearSVC baseline (val Macro F1 = 0.656), demonstrating that explicit verbal hesitancy markers capture meaningful AH signal even without visual or acoustic information. Subsequent challenge editions revealed two recurring patterns: multimodal fusion consistently outperforms unimodal baselines, with text features contributing disproportionately; and the validation-to-test generalisation gap is large for any system that absorbs speaker-specific identity cues from visual or acoustic backbones, because the BAH splits enforce strict speaker disjointness. In the 10th ABAW challenge (CVPR 2026) [10], Bekhouche et al. [4] employed conflict-aware multimodal fusion (val F1 = 0.746); BROTHER [9] used a heterogeneous ensemble (val F1 \(\approx\) 0.758); Team LEYA [5] achieved the highest published validation score (0.854) through end-to-end fine-tuning of four large backbone models, but their validation-to-test drop (to 0.714 on the labeled test split) illustrates the overfitting risk on the small validation set. CF-Net targets a different operating point: frozen backbones, lightweight temporal encoders, and certainty-aware training. It achieves val F1 = 0.716 with a positive val-to-test transfer (\(+\)​0.021), which is the opposite of the overfitting trend observed in the fine-tuned systems.

2.2 Multimodal Fusion with Explicit Conflict↩︎

Standard early-fusion and late-fusion strategies average or concatenate unimodal representations, implicitly assuming that modality signals are correlated [11]. This assumption holds for prototypical emotions but breaks down for AH, where the discriminative signal is precisely the disagreement between channels. Tensor Fusion Networks (TFN) [12] model cross-modal interaction through outer products, capturing higher-order correlations but at quadratic parameter cost; Low-Rank Multimodal Fusion (LMF) [13] reduces this cost through rank decomposition. Attention-based cross-modal pooling [14] learns which modalities are most informative per sample, but the additional parameters risk overfitting on small datasets such as BAH. ConflictFusion [4] takes a structurally different route: it appends pairwise absolute-difference terms \(|\mathbf{e}_i - \mathbf{e}_j|\) to the concatenated modality embeddings, producing a fused vector that grows with cross-modal disagreement. This design adds no trainable parameters to the fusion step itself, making it well-suited to the small-data BAH regime. We adopt ConflictFusion as the core fusion mechanism and extend it by feeding speaker-normalised features, ensuring that pairwise differences reflect genuine affective incongruence rather than stable between-speaker stylistic differences.

2.3 Robustness to Speaker Identity↩︎

Speaker-identity leakage — the encoding of a speaker’s appearance and voice timbre by visual and acoustic backbones — is the primary source of validation-to-test degradation in AH recognition, because the BAH test speakers are entirely unseen [1]. In facial expression recognition, domain adaptation methods have addressed analogous subject-identity shifts: Zeeshan et al. [7] introduced subject-based domain adaptation to align feature distributions across subjects; Sharafi et al. [6] proposed source-free adaptation that personalises a model using the target subject’s unlabelled data; Sharafi et al. [8] further extended this to test-time adaptation via personalised feature caching. These methods are effective in their respective settings but share a common requirement — subject-specific data at adaptation or test time — that is structurally incompatible with the BAH protocol. Text-based models offer a different form of identity immunity: transcripts carry no speaker-level acoustic or visual signal and therefore generalise naturally to unseen speakers [4]. Modality dropout [15] reduces over-reliance on any single channel during training. We combine modality dropout with per-speaker normalisation: subtracting the speaker mean from every visual and audio backbone feature removes the identity offset at source, with no adaptation procedure and no subject-specific data requirement.

2.4 Annotation Uncertainty in Affective Computing↩︎

Affective states such as AH are inherently subjective, and multi-annotator labelling produces disagreements that carry semantic information: a clip on which annotators agree strongly is perceptually unambiguous, while a contested clip is genuinely borderline. Treating all training labels as equally reliable is suboptimal under this condition. The BAH dataset quantifies annotation quality through a per-clip certainty score derived from annotator agreement [1]; low-certainty clips indicate genuine perceptual difficulty rather than annotation error. Instance weighting by annotator confidence is a natural response: placing higher gradient weight on high-certainty samples directs the model toward reliable signal and reduces the cost of errors on ambiguous ones. Closely related are soft-label and label-distribution approaches in emotion recognition, which train models on probabilistic rather than one-hot targets to reflect inter-annotator spread. In CF-Net, we operationalise BAH certainty in two complementary ways: a per-sample focal loss weight \(w(c)\) that scales gradient contribution by annotator agreement, and an auxiliary regression head that is supervised to predict the certainty score, regularising the shared fused representation on ambiguous clips and preventing the model from committing confidently to low-certainty labels.

3 Method↩︎

1 gives an overview of CF-Net. We describe each component in turn.

3.1 Feature Extraction↩︎

Each video is processed offline to extract three per-video feature tensors that are cached before training.

Visual features. We sample one frame every 5 frames (at an assumed 30 fps), obtaining up to 256 frames per video. Face crops are extracted via a lightweight detector and resized to \(224\times 224\). Each crop is encoded by a frozen SigLIP2 (google/siglip2-base-patch16-224) [16], using its built-in attention-pooling head to produce a 768-dimensional embedding per frame. The full video yields a variable-length sequence \(\mathbf{V} \in \mathbb{R}^{T_v \times 768}\).

Audio features. The audio track is resampled to 16 kHz and split into 20 ms chunks overlapping by 10 ms. Each chunk is encoded by a frozen HuBERT (facebook/hubert-base-ls960) [17], and consecutive chunk embeddings are mean-pooled at the frame level, producing a sequence \(\mathbf{A} \in \mathbb{R}^{T_a \times 768}\).

Text features. The manually transcribed utterance is tokenised and encoded by a frozen DistilBERT [18] (distilbert-base-uncased), a knowledge-distilled variant of BERT [19], taking the mean-pooled token representation to give a single vector \(\mathbf{t} \in \mathbb{R}^{768}\).

3.2 Temporal Encoding and Pooling↩︎

Prior to temporal encoding, the visual and audio backbone features are subject to per-speaker normalisation: for each speaker in the training set, the mean embedding is computed across all their videos and subtracted from every frame of every video belonging to that speaker. This removes the constant speaker-identity offset (face appearance, voice timbre) from the features while preserving within-speaker temporal variation — the actual AH signal — and is applied identically to the test set using each test speaker’s own video mean.

The normalised visual and audio sequences are each passed through an independent bidirectional GRU [20] with one layer and hidden size 128 (output size 256 per direction, projected to \(d=128\) by a linear layer). Text is projected by a VectorEncoder: a single \(\operatorname{Linear}(768{\to}128) \to \operatorname{ReLU} \to \operatorname{Dropout}(0.3)\) stack. All three modalities are thus mapped to \(d=128\)-dimensional embeddings: \(\tilde{\mathbf{v}},\tilde{\mathbf{a}} \in \mathbb{R}^{T\times 128}\) and \(\tilde{\mathbf{t}} \in \mathbb{R}^{128}\).

Variable-length visual and audio sequences are pooled via MIL attention [21]: a learned query \(\mathbf{q} \in \mathbb{R}^{128}\) computes a softmax over frame-level compatibility scores, and the result is a weighted sum of frame embeddings. This allows the model to attend to a small subset of discriminative frames rather than committing to uniform temporal aggregation.

3.3 ConflictFusion↩︎

Let \(\mathbf{e}_v, \mathbf{e}_a, \mathbf{e}_t \in \mathbb{R}^{128}\) denote the pooled visual, audio, and text embeddings, respectively. Following the ConflictFusion paradigm [4], we construct the fused representation as \[\mathbf{f} = \bigl[ \mathbf{e}_v;\,\mathbf{e}_a;\,\mathbf{e}_t;\; |\mathbf{e}_v - \mathbf{e}_a|;\; |\mathbf{e}_v - \mathbf{e}_t|;\; |\mathbf{e}_a - \mathbf{e}_t| \bigr] \in \mathbb{R}^{768}, \label{eq:conflict}\tag{1}\] where \([\cdot\,;\cdot]\) denotes concatenation and \(|\cdot|\) is element-wise absolute value. The three pairwise difference terms explicitly encode cross-modal incongruence: a video in which the speaker’s words suggest positive affect while their voice sounds hesitant will produce a large \(|\mathbf{e}_t - \mathbf{e}_a|\) component, directly exposing the discriminative signal.

3.4 Classifier and Auxiliary Heads↩︎

The fused vector \(\mathbf{f}\) is passed to the main classifier: \[\hat{y} = \sigma\bigl( W_2\,\operatorname{ReLU}(W_1 \mathbf{f} + b_1) + b_2 \bigr), \label{eq:classifier}\tag{2}\] where \(W_1 \in \mathbb{R}^{256\times 768}\), \(W_2 \in \mathbb{R}^{1\times 256}\), with dropout (rate 0.3) applied between the two linear layers.

In parallel, an auxiliary certainty head predicts the per-video annotator certainty score \(c \in [1,3]\) from \(\mathbf{f}\): \[\hat{c} = W_4\,\operatorname{ReLU}(W_3 \mathbf{f} + b_3) + b_4, \label{eq:certainty-head}\tag{3}\] where \(W_3 \in \mathbb{R}^{256\times 768}\), \(W_4 \in \mathbb{R}^{1\times 256}\). This head is active only during training and is removed at inference. Its purpose is to regularise the fused representation on ambiguous samples: when annotators disagree strongly (low certainty), the network receives an explicit signal not to be over-confident about the main prediction.

3.5 Training Objective↩︎

The full training loss is \[\mathcal{L} = w(c)\,\mathcal{L}_{\mathrm{focal}}(\hat{y}, y) + \lambda_c\,\mathcal{L}_{\mathrm{MSE}}(\hat{c}, c), \label{eq:loss}\tag{4}\] where \(\mathcal{L}_{\mathrm{focal}}\) is the focal loss [22] with \(\gamma = 2.0\), \(\lambda_c = 0.3\), and \(w(c)\) is a per-sample certainty weight that maps the annotator certainty score \(c \in [1,3]\) to \([0.3,\,1.0]\): \[w(c) = 0.3 + 0.7 \cdot \frac{c - 1}{2}. \label{eq:certweight}\tag{5}\] High-certainty samples (annotators agreed strongly) receive up to \(3\times\) more gradient signal than low-certainty ones, encouraging the model to learn more from unambiguous examples while remaining tolerant of genuinely borderline annotations.

We apply two forms of regularisation during training. Manifold mixup [23], [24]: for each mini-batch we sample \(\lambda \sim \mathrm{Beta}(0.2, 0.2)\) and replace \(\mathbf{f}\) with \(\lambda \mathbf{f} + (1-\lambda)\mathbf{f}[\pi]\), where \(\pi\) is a random permutation. Modality dropout: each modality’s features are independently zeroed with probability 0.15, preventing over-reliance on any single stream and improving robustness to missing or noisy modalities.

3.6 Prediction↩︎

The final hard label is \(\mathbf{1}[\hat{y} \geq 0.5]\), where \(\hat{y}\) is the sigmoid output of the classifier head (Eq. 2 ).

4 Experiments↩︎

4.1 Dataset and Metrics↩︎

We use the BAH dataset [1] from the 3^rd ABAW AH challenge. The dataset contains 1,427 video clips from 300 speakers across three disjoint splits: 778 training, 124 validation, and 525 test videos. Importantly, speakers do not appear in more than one split, so generalisation to unseen speakers is the primary challenge. The positive class (AH\(\!=\!1\)) constitutes approximately 49.5% of training and 60.5% of validation and test videos.

Following the challenge protocol, we report Macro F1 as the primary metric, and Average Precision (AP) on the positive class as a secondary measure.

4.2 Implementation Details↩︎

All three backbone encoders (SigLIP2, HuBERT, DistilBERT) are kept frozen throughout training; only the BiGRU encoders, pooling modules, and classification heads are trained. We train for up to 30 epochs using Adam with learning rate \(3\times 10^{-4}\) and weight decay \(10^{-4}\), with a batch size of 8. Early stopping is applied with patience 8 on validation Macro F1. All experiments are run on a single NVIDIA GeForce RTX 4060 Ti (16 GB VRAM) using PyTorch 2.12 with CUDA 13.0. Training takes approximately 15 minutes per run. Random seed 1234 is used for all experiments.

4.3 Main Results↩︎

1 reports Macro F1 and AP on the validation set and the private challenge test set.

CF-Net achieves 80% No-AH recall and 67% AH recall on the validation set (3). 2 shows that predicted probabilities are well separated between classes, with AH samples concentrated above 0.5 and No-AH samples below 0.3. On the private challenge test set, CF-Net scores Macro F1 = 0.7364, a \(+\)​0.021 improvement over the validation result, confirming that per-speaker normalisation and certainty-weighted focal loss generalise to unseen speakers rather than overfitting the 124-video validation set.

4.4 Ablation Study↩︎

2 isolates the contribution of each component. All ablations use the CF-Net architecture as the base.

4.5 Comparison with Prior Work↩︎

3 compares CF-Net against published methods on both the official BAH validation split and the private challenge test set. Prior methods are drawn from the 10^th ABAW challenge (CVPR 2026) [10]; CF-Net was submitted to the 11^th ABAW challenge (ECCV 2026).

On the validation set, CF-Net (0.716) outperforms the organiser baseline by \(+\)​6.0 points; Team LEYA (0.854) leads through end-to-end fine-tuning but shows severe overfitting (0.854 to 0.714 on the private test). On the 10th ABAW private test, BROTHER achieves the best score (0.747) via ensemble optimisation, followed by ConflictAwareAH (0.715) and LEYA (0.714). CF-Net achieves 0.7364 on the 11th ABAW private test — a different held-out set — which is between ConflictAwareAH and BROTHER, as a single non-ensemble frozen-backbone model. The val-to-test improvement (\(+\)​0.021) confirms that per-speaker normalisation and certainty-weighted focal loss generalise to unseen speakers rather than memorising the validation set.

5 Conclusion↩︎

We presented CF-Net, a deep multimodal network for the ABAW 11^th Ambivalence/Hesitancy challenge. Recognising ambivalence and hesitancy from video is fundamentally different from prototypical emotion recognition: modalities are frequently incongruent, the validation set is small (124 clips), and the speaker-disjoint splits penalise any system that memorises visual or acoustic speaker identity. CF-Net addresses these pressures by encoding visual, audio, and transcript streams with frozen pre-trained backbones and fusing them via ConflictFusion, which explicitly represents cross-modal disagreement through pairwise absolute-difference terms. Training regularisation — certainty-weighted focal loss, manifold mixup, modality dropout, and an auxiliary certainty-regression head — prevents overfitting on the small training set.

The ablation study confirms the value of each component: replacing ConflictFusion with simple mean fusion drops Macro F1 by 1.8 pp; the auxiliary certainty head, focal loss, manifold mixup, and modality dropout together contribute a further 3.7 pp. Per-speaker feature normalisation, switching to a BiGRU temporal encoder, and adopting certainty-weighted focal loss contribute an additional 1.7 pp cumulatively, lifting CF-Net from 0.6984 to 0.7155 on the validation set. On the private challenge test set, CF-Net achieves 0.7364 Macro F1 (AP = 0.7392), a \(+\)​0.021 improvement over the validation result and better than the best published test score (0.714, Team LEYA [5]), while keeping all backbone weights frozen and training in under 15 minutes on a single consumer GPU.

Several directions remain open. Selective fine-tuning of the top layers of SigLIP2 or HuBERT under stronger regularisation could improve modality-specific feature quality without inducing speaker-identity memorisation. Replacing the fixed ConflictFusion concatenation with a learned cross-modal attention layer [14] may better weight which modality pairs are in conflict for each sample. Domain-adversarial training on the acoustic stream could further improve robustness to the zero-speaker-overlap constraint of the BAH dataset.

References↩︎