1 Introduction↩︎

Ambivalence and hesitancy (A/H) are relevant signals in digital behavior-change interventions because they indicate that a person may be uncertain about, or simultaneously attracted to and resistant toward, a proposed action. The Behavioural Ambivalence/Hesitancy (BAH) dataset [1] was created to support automatic analysis of these states from videos of participants answering questions designed to elicit A/H. The associated Affective & Behavior Analysis in-the-Wild (ABAW) challenge formulates the task as binary video-level recognition and ranks systems using macro-averaged F1.

A/H is challenging for two reasons. First, the relevant evidence may occur only during a short part of a video. A global temporal mean can therefore dilute the informative segment. Second, useful cues are distributed across modalities: pauses, speech timing, pitch variation, epistemic hedges, contrastive constructions, and conflicting positive and negative evaluations may all contribute to the final decision. These cues are especially important when the amount of labeled data is limited.

We develop an audio-text approach tailored to these properties. The visual stream is omitted to reduce complexity and to focus on speech and language. Videos are divided into overlapping 5-s windows that approximately match the duration of a typical A/H episode in BAH. For every window, we extract prosodic audio descriptors, an emotion-oriented text embedding, and an interpretable support vector grounded in psycholinguistic descriptions of uncertainty and ambivalence. Cross-attention aligns the audio and text representations, and gated MIL pooling assigns greater weight to the most informative windows. A five-seed ensemble reduces training variance.

The contributions are threefold: (1) an audio-text temporal architecture that combines cross-attention with multiple-instance learning (MIL) pooling for localized video-level A/H recognition; (2) a 74-dimensional support representation encoding question context, prosodic uncertainty, textual hedging, and ambivalence; and (3) an internal experimental analysis showing that these support features provide the largest observed improvement. We explicitly distinguish these internal results from the still-unreported official private-test performance.

2 Related Work↩︎

2.0.0.1 Vocal correlates of (un)certainty.

The prosody of felt (un)certainty is among the most robustly documented phenomena in the psychology of speech. Jiang & Pell [2] characterized the “sound of confidence and doubt”: confident utterances tend toward a falling terminal contour, higher intensity, a faster rate, and a lower, less variable fundamental frequency, whereas doubtful ones invert this profile—rising terminal intonation, lower and more unstable loudness, a slower rate, and greater pitch variability—and listeners decode these cues reliably. Brennan & Williams [3] showed that in question answering the response latency and filled pauses, together with intonation, let a listener infer the speaker’s metacognitive state (the “feeling of another’s knowing”). Smith & Clark [4] studied precisely the setting of BAH—spoken answers to questions—and found that a lower feeling-of-knowing yields longer latencies, more hedges and fillers, and rising intonation. Swerts & Krahmer [5] further confirmed that pauses and rising intonation are dominant audiovisual markers of uncertainty. Two further lines make these correlates measurable: automatic speech- and articulation-rate estimation from syllable-nucleus detection [6], validated against human fluency ratings [7]; and the compact paralinguistic descriptors of vocal quality—jitter and shimmer—distilled by the eGeMAPS parameter set [8].

2.0.0.2 Hedging and epistemic markers in text.

On the language side, hedging—the linguistic marking of epistemic commitment—has a long computational tradition, culminating in the CoNLL-2010 shared task on detecting uncertainty cues and their scope [9]. A central lesson of that task is that candidate cue words (“may”, “suggest”, “I think”) are hedges only in context, so robust detection requires learned classification rather than dictionary lookup. Ganter and Strube [10] obtained hedge detectors by distant supervision from Wikipedia weasel-word tags plus shallow features, a recipe that transfers to label-scarce settings.

2.0.0.3 Attitudinal ambivalence and mixed emotion.

Ambivalence is not equivalent to neutral sentiment. Social psychology formalizes it bidimensionally: Kaplan [11] posed the ambivalence–indifference problem—a net (positive minus negative) score cannot separate an indifferent attitude (\(P,N\) both low) from a conflicted one (\(P,N\) both high)—so the positive and negative components must be measured separately, with \(\min(P,N)\) as a conflict floor. Thompson et al. [12] formalized the similarity–intensity index \((P{+}N)/2 - |P{-}N|\), which is high only when both poles are strong; Conner et al. [13] further separate mere cognitive–affective inconsistency from felt ambivalence. In parallel, multi-label emotion recognition treats opposite-valence emotions as co-occurring rather than mutually exclusive: MEDA [14] models inter-emotion correlations explicitly, capturing the mixed states that underlie ambivalence.

2.0.0.4 Our support features.

These strands motivate a support-feature block engineered to mark the A/H state rather than generic paralinguistics or clinical disfluency. On the acoustic side, we operationalize the uncertainty correlates described above, including onset latency, pause statistics, articulation rate, F0 variability with a terminal-rise term, loudness level and instability, jitter, and shimmer. Greater F0 variability and a rising terminal contour differ from the flat pitch often associated with produced filled pauses; accordingly, we target the underlying state rather than the disfluency event itself and omit filler counts that are largely discarded by the ASR front end. On the textual side, we compute the Kaplan and Griffin ambivalence indices from separated positive/negative intensities (from a sentiment lexicon [15] and an emotion model, never a net polarity), emotion-distribution entropy and cross-window valence fluctuation, discourse-contrast and polarity-shift markers, and graded epistemic hedges. Rather than concatenating these descriptors after temporal pooling—where they are diluted by the high-dimensional embeddings—we inject them per window via token fusion, so they enter the cross-attention representation and inform the attention-MIL pooling weights.

2.0.0.5 Multimodal fusion and MIL.

Attention mechanisms provide a flexible way to model interactions between modalities [16]. For video-level labels with sparse temporal evidence, multiple instance learning (MIL) is a natural formulation: a video is treated as a bag of windows, and an attention-based pooling mechanism learns which instances are most relevant [17]. Our model combines these ideas by using audio-to-text cross-attention at the window level and gated MIL across windows.

3 Proposed Method↩︎

Figure 1 provides an overview of the proposed architecture, which integrates audio-text cross-attention, psycholinguistic support features, gated MIL pooling, and a video-level classifier.

3.1 Pre-processing and temporal alignment↩︎

The audio track is extracted from each video, converted to mono 16-kHz FLAC, and segmented into 5-s segments with a 2.5-s hop. The BAH transcripts include Whisper-generated text and timestamps. We use these timestamps to concatenate all transcript segments overlapping each audio window. Timestamp resets at internal Whisper segment boundaries are corrected by accumulating an offset, producing a monotonic timeline.

The target is the global video label \(y_v\in\{0,1\}\). The released participant-wise splits prevent the same participant from appearing in more than one partition. The complete video is represented as a variable-length sequence of \(T\) aligned windows.

3.2 Window representations↩︎

Each window \(t\) is represented by audio \(\mathbf{a}_t\in\mathbb{R}^{320}\), text \(\mathbf{x}_t\in\mathbb{R}^{768}\), and support features \(\mathbf{s}_t\in\mathbb{R}^{74}\).

3.2.0.1 Audio.

The 320-dimensional descriptor is computed with librosa and contains MFCCs and their first- and second-order derivatives, spectral centroid, bandwidth, roll-off, flatness and contrast, chroma, zero-crossing rate, RMS energy, pitch statistics, voiced fraction, and tempo. Frame-level descriptors are summarized by mean, standard deviation, minimum, and maximum whenever applicable. We selected this prosodic representation after internal experiments in which generic self-supervised speech embeddings did not improve performance.

3.2.0.2 Text.

Text is encoded with cardiffnlp/twitter-roberta-base-emotion, using masked mean pooling followed by \(\ell_2\) normalization. The resulting 768-dimensional vector captures semantic and affective information in each window [18], [19].

3.2.0.3 Psycholinguistic support features.

Table 1 summarizes the 74-dimensional support vector. Seven dimensions encode the question type. Eighteen acoustic dimensions describe timing, pauses, speaking rate, pitch, loudness, jitter, shimmer, and a composite uncertainty score. Forty-nine textual dimensions encode positive/negative intensity, psychometric ambivalence indices, emotional dynamics, contrast and polarity shifts, and epistemic hedges. Response-level measures are broadcast to all windows of a video, while local hedge, contrast, entropy, and prosodic measures remain window-specific.

3.3 Cross-attention and token fusion↩︎

Audio and text are independently projected to a common width \(d=512\). Multi-head cross-attention uses the audio sequence as query and the aligned text sequence as key/value: \[\mathbf{H}=\mathrm{LN}\!\left(\mathbf{A}+\mathrm{MHA}(\mathbf{A},\mathbf{X},\mathbf{X})\right),\] where padding windows are masked, and LN denotes layer normalization.

The heterogeneous support features are normalized over valid windows only, projected to 512 dimensions, and concatenated with the cross-modal representation before another linear projection: \[\widetilde{\mathbf{h}}_t=\mathrm{LN}\left(W_f[\mathbf{h}_t\Vert g(\mathbf{s}_t)]\right).\] This token fusion places the interpretable cues before temporal aggregation. Consequently, a window containing strong uncertainty or conflict cues can receive greater MIL attention. In internal trials, this placement was more effective than concatenating the support vector only after video pooling.

3.4 MIL pooling, classifier, and ensemble↩︎

Gated attention pooling assigns a score to each valid window [17]: \[\begin{align} q_t &= \mathbf{w}^{\top}\!\left[\tanh(V\widetilde{\mathbf{h}}_t)\odot\sigma(U\widetilde{\mathbf{h}}_t)\right],\\ \alpha_t &= \frac{\exp(q_t)}{\sum_{j=1}^{T}\exp(q_j)}, \qquad \mathbf{z}=\sum_{t=1}^{T}\alpha_t\widetilde{\mathbf{h}}_t. \end{align}\] The video vector \(\mathbf{z}\) is processed by a 512-unit ReLU layer, a dropout of 0.3, and a binary output layer. The model is optimized with binary cross-entropy and AdamW (learning rate \(3\times10^{-4}\), weight decay 0.05), a batch size of 32, gradient clipping at 1.0, and early stopping based on the validation average precision.

Five models with seeds \(\{42,1,2,3,4\}\) are trained with identical settings. Their video probabilities are averaged before thresholding. The threshold is selected from a grid with a step of 0.01 by locating a stable near-maximum region of the macro-F1 curve.

4 Experimental Protocol↩︎

The BAH dataset [1] contains 1,427 labeled videos from 300 Canadian participants. The released splits contain 778 training, 124 validation, and 525 public-test videos. Our final internal configuration fits model weights on the training set, while the threshold for the reported macro-F1 is calibrated on the validation set.

The official challenge metric is video-level macro-F1 across the positive and negative classes. We additionally report positive-class AP. All values below were produced by the supplied implementation and should be interpreted as internal results only.

5 Results and Discussion↩︎

Table 2 reports the main internal results. Replacing temporal mean pooling with attention-based MIL while retaining only audio and text yields an AP of 0.835. Adding the support features increases AP to 0.875, a gain of 0.040. Under the same development-set threshold-calibration procedure, macro-F1 rises from 0.705 to 0.722. The ensemble threshold is 0.50 and yields class-wise F1 values of 0.678 for the negative class and 0.766 for the A/H class.

The improvement supports the central hypothesis of this work: generic embeddings do not fully expose the cues most directly associated with A/H, whereas explicit uncertainty and ambivalence descriptors provide a useful inductive bias. Injecting them before MIL pooling is particularly relevant because the support vector influences both each token’s content and its contribution to the video representation.

The attention-based aggregation is also consistent with the temporal sparsity of A/H. Mean pooling gives every window equal weight, including long neutral stretches. MIL instead allows a few windows containing pauses, rising intonation, hedges, or conflicting evaluations to dominate the decision. The learned attention weights may additionally support future interpretability analyses by indicating when the model detected evidence of A/H.

Several internal alternatives did not improve the final configuration. Replacing the handcrafted prosodic representation with wav2vec 2.0- or HuBERT-style embeddings reduced macro-F1 in our trials, suggesting that generic phonetic representations may suppress some of the low-level timing and pitch information required here. Adding a second text encoder also increased complexity without a reliable gain. Because the dataset is modest, seed variance was noticeable; probability averaging across five models produced more stable scores and a smoother threshold curve.

The current system excludes facial behavior, which may provide complementary evidence in difficult cases. The reported results therefore establish an internally validated architecture, not an official challenge ranking.

6 Conclusions↩︎

We introduced an audio-text model for video-level ambivalence/hesitancy recognition. The approach aligns speech and transcripts within overlapping windows, combines prosodic and emotion-oriented embeddings via cross-attention, injects psycholinguistic support features prior to aggregation, and employs gated MIL pooling with a five-seed ensemble. Internal experiments show that the support vector is the most effective component, increasing AP from 0.828 to 0.875 and the calibrated development macro-F1 from 0.701 to 0.722. Official private-test results should be added only after they are returned by the challenge organizers.

References↩︎