1 Introduction↩︎

Behaviour-change interventions fail silently when a person says they will act but does not mean it. The Behavioural Ambivalence/Hesitancy (BAH) dataset [1] was collected to study exactly this phenomenon: 300 participants answering seven predefined behaviour-related questions on video, annotated by experts with video-level A/H labels. The associated ABAW challenge [2] asks for a binary video-level decision, scored by macro-F1 over both classes on a held-out, participant-disjoint test set. In the previous (10th ABAW) edition, five teams competed on this task, using heterogeneous ensembling [3], cross-modality feature differences [4], reliability-weighted fusion [5], a fine-tuned MLLM [6], and a second cross-modality-difference system [7]. Our own system (2), a calibrated ensemble of three fusion models over frozen multimodal embeddings, reaches 0.7358 macro-F1 on this year’s public test set.

This year’s edition introduces a genuinely new private test: 30 participants disjoint from anyone previously used, with only five submissions allowed before the deadline. 4 describes the five configurations we selected, each covering a different failure mode of our main system, together with the outcome of the one submission already scored; 5 grounds these choices in an analysis of which modalities and cues actually drive the model’s decisions.

2 System↩︎

2.1 Frozen multimodal embeddings and fusion↩︎

Training end-to-end on 758 videos invites overfitting: early experiments with partially unfrozen backbones improved validation while degrading test. We therefore precompute frozen per-video embeddings for four modalities. Face: 128 uniformly sampled aligned face crops, encoded by a Vision Transformer [8] fine-tuned for facial expression recognition on FER2013 [9] (\(128 \times 768\)). Audio: HuBERT-base [10] over 10-second chunks, mean-pooled per chunk. Text: the CLS embedding of a RoBERTa [11] model fine-tuned on GoEmotions [12], applied to the Whisper transcript [13] (768). Pose: ViTPose keypoints [14] (17 COCO joints times \((x,y,\text{conf})\)) for 128 uniform frames (\(128 \times 51\)). Sequences are pooled by learned attention pooling and fused by a one-round cross-attention block, each modality querying the others, followed by a residual connection, LayerNorm, and a small MLP classification head.

2.2 Three ensemble members, and calibration↩︎

Our submitted system averages the logits of three models trained on the same frozen embeddings but different architectural biases. CrossAttn (face, audio, text) is the reference model: attention pooling per modality feeding the cross-attention fusion block, with moderate regularization. Reliability (face, audio, text) replaces cross-attention with a gating network that predicts a softmax weight per modality before summing them; it consumes the same input as CrossAttn, so its contribution to the ensemble is small. Pose (face, audio, text, pose) is the only member with body-pose keypoints, using stronger regularization (dropout 0.5, weight decay 0.05, label smoothing 0.1) to control overfitting on the added input; being the only member with genuinely different input, it contributes most of the ensemble’s gain over its best individual member. All three are trained with binary cross-entropy, AdamW, cosine learning-rate scheduling, and early stopping on validation macro-F1.

Validation contains only 124 videos, so any quantity tuned on it is noisy. We restrict validation-tuned quantities to a single global decision threshold and one temperature scalar \(T\) fitted by negative log-likelihood minimization [15] on validation logits (\(T = 1.10\), threshold \(= 0.402\)). Both are frozen before any test split, public or private, is scored.

3 Results on the Public Test Set↩︎

1 reports the baseline, the five previous-edition solutions as published in their own papers, our individual models, and every ensemble variant we evaluate, on our public test split (1).

Ablation. The ensemble’s gains decompose transparently. CrossAttn and Reliability, which share the same input, add only \(+0.002\) macro-F1 when combined: a different fusion mechanism over identical information buys almost nothing. Adding Pose, the only member with genuinely different information, contributes the remaining \(+0.012\). Temperature calibration on top of the equal-weight ensemble adds a further \(+0.0022\) (0.7336 \(\to\) 0.7358), for free at inference time. Separately, on a single model (CrossAttn, trained in isolation), replacing binary cross-entropy with a differentiable macro-F1 surrogate (soft-F1 [17], mixed with BCE at weight 0.3) lifts its score from 0.7219 to 0.733, the largest single-model gain we found; this gain does not compose across the ensemble (each member’s calibration shifts differently under this loss), so it stays outside the main ensemble but is strong and diverse enough on its own to warrant submission 3 (4).

4 Five Submissions for the Private Test↩︎

This year’s private test was released on 2026-07-10: 30 participants (152 videos) disjoint from training, validation, and the public test, with five submissions allowed before the deadline. Because this is the first time we face a private test whose distribution we cannot inspect, our five submissions are not five attempts at the same best model; each is chosen to remain competitive under a different assumption about how the private test might behave relative to the public test, ordered by validated strength.

Submission 1: calibrated 3-member ensemble (\(T{=}1.10\), threshold \(0.402\), 1). Our best validated system, and the default choice: if the private test behaves similarly to the public test, this submission wins.

Submission 2: submission 1 with the decision threshold fixed at \(0.5\). A hedge against threshold-transfer failure. The val-fitted threshold (\(0.402\)) may not carry over if the private test’s score distribution shifts, whereas \(0.5\) is a distribution-agnostic default; the public-test F1 curve for this system is flat between threshold \(0.3\) and \(0.6\), so we expect a negligible change on the public test, and include it for coverage rather than an expected gain.

Submission 3: the soft-F1 individual model, not part of the ensemble (3). Our best single model, trained with a different loss geometry. It covers the scenario where the ensemble-and-fusion mechanism has jointly overfit to the validation/public-test distribution; among our five submissions it is the least correlated with the ensemble-based ones.

Submission 4: calibrated ensemble plus cache-personalization TTA [18], global mode. The only submission designed specifically for a real distribution shift (new participants, different recording conditions). We saw no gain from this on the public test (\(0.725\) vs.\(0.7358\)), because the public test turned out not to be harder than validation, but the private test is a genuine unknown, unlike the public test, whose behavior we have already characterized directly.

Submission 5: the Pose member alone (face, audio, text, pose). The only component with genuinely different input information (pose keypoints), which may transfer better to new participants than appearance-based features. It covers the scenario where the fusion or ensembling mechanism itself has overfit, while the strongest individual component holds up.

2 tracks the public-test estimate and the private-test outcome for all five submissions as they are scored.

Submission 1 outcome. As of this writing, submission 1 has already been used as a trial run and scored by the organizers: macro-F1 \(= 0.7361\), average precision (positive class) \(= 0.7677\), on the actual private test. This macro-F1 essentially matches our honest public-test estimate (0.7358) almost exactly, confirming that the whole pipeline, embedding extraction, inference, and frozen calibration, generalizes correctly to 30 entirely new participants and was not overfit to the public 511-video test. The average precision is noticeably lower than on the public test (\(\approx\)​0.835 for the same ensemble), indicating the hard-threshold decision holds up well while the underlying probability ranking is somewhat weaker on these new participants; this is consistent with the rationale for submissions 2 and 4 above, which specifically hedge against threshold and distribution drift.

5 Analysis and Explainability↩︎

The organizers call for interpretable solutions that highlight when A/H occurs and which modalities and cues drive decisions. We answer with controlled measurements that isolate which modality carries the signal, how it is used, and where our system still fails, since this directly explains the diversification logic behind submissions 3 and 5.

Which modality carries the signal. Trained end to end with our best pooling, a text-only model reaches 0.716 macro-F1, within noise of the full multimodal system, while a face-plus-audio-only model, with no transcript at all, saturates at 0.60. The full model’s errors correlate at 0.91 with the text-only model’s errors, so functionally the multimodal system behaves like a text model with non-verbal refinements. This is not because non-verbal cues are irrelevant to A/H: annotators cited facial cues on 550 occasions and body cues on 458 in the dataset’s free-text rationale. Rather, current frozen encoders extract little of that signal, and what they do extract is largely redundant with text, because Whisper transcribes the behaviour of speech (fillers, hedges, self-corrections survive transcription), so the text branch already implicitly encodes part of the audio channel. This is precisely why submission 5, which adds pose, the one modality with information genuinely disjoint from text, is included as a hedge: it is the ensemble member most likely to hold up if this year’s participants shift the text/non-verbal balance.

Style, not topic: a causal probe. Each BAH video answers one of seven questions, and the A/H prior varies enormously by question, from 0.19 for question 1 to 0.92 for question 4 (1a). Is the text branch merely exploiting topic as a proxy for the question prior? We test this causally with an LLM that rewrites every transcript onto a different, neutral topic while preserving delivery style (hedges, fillers, sentence rhythm, and self-corrections). A classifier trained on these topic-swapped transcripts loses almost nothing, 0.706 versus 0.712 on the originals: the model reads how things are said, not what they are about. Consistent with this, simple lexical markers are strongly predictive on training data: given that the transcript contains “I mean”, \(P(\text{A/H}) = 0.81\), and hedges such as “maybe”, “I guess”, or “sort of” give \(0.70\), against a base rate of \(0.49\).

Where the task is decided. 1a decomposes performance by question. A prior-only classifier, predicting from question identity alone with no video input, already reaches 0.653 macro-F1; our advantage over this trivial baseline concentrates in the four mid-prior questions (Q2, Q3, Q5, Q6, \(n=292\)), scoring 0.647 against 0.847 on the easy questions. Temporally, A/H is localized: annotated spans cover a median of around 20% of frames, and our miss rate rises sharply for short spans, 50% under 5% of the video against 22% over 40%, a concrete failure mode that persists across every submission configuration.

Why the remaining errors are, mostly, not fixable. Of our 132 public-test errors, false negatives concentrate overwhelmingly on videos whose annotators were themselves unsure (1b): the miss rate falls from 49% at certainty level 1 to 6% at level 3, so roughly half our residual error is irreducible label ambiguity, not a gap the system could close. The recoverable remainder is well characterized: 29 false negatives are high-certainty positives with no verbal hedging at all, hesitancy expressed purely non-verbally, exactly where the measured non-verbal ceiling of 0.60 binds, and a further 49 false positives sit within 0.25 of the decision boundary, which post-hoc calibration partially addresses (submission 1’s \(+0.0022\) over the uncalibrated ensemble, 3). Neither failure mode is specific to the public test, so we expect the same error structure, dominated by genuine label ambiguity and a hard non-verbal ceiling, to persist on the private test regardless of which submission is scored.

6 Conclusion↩︎

We presented a calibrated, equal-weight ensemble of three fusion models over frozen face, audio, text, and pose embeddings that reaches 0.7358 macro-F1 on the BAH public test set, and described the five-submission strategy we designed for this year’s genuinely new private test. Our first submission, an unmodified transfer of the validated system, already scored 0.7361 macro-F1 on the private test, closely matching our public-test estimate and validating the leak-free, threshold-frozen protocol behind every number in this paper. The remaining four submissions trade a small amount of expected score for coverage against threshold drift, ensemble overfitting, and genuine distribution shift, risks we cannot rule out until the private test is fully scored.

Acknowledgments↩︎

References↩︎