1 Introduction↩︎

The 11th Affective Behavior Analysis in-the-wild (ABAW) competition includes a Multi-Task Learning (MTL) challenge in which each facial frame is associated with three complementary affect representations: continuous valence-arousal (VA) values, an eight-way expression label, and twelve binary Action Unit (AU) labels. This setting is challenging because the three targets differ in output type and supervision density, and because s-Aff-Wild2 exhibits severe class imbalance (an 8\(\times\) expression imbalance and a 29\(\times\) AU imbalance) while joint three-task annotation covers only 37% of training frames.

We present AffectFlow-DINO, a frame-level MTL system that augments a DINOv3 ViT-S/16 backbone with a conditional rectified-flow head [1] to model the full joint affect distribution \(p(y \mid x)\). The core motivation is the inherent perceptual ambiguity of in-the-wild facial behavior: a subtle smile, partial occlusion, or low-intensity expression may correspond to several plausible configurations in the joint VA, expression, and AU space, yet deterministic point estimates collapse this ambiguity into a single vector and discard the uncertainty. The rectified-flow head learns instead a generative transport map from noise to \(p(y \mid x)\), producing a family of plausible affect vectors per image whose mean improves over the point estimate and whose spread characterizes prediction uncertainty. Our main contributions are:

• A masked rectified-flow MTL objective that jointly optimizes deterministic task heads and a generative conditional distribution over the 22-dimensional affect vector.

• A systematic ablation study spanning 26 design choices on s-Aff-Wild2, covering flow loss weight, inference efficiency, imbalance remedies, backbone fine-tuning, and post-hoc calibration strategies.

• Post-hoc per-AU and per-class expression threshold calibration, yielding a combined \(P_{MTL}=1.177\) on the validation split, a \(+0.054\) improvement from calibration alone, driven by recovering Fear F1 from \(3.8\%\) to \(33.1\%\) and Sadness from \(17.1\%\) to \(28.2\%\).

2 Related Work↩︎

2.0.0.1 ABAW and Aff-Wild lineage.

The Aff-Wild/Aff-Wild2 benchmark series introduced in-the-wild dimensional affect prediction and progressively combined the three core targets, continuous valence-arousal (VA), categorical expressions, and binary Action Units (AUs), into a joint multi-task challenge [2]–[5]. ABAW1–6 built on this foundation, expanding dataset scale, task coverage, and evaluation protocols [6]–[11]; the 7th ABAW competition then established the static-frame, three-task MTL format with the summed metric \(P_{MTL}\) used in the current 11th ABAW challenge [12]. ABAW8–10 expanded toward multimodal and fine-grained behavior analysis, and ABAW11 continues this trajectory while retaining the VA+EXPR+AU MTL track [13]–[15].

2.0.0.2 Datasets.

s-Aff-Wild2 [16] is the benchmark for this challenge; related datasets (AffectNet, RAF-DB, DFEW, EmotioNet, BP4D, DISFA, and others) cover partial subsets of the task triad and are summarized in Appendix 7 (Table 7) [17]–[27].

2.0.0.3 MTL for affect.

Early MTL models showed that VA, expressions, and AUs are complementary representations learnable via shared encoders [5], [28]–[30]. Practically, label incompleteness, task-structure heterogeneity (regression vs.classification vs.multi-label detection), and negative transfer between tasks require per-task masking, selective fusion, and carefully tuned loss weights [31]. Recent ABAW solutions combine strong pretrained backbones (MAE, DINOv2) with task-aware fusion, staged training, and temporal modeling [32]–[37].

2.0.0.4 Generative and uncertainty-aware prediction.

Diffusion and flow-matching models applied to structured prediction have shown that generative transports over label space can outperform deterministic regressors when targets are inherently multimodal; in affective computing, uncertainty in VA estimation has been addressed via Gaussian heads and evidential regression, but these scalar-variance extensions cannot capture multimodal ambiguity across the joint affect space. Conflict-aware fusion approaches specifically highlight that ambivalent facial behavior, where modality cues disagree, calls for richer predictive models than point estimates [38]. Rectified flow [1] offers an elegant solution: a straight-line transport from noise to data that enables fast few-step inference over the joint 22-dimensional affect target, to our knowledge the first such application in a heterogeneous MTL setting.

2.0.0.5 SOTA and direct comparisons.

The most directly comparable public SOTA is the 7th ABAW MTL leaderboard (same task triad and metric); ABAW8/9 use different task splits and are not directly comparable [13], [14]. The leading entry reaches \(P_{MTL}=1.529\) via progressive staged training, task-selective fusion, and temporal context [34]; the official challenge baseline is \(P_{MTL}=0.34\) [12].

3 Method↩︎

3.1 Overview↩︎

We propose AffectFlow-DINO, a frame-level multi-task model for the ABAW 2026 MTL challenge. Since s-Aff-Wild2 distributes data as isolated cropped face images rather than video sequences, designing a strictly frame-level system is the faithful choice for this benchmark; temporal sequence reconstruction is treated as an orthogonal future extension rather than a limitation of the current scope. Given a cropped face image \(x\), the model simultaneously predicts continuous valence-arousal (VA), categorical facial expression (EXPR), and binary Action Unit (AU) activations. Beyond deterministic point prediction, it learns a conditional rectified-flow distribution over the joint affect representation, enabling uncertainty-aware, one-to-many prediction at inference.

The core motivation is the inherent perceptual ambiguity of in-the-wild facial behavior: a subtle smile, partial occlusion, or low-intensity expression may correspond to several plausible VA and AU configurations. Rather than collapsing this uncertainty to a single point estimate, AffectFlow learns a generative transport map from isotropic Gaussian noise to the conditional target distribution \(p(y \mid x)\). This produces a family of plausible affect vectors per image, whose mean improves over deterministic prediction while their spread reflects prediction uncertainty.

The model comprises four components, illustrated in Figure 1:

1. a frozen DINOv3 ViT-S/16 backbone that extracts rich facial representations;

2. a shared projection head mapping backbone features into a compact affect embedding;

3. three deterministic task heads for VA regression, expression classification, and AU detection;

4. a conditional rectified-flow head that models the joint affect distribution conditioned on the image embedding.

3.2 Joint Affect Representation and Masked Supervision↩︎

Each training sample consists of a cropped, aligned face frame \(x_i\) paired with up to three partially available annotations. We represent the full affect target as a 22-dimensional vector: \[y_i = \bigl[v_i,\, a_i,\, e_i^{(1)}, \ldots, e_i^{(8)},\, u_i^{(1)}, \ldots, u_i^{(12)}\bigr] \in \mathbb{R}^{22}, \label{eq:joint95target}\tag{1}\] where \(v_i, a_i \in [-1,1]\) are valence and arousal, \(e_i \in \{0,1\}^8\) is a one-hot expression vector over eight categories, and \(u_i \in \{0,1\}^{12}\) is a binary AU vector. Embedding heterogeneous targets, continuous VA, categorical expressions, and binary AUs, into a single \(\mathbb{R}^{22}\) space is a deliberate design choice that enables a unified Euler integration over a shared continuous manifold (Eq. 10 ). Categorical and binary dimensions are treated as unconstrained real-valued scores during the flow trajectory; discrete predictions are recovered only at the final inference step via argmax and sigmoid thresholding (Eq. 11 ), avoiding per-task generative processes at the known cost of trajectories visiting off-manifold intermediate values, a trade-off shared with continuous label-space generative models. Step-wise projection is deliberately avoided since rectified flow’s straight-line trajectories reliably arrive on the target manifold at \(t{=}1\). The s-Aff-Wild2 annotations are incomplete: a significant fraction of frames are annotated for only one or two tasks. We therefore construct a per-sample binary validity mask \(m_i \in \{0,1\}^{22}\) and apply it to every loss term, ensuring that missing targets neither contribute gradients nor distort the flow training signal.

3.3 DINOv3 Visual Backbone↩︎

The visual encoder is a DINOv3 ViT-S/16 kept frozen throughout training. Its self-supervised DINO objectives [33] enforce spatial semantic consistency across patch tokens, unlike reconstruction-based pre-training such as MAE [32], making it well-suited to Action Unit detection, where muscle activations manifest as spatially localised deformations. For each face image \(x_i\) resized to \(224 \times 224\), the encoder extracts a \(d\)-dimensional feature vector from the [CLS] token: \[h_i = f_{\theta}(x_i) \in \mathbb{R}^{d}. \label{eq:backbone}\tag{2}\] A shared projection head then maps \(h_i\) to the working embedding dimension: \[z_i = \phi\bigl(W_h\, h_i\bigr), \label{eq:projection}\tag{3}\] where \(\phi\) denotes LayerNorm followed by GELU activation, and \(W_h \in \mathbb{R}^{D \times d}\) is a learned linear projection with \(D=768\).

3.4 Deterministic Multi-Task Heads↩︎

Three task-specific two-layer MLP heads with LayerNorm, GELU activation, and dropout are attached to the shared embedding \(z_i\): \[\begin{align} \hat{y}^{VA}_i &= \tanh\!\bigl(\mathrm{MLP}_{VA}(z_i)\bigr) \in [-1,1]^2, \tag{4}\\ \hat{y}^{EXPR}_i &= \mathrm{MLP}_{EXPR}(z_i) \in \mathbb{R}^{8}, \tag{5}\\ \hat{y}^{AU}_i &= \mathrm{MLP}_{AU}(z_i) \in \mathbb{R}^{12}. \tag{6} \end{align}\] The \(\tanh\) nonlinearity in Eq. 4 constrains VA predictions to the official annotation range. Expression logits in Eq. 5 are trained with masked cross-entropy, and AU logits in Eq. 6 with masked binary cross-entropy. AU activations are obtained at inference by thresholding the sigmoid output.

3.5 Conditional Rectified Flow↩︎

The deterministic heads yield accurate point predictions but cannot represent the uncertainty and multimodality inherent in ambiguous facial images. The AffectFlow component addresses this by learning a conditional generative model \(p(y \mid x)\) via rectified flow [1].

3.5.0.1 Training.

For each sample \((x_i, y_i)\), we draw noise \(\epsilon \sim \mathcal{N}(0, I_{22})\) and a time scalar \(t \sim \mathcal{U}(0,1)\), and form the linear interpolant: \[y_t = (1 - t)\,\epsilon + t\,y_i. \label{eq:interpolant}\tag{7}\] The rectified-flow objective minimizes the squared error between the predicted and the true constant velocity field \(v^* = y_i - \epsilon\): \[\mathcal{L}_{\text{flow}} = \frac{ \displaystyle\sum_{j=1}^{22} m_i^{(j)}\Bigl(\hat{v}_i^{(j)} - v_i^{*(j)}\Bigr)^2 }{ \displaystyle\sum_{j=1}^{22} m_i^{(j)} + \varepsilon }, \label{eq:flow95loss}\tag{8}\] The mask \(m_i\) ensures that absent labels do not contribute to the flow training signal. Formally, zeroing the missing velocity dimensions supervises only the annotated marginal \(p(y_{\mathcal{T}} \mid x)\) (where \(\mathcal{T}\) denotes the annotated task subset for that frame); under the mild assumption that annotation missingness is independent of true affect values, each masked step is a valid update for the full joint \(p(y \mid x)\), and the union of annotated marginals across the dataset collectively constrains the full joint conditional while preserving its marginal consistency. This assumption is not fully met in practice: if rare expressions or AUs are systematically less likely to be annotated (e.g.because annotators skip ambiguous or low-intensity frames disproportionately common to those classes), the annotated marginal is a biased estimate of the true one, and the learned flow could inherit that bias rather than merely inheriting the class imbalance itself. The velocity \(\hat{v}_i\) is predicted by a flow network \(r_\psi\): \[\hat{v}_i = r_\psi\bigl([z_i,\; y_t,\; \gamma(t)]\bigr), \label{eq:flow95network}\tag{9}\] with \(\gamma(t)\) a sinusoidal time embedding and \([\cdot]\) denoting concatenation.

3.5.0.2 Inference.

To obtain a prediction for image \(x\), we draw \(N\) independent noise samples \(\{\epsilon^{(n)}\}_{n=1}^N\), each initialized at \(y_0^{(n)} = \epsilon^{(n)}\). Each sample is propagated forward with \(T\) Euler integration steps of step size \(\Delta t = 1/T\): \[y_{k+1}^{(n)} = y_k^{(n)} + \Delta t\; r_\psi\!\bigl([z,\; y_k^{(n)},\; \gamma(t_k)]\bigr), \quad k = 0, \ldots, T-1, \label{eq:euler}\tag{10}\] where \(t_k = k\,\Delta t\). The final predictions are obtained by averaging over the \(N\) trajectories at \(t=1\): \[\begin{align} \bar{y} &= \tfrac{1}{N}\sum_{n=1}^N y_T^{(n)}, \label{eq:average}\\ \hat{y}^{VA} &= \text{clamp}\bigl(\bar{y}_{1:2},\; -1, 1\bigr), \nonumber\\ \hat{y}^{EXPR} &= \arg\max\bigl(\bar{y}_{3:10}\bigr), \nonumber\\ \hat{y}^{AU}_k &= \mathbf{1}\bigl[\sigma(\bar{y}_{10+k}) > \tau\bigr], \quad k=1,\ldots,12. \nonumber \end{align}\tag{11}\] The threshold \(\tau\) is a tunable decoding parameter; we ablate it across validation data in Section 4. This inference procedure contrasts fundamentally with deterministic MTL: the same face image generates a distribution over affect states, providing uncertainty estimates alongside the official prediction.

3.6 Training Objective↩︎

The full training objective is a weighted combination of the deterministic multi-task loss and the flow loss: \[\mathcal{L} = \mathcal{L}_{\text{det}} + \beta\,\mathcal{L}_{\text{flow}}, \label{eq:total95loss}\tag{12}\] where \[\mathcal{L}_{\text{det}} = \lambda_{VA}\,\mathcal{L}_{VA} + \lambda_{EXPR}\,\mathcal{L}_{EXPR} + \lambda_{AU}\,\mathcal{L}_{AU}. \label{eq:det95loss}\tag{13}\] The flow weight \(\beta\) controls the balance between learning a high-quality generative model and optimizing task metrics directly. An ablation over \(\beta \in \{0.25, 0.5, 1.0, 2.0\}\) (Appendix 8.1) finds that \(\beta \in \{0.5, 1.0\}\) achieve equivalent best validation \(P_{MTL}\); we adopt \(\beta = 1.0\) as the default. The model is optimized with AdamW and the best checkpoint is selected by validation \(P_{MTL} = P_{VA} + P_{EXPR} + P_{AU}\).

4 Experiments↩︎

4.1 Experimental Setup↩︎

All models are trained on the official ABAW 2026 frame-level training split of s-Aff-Wild2 and evaluated on the provided validation split. The training set contains 142,382 frames with partially available labels: 103,917 valid VA, 90,645 valid expression, and 103,316 valid AU rows. The validation set contains 26,876 frames with full VA and AU labels and 15,440 valid expression labels. All invalid labels are excluded via per-task binary masks as described in Section 3.

Label statistics and class imbalance. Expression labels exhibit an 8\(\times\) imbalance: Other (27.4%), Neutral (26.5%), and Happiness (20.0%) dominate, while Fear (3.4%) and Disgust (3.5%) together account for under 7% of valid labels. For AUs, AU25 is active in 68% of valid training rows while AU15, AU23, and AU24 are positive in only 2–5%, yielding a 29\(\times\) overall imbalance; full per-class statistics are in Appendix 11 (Table ¿tbl:tab:dataset95stats?). Notably, joint three-task supervision is available for only 36.6% of training frames, making per-task masking essential.

Unless stated otherwise, all experiments share the following configuration: a frozen DINOv3 ViT-S/16 backbone, input resolution \(224\times224\), AdamW optimizer (lr=1e-4, weight decay=1e-2), 20 training epochs, batch size 64, and checkpoint selection by the validation composite metric \(P_{MTL} = P_{VA} + P_{EXPR} + P_{AU}\), where \(P_{VA}\) is the mean CCC of valence and arousal, \(P_{EXPR}\) is expression macro-F1, and \(P_{AU}\) is mean AU F1. For flow-based evaluation, predictions are produced with \(N=16\) samples and \(T=30\) Euler steps unless the ablation varies those parameters.

4.2 What Does Each Training Objective Contribute?↩︎

We isolate the contribution of each training objective by comparing three training regimes: deterministic supervision only (\(\beta=0\), no flow loss); flow supervision only (no deterministic task losses, \(\lambda_{det}=0\)); and both objectives jointly (\(\beta=1\)), which we call AffectFlow. Each checkpoint is decoded both deterministically and with flow sampling, yielding the six-condition study in Table 1.

The det-only sanity check (det-only checkpoint decoded with flow) collapses to \(P_{MTL}=0.402\), and the flow-only sanity check (flow-only checkpoint decoded deterministically) collapses to \(P_{MTL}=0.408\): each decode mode requires its corresponding training objective. Flow-only training with flow decoding reaches \(P_{MTL}=0.773\), confirming that the flow head does learn a meaningful conditional distribution even without deterministic task supervision. However, AffectFlow’s flow decoding (\(P_{MTL}=0.826\)) substantially outperforms flow-only training (\(P_{MTL}=0.773\)), revealing that deterministic supervision acts as an auxiliary training signal for the flow head: the task losses encourage the shared embedding to encode affect-discriminative features, which in turn improve the quality of the learned flow trajectories. Symmetrically, deterministic heads trained without the flow objective (det-only) perform comparably to those in AffectFlow under deterministic decoding (0.793 vs.), indicating that the flow loss does not harm the task heads but adds the generative pathway as an orthogonal capability. Flow decoding on trained AffectFlow yields large gains in CCC-V (\(+0.058\)) and CCC-A (\(+0.018\)) relative to its own deterministic decode, lifting \(P_{VA}\) from 0.200 to 0.238 at a modest cost to \(P_{AU}\). This pattern, flow sampling improving VA most, is consistent throughout all ablations: continuous targets benefit most from distributional averaging, while categorical and binary tasks benefit somewhat less.

Flow loss weight \(\beta\) and inference efficiency sweeps are in Appendix 8: \(\beta \in [0.5, 1.0]\) is a stable plateau (we adopt \(\beta=1.0\)); flow sampling saturates at \(N=8\) samples and \(T=10\) Euler steps (we use \(N=16\), \(T=30\)), consistent with the straight-line trajectory property of rectified flow [1]. Minor sweeps for VA loss weight and global AU threshold yield no improvement (Appendix 10.1, 10.2); the latter motivates per-AU calibration in Section 4.4.

4.3 Low-Learning-Rate DINOv3 Fine-Tuning↩︎

The frozen-backbone experiments isolate the contribution of AffectFlow, but they leave the visual representation unchanged. We therefore fine-tune the DINOv3 backbone end-to-end with a smaller learning rate (\(10^{-5}\)), batch size 32, and the same 20-epoch schedule. Table 2 compares deterministic and flow decoding for the fine-tuned checkpoint.

Fine-tuning provides the largest gain observed so far, raising \(P_{MTL}\) from the best frozen-backbone result of 0.831 to 1.045. Interestingly, deterministic decoding outperforms flow decoding after fine-tuning. This suggests that once the visual encoder is adapted to the target domain, the deterministic heads become substantially stronger, while the flow head may require retuning of its loss weight, sampling temperature, or training schedule.

We explored four expression-specific modifications to address the majority-class collapse: expression loss weight scaling, focal loss, per-task separate projection heads, and their combination. All four fail to improve \(P_{EXPR}\) over the AffectFlow baseline and consistently degrade \(P_{VA}\); full results are in Appendix 10.3. The common failure mode is that scaling or reweighting the cross-entropy loss, regardless of form, cannot overcome the gradient domination by Neutral and Other without structural sampling changes. This motivates the class-balanced sampling and positive-weighting strategies evaluated below.

We also explored patch soft-pool (PSP) aggregation, which learns per-channel attention weights over the 196 frozen ViT patch tokens and concatenates the result with the CLS token (Appendix 8.3). PSP deterministic reaches \(P_{MTL}=0.859\), the best frozen-backbone result without calibration, by capturing spatially-localised AU information, but is not combined with backbone fine-tuning in this study.

4.4 Per-AU Threshold Calibration↩︎

Appendix 10.2 showed that a single global AU threshold cannot improve over \(\tau=0.5\). We therefore tune an independent threshold per AU on the validation set by grid search, leaving VA and expression predictions unchanged. Table 3 reports the gain from this post-hoc step on three representative checkpoints.

3pt

Per-AU calibration is theoretically principled rather than an ad-hoc leaderboard trick: the AU head predicts sigmoid activation probabilities \(p(AU_k{=}1 \mid x)\) conditioned on the image, so the optimal decision boundary need not equal 0.5, particularly when class-imbalanced BCE training systematically suppresses predicted probabilities for rare AUs. This suppression is confirmed by the per-class analysis in Section 4.5: AU15 and AU23 have near-zero F1 at threshold 0.5 despite non-trivial true positive rates, meaning the model has learned genuine discriminative signal that the default threshold cannot recover. The rectified-flow head further reinforces this framing: by modeling \(p(y \mid x)\) as a full conditional distribution, AffectFlow provides per-sample uncertainty information alongside point predictions; per-AU calibration exploits this probabilistic output to find the threshold at which the flow-informed sigmoid posterior best separates positives from negatives. Per-AU calibration yields a consistent \(P_{AU}\) gain of approximately +0.06 across all checkpoints, with no retraining cost. Rare AUs such as AU15 and AU23 benefit most from lower thresholds, while frequently active AUs (AU25, AU7) prefer thresholds near 0.55–0.60. The best overall validation score is \(P_{MTL}=1.101\), obtained by combining low-LR DINOv3 fine-tuning with deterministic decoding and per-AU calibration. An analogous post-hoc calibration strategy applied to expression class predictions (Section 4.7) yields a further \(+0.054\) \(P_{MTL}\) gain on top of per-AU calibration.

4.5 Per-Class Analysis and Class-Weighted Expression Loss↩︎

The macro-F1 metric used for checkpoint selection conceals large per-class disparities. Full per-class expression F1 for the AffectFlow and fine-tuned models is in Appendix 14 (Table 22), alongside the per-AU breakdown (Table 23); Happiness and Other achieve \(F1 > 0.49\), but Fear (\(F1 = 6.3\)–\(6.9\%\)) and Sadness (\(F1 = 4.3\)–\(7.1\%\)) are catastrophically low even after fine-tuning. The model collapses 77% of Sadness, 64% of Surprise, and 41% of Fear into “Other”, which absorbs the bulk of gradient mass under standard cross-entropy. AU prediction mirrors the same pattern: AU15 and AU23 have near-zero F1 at \(\tau=0.5\) despite non-trivial true positive rates (see Appendix 14).

4.5.0.1 Class-weighted cross-entropy.

We replace standard cross-entropy with an inverse-frequency weighted variant assigning each class \(c\) weight \(w_c = N_\text{valid}/(8\,n_c)\) (Appendix 11, Table ¿tbl:tab:dataset95stats?). After seven training epochs (deterministic validation, frozen backbone), the best checkpoint reaches \(P_{EXPR} = 0.239\) and \(P_{MTL} = 0.834\), compared to \(P_{EXPR} = 0.218\) and \(P_{MTL} = 0.802\) for the AffectFlow baseline at the same evaluation protocol. The improvement is concentrated in the minority classes: Disgust F1 rises from \(11.8\%\) to \(39.0\%\) and Surprise from \(2.8\%\) to \(12.8\%\), while Fear (\(0.3\%\)) and Sadness (\(8.5\%\)) remain the primary bottleneck, suggesting that their imbalance severity requires additional intervention beyond loss reweighting.

Imbalance remedies are detailed in Appendix 12. In brief: a class-balanced WeightedRandomSampler raises \(P_{EXPR}\) to \(0.251\) (\(+0.039\) over AffectFlow flow), the best frozen-backbone expression result; per-AU BCE positive weighting raises \(P_{AU}\) to \(0.427\)–\(0.439\), the best frozen-backbone AU result without calibration; and label smoothing (Appendix 10.4) and global AU loss weight (Appendix 10.5) provide no meaningful gains. Critically, AU positive-weighted training is incompatible with flow decoding (\(P_{MTL}\) degrades to \(0.745\)–\(0.777\)) because the reweighted loss shifts predicted probability distributions incompatibly with the rectified-flow sampler.

4.6 Flow Retuning after Backbone Fine-Tuning↩︎

Section 4.3 showed that low-LR fine-tuning of the DINOv3 backbone improves all tasks substantially, but leaves the flow head undertuned: deterministic decoding outperforms flow decoding in the fine-tuned setting. We address this by reactivating the flow objective (\(\beta=0.5\)) during fine-tuning, retuning the flow head jointly with the backbone. Table 4 compares fine-tuning with a frozen flow head against fine-tuning with flow retuning under both decode modes.

3pt

Flow retuning at \(\beta{=}0.5\) under deterministic decoding reaches \(P_{MTL}=1.062\), surpassing plain fine-tuning without calibration (\(1.045\)), while flow decoding (\(P_{MTL}=0.956\)) substantially outperforms the AffectFlow frozen-backbone flow baseline (\(0.826\)), confirming that the flow head recovers distributional capability after backbone adaptation.

Increasing the flow weight to \(\beta{=}1.0\) further improves deterministic decoding to \(P_{MTL}=1.073\), driven by a large CCC-V gain (\(0.387\) vs.\(0.308\) for \(\beta{=}0.5\)) at a modest cost to CCC-A and \(P_{AU}\). After per-AU calibration, this configuration reaches \(P_{MTL}=1.123\), surpassing the calibrated plain fine-tuning baseline of \(1.101\); post-hoc expression calibration (Section 4.7) further raises this to \(P_{MTL}=1.177\). Flow decoding for \(\beta{=}1.0\) (\(P_{MTL}=0.931\)) trails the \(\beta{=}0.5\) flow variant (\(0.956\)), suggesting that the stronger flow objective reshapes the learned target distribution in a way that marginally reduces flow-sampled consistency while benefiting the deterministic readout. In both variants, deterministic decoding outperforms flow decoding after retuning, indicating that further alignment of the flow head with the adapted backbone remains an open direction.

Kitchen-sink fine-tuning (Appendix 13) applies class-weighted cross-entropy and per-AU positive weighting jointly during backbone fine-tuning. The combination does not improve \(P_{MTL}\) over plain fine-tuning: \(P_{MTL}=1.041\) vs.\(1.045\), with the AU gain (\(+0.042\)) offset by a \(P_{VA}\) drop (\(-0.047\)). This confirms that frozen-backbone reweighting strategies do not stack with backbone adaptation: once the backbone is fine-tuned, flow retuning is the more effective path. Replacing ViT-S with ViT-B improves \(P_{VA}\) (\(0.354\) vs.\(0.325\)) but degrades \(P_{EXPR}\) (\(0.255\) vs.\(0.296\)), yielding \(P_{MTL}=1.116\) calibrated; horizontal-flip test-time augmentation reduces all scores to \(P_{MTL}=1.111\) calibrated, full results in Appendix 9.

4.7 Per-Class Expression Threshold Calibration↩︎

Section 4.4 showed that per-AU threshold calibration yields a consistent \(+0.06\) \(P_{AU}\) gain by finding per-AU decision boundaries that maximise F1 on the validation set. We apply the same post-hoc principle to the expression head: per-class logit weights are tuned on the validation set to maximise macro-F1, directly addressing the systematic under-prediction of minority classes (Fear, Sadness) caused by the \(8\times\) expression imbalance. The full per-class breakdown is in Appendix 14 (Table 24, Figure 3). Fear benefits most (\(0.038 \to 0.331\), \(+0.293\)), followed by Sadness (\(0.171 \to 0.282\), \(+0.111\)) and Disgust (\(0.465 \to 0.507\), \(+0.042\)); majority classes (Happiness, Other) are essentially unchanged. The large Fear gain confirms that the model has learned genuine discriminative signal for the rarest class: the default argmax simply never predicts it because its logit magnitude is suppressed relative to Neutral and Other by the imbalanced training distribution. Calibration recovers this latent signal without any retraining cost, in direct analogy to how per-AU threshold calibration recovers rare-AU signal (AU15, AU23) in Section 4.4.

Combining per-class expression calibration with per-AU calibration on the fine-tuned+flow retune (\(\beta{=}1.0\)) checkpoint yields the best overall result: \(P_{VA}=0.325\) \(+\) \(P_{EXPR}=0.350\) \(+\) \(P_{AU}=0.502\) \(=\) \(\mathbf{P_{MTL}=1.177}\), a \(+0.054\) improvement over the flow-retuned, per-AU-calibrated baseline (\(1.123\)).

4.8 Summary of Ablation Results↩︎

Table 5 collects the best representative result per configuration (all models use frozen DINOv3 unless noted; full 18-configuration table in Appendix 6). Both training objectives are needed: flow-only (\(P_{MTL}=0.773\)) and det-only-with-flow-decode (\(0.402\)) both underperform jointly-trained AffectFlow (\(0.826\)), and patch soft-pool further lifts the frozen-backbone deterministic result to \(0.859\). The best frozen-backbone result without calibration is \(0.890\) (AU positive weighting combined with class-weighted cross-entropy; Appendix 12), while per-AU calibration alone raises AffectFlow flow to \(0.888\). Backbone fine-tuning is the dominant lever, lifting the best result to \(1.045\) (\(1.101\) calibrated); flow retuning (\(\beta{=}1.0\)) on top of it reaches \(1.073\) (\(1.123\) calibrated), and post-hoc expression calibration adds a further \(+0.054\) for the best overall result \(\mathbf{P_{MTL}=1.177}\). Kitchen-sink fine-tuning (Appendix 13) and all expression/AU reweighting ablations (Appendices 10, 12) do not compound with backbone fine-tuning.

4pt

4.9 Discussion and Open Challenges↩︎

All results in this paper are on the local s-Aff-Wild2 validation split; the SOTA scores cited in Section 2 are from the 7th ABAW hidden test set and are not directly comparable. The current best validated result (\(P_{MTL}=1.177\)) substantially improves over the official baseline (\(P_{MTL}=0.45\)); Figure 2 in Appendix 10 visualises the full progression across key configurations. Within our controlled ablation study the gap is most pronounced in \(P_{EXPR}\) (best uncalibrated: \(0.296\); best calibrated: \(0.350\)); two principal factors account for it. First, expression prediction remains the most challenging task: the per-class breakdown (Table 22) reveals that Fear and Sadness F1 remain below 7% at the default threshold, with predictions collapsed into “Other”. The \(8\times\) expression imbalance is the proximate cause; class-weighted cross-entropy and balanced sampling partially close the gap (\(P_{EXPR}=0.239\)–\(0.251\)), while post-hoc expression calibration recovers the latent rare-class signal directly, raising Fear F1 from \(0.038\) to \(0.331\) and Sadness from \(0.171\) to \(0.282\) without retraining. Second, the flow head requires retuning after backbone fine-tuning to recover distributional prediction capability: reactivating the flow objective during fine-tuning is beneficial, with \(\beta{=}1.0\) yielding the strongest deterministic result (\(P_{MTL}=1.073\); \(1.123\) with per-AU calibration; \(1.177\) with combined calibration), though deterministic decoding continues to outperform flow decoding in all fine-tuned variants. This suggests the flow objective acts as a structure-aware regularizer for \(z\), enforcing joint affect structure that benefits the deterministic heads even when flow decoding is not used at inference.

5 Conclusion↩︎

We introduced AffectFlow-DINO, a multi-task facial affect model that treats in-the-wild facial behavior as a conditional distribution \(p(y \mid x)\) rather than a deterministic mapping, learned via a rectified-flow head over the joint 22-dimensional affect space and validated through 26 controlled ablations on s-Aff-Wild2.

Three lessons emerge. The flow and deterministic objectives are complementary: flow-only training collapses without the deterministic anchor, and deterministic-only inference discards distributional capability. Flow sampling benefits VA estimation most, the task most sensitive to continuous ambiguity, and saturates rapidly, consistent with rectified flow’s straight-line transport. Backbone adaptation is the dominant lever, but the flow head must be retrained jointly during fine-tuning, or the generative pathway degrades.

A broader implication is that severe class imbalance suppresses learned signals without erasing them: post-hoc calibration unlocks near-zero F1 for rare categories (Fear, Sadness, AU15, AU23) at zero cost, suggesting decision boundaries can surface rare-class signal already encoded in the representation.

Closing the gap to SOTA likely requires temporal aggregation, more expressive fine-tuned decoding, and joint backbone-flow multi-task scaling from the start.

6 Full Results Summary↩︎

Table 6 is the unabridged version of the main paper’s results summary (Section 4.8), including intermediate and negative-result configurations omitted there for space.

4pt

7 Related Datasets↩︎

Table 7 provides an overview of datasets relevant to ABAW-style joint affect modeling. s-Aff-Wild2 is the only benchmark that combines all three task targets (VA, EXPR, AU) as static image crops, making it the natural fit for this challenge.

8 Flow Weight, Inference Efficiency, and Patch Soft-Pool↩︎

8.1 Flow Loss Weight↩︎

Table 8 sweeps the flow loss weight \(\beta \in \{0.25, 0.5, 1.0, 2.0\}\), with all other hyperparameters fixed and flow decoding at inference (\(N=16\), \(T=30\)).

4pt

Performance is stable for \(\beta \in [0.5, 1.0]\); at \(\beta=2.0\) the flow objective dominates and \(P_{MTL}\) drops. We adopt \(\beta=1.0\) as the default.

8.2 Inference Efficiency↩︎

Table ¿tbl:tab:e595e6? sweeps sample count \(N\) (left) and integration steps \(T\) (right) on the AffectFlow checkpoint.

4pt

Sampling saturates at \(N=8\) and all step counts are within 0.007 \(P_{MTL}\) of each other. \(N=1\) already outperforms the official baseline (\(P_{MTL}=0.45\)), consistent with the straight-line trajectory property of rectified flow; we use \(N=16\), \(T=30\) as the default.

8.3 PSP: Patch Soft-Pool Feature Aggregation↩︎

Rather than using only the DINOv3 [CLS] token, patch soft-pool (PSP) learns per-channel attention weights over the 196 spatial patch tokens and concatenates the result with the CLS token before projection: \[h_i^{\text{PSP}} = \bigl[h_i^{\text{CLS}},\; \textstyle\sum_{p=1}^{196} \sigma_p^{(j)} \cdot h_i^{(p,j)}\bigr]_{j=1}^{d}, \label{eq:psp}\tag{14}\] where \(\sigma^{(j)} = \mathrm{softmax}(w^{(j)})\), weights initialised to uniform. This doubles the projection input to \(2d=768\); only the pooling weights are trained (frozen backbone).

3pt

PSP Det reaches \(P_{MTL}=0.859\), the best frozen-backbone result without calibration (\(+0.033\) over AffectFlow flow), with consistent gains in all three tasks. PSP Flow underperforms AffectFlow Flow (\(0.801\) vs.\(0.826\)), likely because the doubled input dimension requires longer retuning of the flow head; combining PSP with backbone fine-tuning is left as future work.

9 ViT-B Backbone and Test-Time Augmentation↩︎

Table 10 compares the ViT-B backbone and horizontal-flip test-time augmentation (TTA) against the flow-retuned (\(\beta{=}1.0\)) best checkpoint.

3pt

ViT-B (ViT-B/16, 86M parameters) improves valence (CCC-V \(0.416\) vs.\(0.387\)) and per-AU calibrated score (\(0.507\) vs.\(0.502\)), but degrades \(P_{EXPR}\) by \(0.041\), yielding \(P_{MTL}=1.116\) calibrated, below the ViT-S best (\(1.123\)). The VA–EXPR trade-off suggests ViT-B captures finer spatial VA signal but is harder to fine-tune for balanced expression prediction in 20 epochs. Horizontal-flip TTA hurts all three sub-scores; flip changes asymmetric AU patterns (e.g.AU12 corner-of-mouth), explaining the regression.

10 Additional Ablation Studies↩︎

This appendix collects ablations that did not improve over the AffectFlow baseline: VA loss weight, global AU threshold sweep, expression-enhancement strategies, label smoothing, and global AU loss weight.

10.1 VA Loss Weighting↩︎

Arousal estimation is the weakest component across all runs. Table 11 tests whether increasing the VA loss weight \(\lambda_{VA} \in \{1, 2, 4\}\) can improve the arousal CCC without sacrificing other tasks.

Doubling the VA weight (\(\lambda_{VA}=2\)) yields the best \(P_{MTL}\) by improving \(P_{EXPR}\) without harming \(P_{VA}\), suggesting minor positive cross-task regularization. At \(\lambda_{VA}=4\) the stronger VA supervision improves arousal but reduces expression performance and the overall score, consistent with negative transfer at extreme task weight imbalances [31]. The gain at \(\lambda_{VA}=2\) is marginal (\(+0.004\) \(P_{MTL}\)) and not consistent across ablations; we retain the default \(\lambda_{VA}=1\).

10.2 Global AU Threshold Sweep↩︎

The default AU decision threshold is 0.5. Because AU labels are multi-label and class-imbalanced, we additionally evaluate a simple global threshold sweep on the AffectFlow checkpoint. Table 12 shows that a single global threshold does not improve over the default value; lower thresholds increase false positives, while a high threshold of 0.7 sharply reduces AU F1.

This result confirms that a single global threshold is too coarse for the heterogeneous AU imbalance structure; per-AU calibration (Section 4.4 of the main paper) is necessary.

10.3 Expression Enhancement Failures↩︎

Four expression-specific modifications were explored in sequence. All fail to improve \(P_{EXPR}\) over the AffectFlow baseline and consistently degrade \(P_{VA}\).

10.3.0.1 Expression Loss Weight Boost.

A lightweight proxy for class-balanced training is to increase \(\lambda_{EXPR}\), the scalar weight on the expression cross-entropy term, without changing the sampler or loss form. Table 13 sweeps \(\lambda_{EXPR} \in \{1, 2, 4\}\) on the main AffectFlow checkpoint with flow decoding.

Increasing \(\lambda_{EXPR}\) does not improve expression macro-F1 and consistently degrades \(P_{VA}\). Because the cross-entropy loss is dominated by majority classes (Neutral, Happiness) regardless of the weight, scaling the loss up merely competes for capacity with the VA task without directing gradient towards minority classes.

10.3.0.2 Focal Loss for Expression.

We replace the masked cross-entropy for expression with a focal loss [39] that down-weights easy majority-class samples via the modulating factor \((1-p_t)^\gamma\). Table 14 reports flow-decoding results for \(\gamma \in \{1, 2\}\) with all other settings identical to AffectFlow.

Focal loss does not improve \(P_{EXPR}\) and reduces \(P_{VA}\) by approximately 0.016. We attribute this to the relatively small number of valid expression labels (90K training frames) combined with heavy class skew: the focal modulator suppresses gradient from majority classes without providing enough additional signal on rare classes to compensate for the lost VA supervision.

10.3.0.3 Per-Task Separate Projection Heads.

To reduce negative transfer through the shared \(768\to768\) projection, we give each task head its own projection MLP while keeping a shared projection for the flow network. Table 15 compares this design against the shared-projection AffectFlow model.

Separate projections slightly reduce all three sub-scores, suggesting that the shared representation acts as a useful regularizer rather than a harmful bottleneck in the frozen-backbone regime.

10.3.0.4 Focal Loss with Separate Projection Heads.

Combining focal loss with separate projection heads yields the configuration in Table 16: focal loss (\(\gamma=2\)) plus per-task projections.

The combined variant improves \(P_{EXPR}\) marginally over separate projection heads alone (\(+0.003\)) but remains well below the AffectFlow baseline. The consistent failure of these expression-enhancement strategies establishes that expression imbalance requires structural remedies (class-balanced sampling, positive-weighted losses) rather than loss reweighting or architectural changes alone.

10.4 Label Smoothing↩︎

Label smoothing distributes a fraction \(\epsilon\) of the one-hot probability mass uniformly across expression classes, providing mild regularization against over-confident predictions. Table 17 sweeps \(\epsilon \in \{0.05, 0.10, 0.20\}\).

4pt

Mild smoothing (\(\epsilon=0.05\)) raises \(P_{EXPR}\) to \(0.230\) and CCC-V to \(0.276\), but the gain is small and diminishes rapidly: at \(\epsilon=0.20\), \(P_{EXPR}\) falls below the unsmoothed baseline. Label smoothing alone is insufficient to close the expression imbalance gap.

10.5 Global AU Loss Weight↩︎

As an alternative to per-AU positive weighting, Table 18 sweeps the global AU loss weight \(\lambda_{AU} \in \{0.5, 1.0, 2.0\}\).

4pt

Neither reducing nor increasing \(\lambda_{AU}\) changes \(P_{MTL}\) appreciably; both variants plateau at \(0.819\). Per-AU positive weighting (Section 12.2 below) achieves a larger \(P_{AU}\) gain (\(+0.043\)) than any global weight setting, confirming that per-class reweighting is more effective than global loss scaling for AU detection.

11 Class Distribution Statistics↩︎

Table ¿tbl:tab:dataset95stats? provides the full per-class expression counts and AU positive rates referenced throughout the paper.

3pt

12 Imbalance Remedies: Balanced Sampling and AU Positive Weighting↩︎

12.1 Balanced Expression Sampling↩︎

WeightedRandomSampler assigns each training frame a sampling probability inversely proportional to its expression class count, rebalancing the effective class distribution without modifying the loss function. Table 19 compares the balanced sampler alone and the balanced sampler combined with class-weighted cross-entropy against the AffectFlow flow baseline.

4pt

The balanced sampler raises \(P_{EXPR}\) to \(0.251\) (\(+0.039\) over AffectFlow flow) and lifts \(P_{MTL}\) to \(0.864\), the best frozen-backbone result for expression. Combining balanced sampling with class-weighted cross-entropy gives \(P_{EXPR}=0.246\) under deterministic decode but \(P_{VA}\) drops substantially (\(0.193\) vs.\(0.238\)), indicating that the combined strategy over-suppresses gradient on majority classes at the cost of valence-arousal prediction.

12.2 BCE Positive Weighting for AUs↩︎

We set each AU’s BCE positive weight to \(n_\text{neg}/n_\text{pos}\) computed from the training set (Table ¿tbl:tab:dataset95stats?). One variant applies positive weighting to the AU head alone; another further adds class-weighted expression cross-entropy. Table 20 reports deterministic predictions throughout: flow decoding degrades both runs (\(P_{MTL}=0.777\) for AU positive weighting alone and \(0.745\) for the combined variant) because the reweighted loss shifts predicted probability distributions incompatibly with the rectified-flow sampler.

4pt

AU positive weighting raises \(P_{AU}\) from \(0.384\) to \(0.427\) (\(+0.043\)) at training time. Adding expression class weighting further lifts \(P_{AU}\) to \(0.439\), yielding \(P_{MTL}=0.890\), the best frozen-backbone result without calibration, marginally surpassing AffectFlow after per-AU calibration (\(0.888\)).

13 Kitchen-Sink Fine-Tuning↩︎

We test whether the most effective frozen-backbone interventions, class-weighted cross-entropy and per-AU BCE positive weighting, transfer additively to the fine-tuned setting. Both are applied jointly during the same low-LR fine-tuning run as the plain fine-tuning experiment (Section 4.3 of the main paper). Table 21 reports deterministic predictions with and without per-AU calibration.

3pt

The kitchen-sink combination does not improve \(P_{MTL}\) over plain fine-tuning without calibration (\(1.041\) vs.\(1.045\)), despite raising \(P_{AU}\) from \(0.441\) to \(0.483\). The AU gain is offset by a \(P_{VA}\) drop from \(0.318\) to \(0.271\): class-weighted cross-entropy reduces gradient allocation for the VA task when the backbone is jointly optimized, consistent with the pattern seen when combining balanced sampling with class-weighted CE (Section 12.1). Expression macro-F1 (\(0.288\)) is nearly identical to plain fine-tuning (\(0.285\)), confirming that AU positive weighting absorbs gradient competition without a meaningful expression benefit. After per-AU calibration, the kitchen-sink variant reaches \(P_{MTL}=1.061\), below the best overall flow-retuned (\(\beta{=}1.0\)) calibrated result (\(1.123\)). These results indicate that once the backbone is fine-tuned, frozen-backbone class-reweighting strategies add little value and can harm \(P_{VA}\); flow retuning is a more effective path.

14 Per-Class and Per-AU F1 Breakdowns↩︎

Table 22 reports the full per-class expression F1 for the AffectFlow and fine-tuned models referenced in Section 4.5 of the main paper, and Table 23 reports the per-AU F1 scores before and after per-AU threshold calibration for the fine-tuned checkpoint. Table 24 and Figure 3 report the per-class expression calibration results from Section 4.7 of the main paper.

4pt

5pt

AU15 and AU23, the two rarest AUs (positive rate 2.4% and 2.9%), have near-zero F1 at the default threshold but recover to \(10.8\%\) and \(18.9\%\) after per-AU calibration, confirming that the model has learned discriminative signal that the imbalanced training objective systematically suppresses.

References↩︎