1 Introduction↩︎

Video-based facial expression quality assessment (FEQA) has emerged as a promising approach for quantifying subtle facial motor impairments associated with neurological disorders, such as Parkinson’s disease (PD) [1] and amyotrophic lateral sclerosis (ALS) [2], as well as other clinically relevant affective states, e.g. pain [3]. However, despite recent progress, most existing FEQA approaches [1], [4]–[11] ultimately return only a single severity score. While such outputs support quantitative monitoring, a score alone can only be interpreted through predefined clinical descriptors, such as MDS-UPDRS [12], which are coarse and generic and do not preserve the specific evidence needed for later audit or review. As shown in Fig. 1 for ‘Squeeze eyes’, two clips may receive the same rating, level 2, despite different facial regions and movement patterns.

Recording the observations behind a score, even in a short text form, can therefore make FEQA outputs more transparent and auditable. It links ratings to visible motor evidence, allows later review or rescoring, supports more informative longitudinal comparison, and helps analyse disagreements between raters or between human and model assessments [13].

Traditional interpretability methods, such as saliency maps and other post-hoc spatial attribution techniques [14], [15], remain limited in this setting. Although they can indicate which facial region influenced a prediction, they do not describe the spatio-temporal motor patterns that matter clinically, such as reduced range of motion, persistent blinking, or subtle asymmetry. Conversely, off-the-shelf Large Language Models (LLMs) and Video-Language Models (VLMs) can generate fluent explanations, but may hallucinate clinically irrelevant or visually unsupported details when asked to justify a prediction [16].

In this paper, we address this limitation by proposing TraMP-LLaMA, a unified multi-modal framework that jointly predicts a severity score and generates a structured textual report. We deploy VideoLLaMA3 [17] as the base VLM to provide general vision-language capability, and extend it with a landmark trajectories stream to encode facial motion features. For severity prediction, the scoring head operates solely on the joint trajectory-RGB representation obtained by fusing trajectory and RGB features. For report generation, in contrast, the language model is instruction-tuned on the concatenation of evidence tokens derived from RGB appearance, trajectory dynamics, and their fused representation. A key technical challenge in this joint formulation is avoiding task interference. Jointly optimising language generation and severity regression can distort spatio-temporal feature learning and degrade assessment performance. We therefore introduce a decoupled instruction tuning strategy: both tasks condition on the same shared spatio-temporal evidence, but only the scoring objective is allowed to update the motion encoder and the fusion module, while gradients from the language objective are blocked. This preserves reliable scoring while still enabling clinically grounded report generation.

The recently released PFED5 [1] is a unique facial expression assessment dataset that provides MDS-UPDRS scores, but lacks textual annotations of the evidence that underlines the score in each case. We construct PFED5+ by augmenting PFED5 with expert-guided, clinically reviewed, clip-level motion descriptions. These annotations describe visible, region-focused motor patterns without directly encoding the severity label, enabling report supervision while reducing label leakage.

Our main contributions can be summarised as: (i) We propose TraMP-LLaMA, a multimodal framework that jointly predicts severity scores and generates structured evidence reports, producing inspectable facial motion evidence alongside quantitative assessments; (ii) We develop a decoupled instruction-tuning strategy that mitigates task interference between severity regression and report generation by blocking language-generation gradients from updating the motion encoder and fusion module, while allowing both tasks to condition on the same spatio-temporal evidence; (iii) Experiments on PFED5+, our annotation-augmented extension of PFED5, demonstrate that TraMP-LLaMA achieves state-of-the-art severity prediction under joint multi-expression training, outperforming prior methods by at least 4.39%, while surpassing VLM baselines in report generation by various metrics.

2 Related works↩︎

We briefly review recent literature relevant to our work, focusing on facial expression analysis from videos in healthcare, as well as VLMs for generative interpretability.

Video-based Facial Expression Analysis in Healthcare – Numerous studies have explored video-based facial analysis for early disease detection, aiming to distinguish patients from healthy controls in disorders such as PD [18]–[21] and ALS [22]. For example, Razzouki et al. [21] identified hypomimia in early-stage PD by extracting optical flow and RGB facial features with a video transformer and classifying patients using a Random Forest model. While such methods are useful for screening, they remain limited to coarse patient-control discrimination. Others have developed classification-based approaches that predict multiple intensity levels to evaluate clinically relevant affective states, such as pain [23], [24] and depression [25]–[27]. For example, Thiam et al. [23] proposed a two-stream CNN-BiLSTM combining optical flow and motion history to capture spatio-temporal pain-related features for classifying pain levels. While more fine-grained than binary detection, categorical models still discretise impairment into bins, overlooking subtle gradations [1].

Continuous severity assessment, often referred to as FEQA, has therefore emerged as a clinically relevant direction, where models regress facial dynamics to scores on validated clinical scales [1], [4]–[11], [28]–[32]. For example, Duan et al. [11] combined facial trajectories with RGB features to improve regression of subtle motion variations across severity levels. However, despite differences in task formulation, existing methods across detection, multi-classification, and regression generally return only an outcome label, category, or severity score, without explicitly describing the observable spatio-temporal motor evidence supporting that output. This limits their usefulness for clinically interpretable assessment.

Vision-Language Models for Video and Facial Behavior Description – Recent VLMs have made it increasingly feasible to translate videos into natural language, supporting tasks such as captioning, question answering, and open-ended reasoning [17], [33]–[36]. For example, LongVU [36] improves long-video understanding through spatio-temporal adaptive compression, reducing token redundancy while preserving informative visual content under limited context length. Likewise, VideoChat2 [35] improves video reasoning through staged alignment and instruction tuning on diverse video-language tasks. While such models demonstrate strong general video-to-text capability, they are primarily trained on web-scale corpora dominated by coarse events and scene-level semantics, and therefore are not designed to describe fine-grained, low-amplitude, and temporally nuanced evidence in a faithful manner [16], [37]. This limitation is particularly problematic in our setting, where the relevant evidence lies in subtle facial motor patterns rather than broad scene content.

Others have moved facial expression analysis beyond categorical recognition towards language-based description [38]–[42]. For example, EmoVIT [41] adapts an instruction-following VLM to affective reasoning and generates textual justifications for emotional states using Action Unit (AU)-to-language supervision. Similarly, Face-LLaVA [42] introduces facial priors and region-guided attention to improve anatomically grounded facial descriptions. While these approaches improve interpretability compared with label-only prediction, most are developed for affective analysis or general face understanding rather than clinical motor assessment. Their outputs are often image-centric, AU-centric, or optimised for emotion understanding, and may therefore under-specify the spatio-temporal motor signs required in neurological evaluation, such as reduced movement amplitude, delayed initiation, asymmetry, or abnormal coordination across facial regions. Moreover, although datasets such as MAFW provide textual descriptions, many existing methods [43]–[46] use text primarily as auxiliary supervision or an additional modality to improve recognition performance rather than as a target output for evidence-grounded reporting.

This direction is conceptually related to radiology report generation [47]–[50], where models translate visual findings into clinically usable text rather than returning only a prediction. However, unlike radiology, the relevant evidence in our setting is not static anatomical structure but subtle facial motion unfolding over time. Existing video-language and facial description models therefore do not directly address the problem studied here, which is to generate evidence-grounded reports that remain tightly aligned with the facial motor patterns supporting a clinical severity rating.

3 Language Annotation Pipeline↩︎

To enable report generation grounded in observable motion evidence, we develop a systematic annotation framework that produces structured, clinically guided descriptions for each PFED5 video clip (see Fig. 2). We call this extension PFED5+. These new textual annotations introduced here explicitly describe observable motor evidence (approved by two expert PD clinicians) and are used as supervision for report generation.

Temporal Structure and Standardised Template – Facial-expression actions unfold over time and typically exhibit distinct phases, with phase boundaries following a consistent rule at the clip level. We therefore organise each annotation into three temporal components: (i) Pre-action corresponds to the neutral baseline before visible motion onset; (ii) In-action covers the main execution phase of the target movement; (iii) Post-action describes the recovery stage after the movement peak, covering the offset and stabilisation of the face as it relaxes back toward a neutral state. Importantly, we do not annotate exact frame indices or timestamps; instead, phases are described at the clip level using this standard rule to ensure consistent coverage of the full temporal evolution. This three-phase template is enforced for all action clips, where each phase is described using the same set of region-specific slots. A complete template example is illustrated in Fig. 2. When information in a phase is not observable due to incomplete face visibility, such as partial face crops or missing regions after preprocessing, we use a fixed placeholder (e.g., [state unobservable]) to explicitly mark the missing phase. This avoids implicit omissions and prevents the model from treating absent mentions as negative evidence. For PFED5+, four task-elicited actions (‘Smile’, ‘Frown’, ‘Squeeze eyes’, and ‘Clench teeth’) follow the pre/in/post template. The ‘Sit at rest’ task does not contain a distinct onset, peak, and offset. We therefore use a single-phase description that corresponds to the in-action component, and summarises resting facial state and spontaneous movements over the clip.

Clinically Relevant Semantic Content – The semantic richness of our annotations is derived from fine-grained observations of clinically-relevant facial regions. Under expert guidance, we define which regions should be inspected in each phase. Annotators then review each clip and record the movement state of each region in each phase. Observations include movement versus no movement, direction of change when visible, and left–right asymmetry when applicable. These structured region-level observations are then assembled into a unified textual description following the standard template.

To maintain clinical faithfulness and reduce hallucination risks during later model training, we adopt a strictly observational annotation policy. Annotators avoid inferential or affective terms and report only what is visually observable. Regions showing no change are explicitly annotated as no movement rather than being omitted, which prevents missing mentions from being interpreted as unknown or implicitly active. This design ensures that the textual labels function as factual evidence statements rather than severity explanations.

Annotation Process and Coverage – The textual annotations cover the full PFED5 dataset (2,811 clips). The workflow consists of three stages: (i) Clinicians guide the schema design by defining the regions and observation checklist used for each phase; (ii) Annotators perform manual region-state annotation by recording the motion state of each region per phase for every clip; and (iii) template-based text assembly is performed automatically, where code converts the structured annotations into consistent textual descriptions.

Consistency and Quality Assurance – To improve fluency and reduce redundancy without changing semantics, we apply GPT-4o to post-process the assembled text. The model is provided only with draft textual descriptions. No clinical videos, frames, or identifiable visual data are uploaded for this refinement step and the refinement prompt enforces strict constraints. It forbids adding new facts, forbids inference beyond the provided observations, and requires preserving the meaning of all region-level evidence statements.

All refined descriptions are then fully double-checked and validated by clinical experts to ensure factual correctness and template compliance. Additional details, including the complete template, slot schema, refinement prompt, and annotation statistics, are provided in Supplementary Materials.

4 TraMP-LLaMA↩︎

4.1 Problem Formulation↩︎

This work proposes a joint scoring and reporting solution for video-based FEQA. Let’s consider a facial video clip \(X_{v} \in \mathbb{R}^{T_v \times H \times W \times 3}\) and its corresponding landmark trajectories \(X_{m} \in \mathbb{R}^{T_m \times P \times 5}\), where \(T_v\) and \(T_m\) indicate the frame length and trajectory length, respectively, \(P=63\) is the number of landmarks and the 5 channels encode \((x,y)\) coordinates along with the RGB value sampled at each landmark location. Then, the model \(\mathcal{F}\), where \[\hat{s},~ \hat{r} = \mathcal{F}(X_{v}, X_{m}, I)~,\] outputs (i) a severity score \(\hat{s}\) aligned with clinical ratings (e.g., MDS-UPDRS [12]), and (ii) a template-constrained structured report \(\hat{r}\) that describes observable facial motor evidence under the template defined in Section 3. \(I\) denotes the instruction prompt that specifies the reporting format and task context. We temporally align the video and trajectory streams such that \(T_v\) and \(T_m\) are consistent. Supervision is provided by the severity label \(s\) and the motion-description label \(r\). The report is intended to externalise evidence that supports the predicted score, rather than to restate the score or provide diagnostic conclusions.

4.2 Network Architecture↩︎

Fig. 3 illustrates the proposed TraMP-LLaMA framework. The architecture comprises a trainable trajectory encoder for landmark motion, a frozen vision encoder for RGB frames, a trainable cross-fusion module that integrates motion and appearance into a fused representation, and two decoupled output branches for severity scoring and report generation.

Visual encoding – We adopt VideoLLaMA3 [17], consisting of SigLIP-NaViT [52] as vision encoder and Qwen-2.5 [53] as LLM, as the base VLM due to its strong video understanding capability and its native compatibility with LLM-conditioned token interfaces, which aligns with our design of projecting multimodal evidence into the LLM embedding space. RGB frames \(X_v\) are encoded by SigLIP-NaViT, producing a sequence of visual tokens \({t_v}\) that capture appearance and coarse spatio-temporal context. In our implementation, the vision encoder is frozen during training, as indicated in Fig. 3. The token sequence \({t_v}\) is mapped to the LLM embedding space by the visual projector, yielding the visual token embeddings \(E_v \in \mathbb{R}^{T_{v'}\times d}\), where \(T_{v'} < T_{v}\) due to token compression used in VideoLLaMA3 [17] and \(d\) denotes the LLM embedding dimension. For cross-fusion, we apply global average pooling over \(t_v\) to obtain a clip-level visual embedding \(\bar{t}_v\), which serves as the visual input to the cross-fusion module.

Trajectory encoding – To provide explicit motion evidence, we encode landmark trajectories \(X_m\) using a trainable SkateFormer [51], following the trajectory representation introduced in [11]. SkateFormer produces a sequence of motion embeddings which are aggregated by global average pooling to produce a clip-level motion feature \(\bar{t}_m\). The motion projector then maps \(\bar{t}_m\) to the LLM embedding space to obtain \(E_m \in \mathbb{R}^{1 \times d}\). We study the effect of this aggregation choice in Section 5.3.

Cross-fusion representation – The clip-level features \((\bar{t}_m, \bar{t}_v)\) are fed into a trainable cross-fusion module, following [11], producing a compact fused representation \(t_f\) that aggregates motion-relevant cues. This fused representation is projected to the LLM embedding space to obtain \(E_f \in \mathbb{R}^{1 \times d}\) for report generation. In parallel, it is projected to a scoring embedding \(E_s \in \mathbb{R}^{1 \times d_r}\) for severity prediction by the scoring branch, where \(d_r\) is the input dimension of the regression head.

Token fusion and LLM conditioning – For report generation, TraMP-LLaMA conditions the LLM on evidence from three sources: the visual token embeddings \(E_v\) (RGB appearance context), the trajectory-based motion embedding \(E_m\) (explicit motion cues), and the cross-fused embedding \(E_f\) (a compact motion-aware summary). We inject \(E_m\) and \(E_f\) as additional evidence tokens and concatenate them with the visual token sequence \(E_v\) to form the multimodal context for the LLM (i.e., \([E_v; E_f; E_m]\)). Together with the tokenised instruction prompt \(E_t\), the model is instruction-tuned to generate reports. The description instructions for training and inference are detailed in the Supplementary Materials.

Compared with conditioning the LLM directly on heterogeneous raw visual and motion token sequences, the cross-fusion pathway distills motion-relevant information into a compact summary under a limited token budget, reducing the burden on the LLM to infer fine-grained motor evidence from long and weakly aligned sequences. In addition, providing \(E_m\) alongside \(E_f\) helps retain explicit trajectory cues that may be reduced by fusion, enabling more precise evidence statements (e.g., directionality and asymmetry). The contribution of each evidence source (\(E_v\), \(E_m\), and \(E_f\)) is ablated in Section 5.3.

The tokenizer is frozen, and the base LLM is adapted using LoRA [54]. The model generates the report autoregressively conditioned on the instruction and the multi-modal evidence. The visual, motion, and fusion projectors follow the multimodal projector design in VideoLLaMA3 [17], implemented as a two-layer MLP with GELU activation. The regression projector is a single fully-connected layer.

Two-head outputs – TraMP-LLaMA produces two outputs per clip: a report \(\hat{r}\) generated autoregressively by the LLM conditioned on \(E_t\) and the evidence embeddings \([E_v; E_f; E_m]\), and a severity score \(\hat{s}\) predicted by a regression head from \(E_s\). The scoring branch bypasses the LLM.

4.3 Decoupled Training and Stop-gradient Design↩︎

A core design choice in our approach (as depicted in Fig. 3) is the decoupling between score learning and language supervision. TraMP-LLaMA is trained with two supervision signals: (i) clinical severity supervision for score prediction (\(\mathcal{L}_{score}\)), and (ii) text supervision for report generation (\(\mathcal{L}_{text}\)). Naively propagating both signals through shared spatio-temporal representations can introduce task interference, where \(\nabla \mathcal{L}_{text}\) driven by language generation distort the spatio-temporal features required for accurate severity grading. To mitigate this, we adopt a stop-gradient mechanism that decouples the optimisation of the scoring representations from the language-generation objective.

The scoring branch is optimised only by \(\mathcal{L}_{score}\), while the report-generation branch is optimised by \(\mathcal{L}_{text}\). We explicitly block \(\nabla \mathcal{L}_{text}\) from flowing into the spatio-temporal representations used for scoring, namely the motion encoder output \(\bar{t}_m\) and the fused representation \(t_f\). That is, we apply stop-gradient at \((\bar{t}_m, t_f)\) with respect to \(\mathcal{L}_{text}\), such that the motion encoder and the cross-fusion module are updated only by \(\mathcal{L}_{score}\), while detached \(\bar{t}_m\) and \(t_f\) can still be consumed as evidence by the report-generation branch.

Meanwhile, severity supervision does not update the LLM parameters: \(\nabla \mathcal{L}_{score}\) is not allowed to backpropagate into the LLM. During joint training, \(\mathcal{L}_{text}\) updates only the language-side lightweight adaptation components, i.e., LoRA parameters and projection layers (with the vision encoder frozen), whereas \(\mathcal{L}_{score}\) updates only the scoring branch, i.e., the motion encoder, cross-fusion module, and regression head. This gradient routing is empirically validated in Table ¿tbl:tab6:stop95gradient?.

4.4 Optimisation↩︎

TraMP-LLaMA is trained using a multi-task learning framework with two objectives: clinical severity scoring and structured report generation. For score prediction, we minimise the MSE loss between the predicted score \(\hat{s}\) and the target clinical rating \(s\), \[\mathcal{L}_{score}=\frac{1}{N}\sum_{i=1}^{N}(\hat{s}_i-s_i)^2~,\] where \(N\) denotes the batch size. For the text generation task, the LLM is fine-tuned via LoRA using the standard autoregressive next-token cross-entropy loss \[\mathcal{L}_{text}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{L_i}\log p_\theta\!\left(t_{i,j}\mid t_{i,<j},E_t,{E}_i\right)~,\] where \(t_{i,j}\) denotes the \(j^{th}\) ground-truth token of the reference report for sample \(i\), \(t_{i,<j}\) are the preceding ground-truth tokens, and \(L_i\) is the report length. \(p_\theta(\cdot)\) is the LLM next-token probability parameterised by \(\theta\), conditioned on the instruction prompt embeddings \(E_t\) and the multimodal evidence \({E}_i=[E_v;E_f;E_m]\). During training, we jointly optimise these objectives with the decoupled gradient routing as \[\mathcal{L}_{total}= \mathcal{L}_{text}+ \lambda \cdot\mathcal{L}_{score}~, \label{eq:loss}\tag{1}\] where \(\lambda\) balances the two terms. We ablate \(\lambda\) in Section 5.3.

5 Experiments and Results↩︎

This section evaluates TraMP-LLaMA on PFED5+ for joint severity scoring and structured report generation. We provide quantitative performance metrics, qualitative analysis of the generated reports, and ablation studies to validate key components such as evidence composition and stop-gradient decoupling. We use PFED5+ (as detailed in Section 3), which includes MDS-UPDRS scores and the structured text annotations. Following [1], [11], 30 subjects are allocated for training and 11 for inference. Faces are detected and cropped using the SCRFD [55]. Video frames are processed at a resolution of \(256 \times 256\). Each video clip is annotated with a severity score for quantitative evaluation and a structured report for report-generation evaluation.

Evaluation Metrics – We assess the model using metrics tailored to both severity regression and clinical report generation: for Severity scoring, Spearman’s rank correlation coefficient is reported to measure the monotonic relationship between predicted scores and clinical ratings, following [11]. For Report generation, The quality of generated clinical reports is evaluated against expert references using a combination of semantic and n-gram based metrics. For semantic similarity, we report BERTScore (F1) [56], which aligns reference and generated tokens via contextual embeddings and cosine similarity. Specifically, given a reference sequence \(t=(t_1,\dots,t_m)\) and a generated sequence \(\hat{t}=(\hat{t}_1,\dots,\hat{t}_n)\), let \(h(\cdot)\) denote the contextual embedding of a token and \(c_{ij}=\cos(h(t_i),h(\hat{t}_j))\). BERTScore computes \[P_{BERT}=\frac{1}{n}\sum_{j=1}^{n}\max_{i} c_{ij},\quad R_{BERT}=\frac{1}{m}\sum_{i=1}^{m}\max_{j} c_{ij},\quad F_1=\frac{2{P_{BERT}}R_{BERT}}{P_{BERT}+R_{BERT}}~.\] We compute BERTScore using DeBERTa-XLarge-MNLI¹ as the underlying contextual encoder. To quantify the overlap of word sequences, we report BLEU-4 [57] (modified n-gram precision with a brevity penalty) and ROUGE-L [58] (longest common subsequence–based overlap). We additionally report CIDEr [59], which measures term frequency–inverse document frequency weighted n-gram consensus, down-weighting frequent generic phrases and emphasising informative terms. All text metrics are computed on the full generated report under the template, using a single expert reference per clip.

5.1 Implementation Details↩︎

Base VLM – We build on the VideoLLaMA3-7B Stage-3 fine-tuned checkpoint² [17], which uses Qwen2.5-7B [53] as the language backbone with embedding dimension \(d=3584\) and a maximum context length of \(8192\) tokens. The vision encoder is SigLIP-NaViT [52] with hidden size \(1152\). To reduce the effective visual token length, we apply (i) a video merge size of \(2\), which reduces the spatial token grid by a factor of \(2\) in each spatial dimension (i.e., \(4\times\) fewer tokens per frame) before visual encoding, and (ii) token compression (enabled for both training and inference), which prunes temporally redundant tokens after visual encoding, yielding the processed visual tokens \(t_v\) (default threshold \(0.1\)).

We sample RGB frames at \(25\) fps and set the maximum number of frames to \(T_v=64\). If a clip contains fewer than \(64\) frames, all available frames are used; otherwise, \(64\) frames are uniformly sampled. Frames are resized to \(224\times224\). To accommodate variable-length clips without temporal padding, SigLIP-NaViT uses NaViT-style [60] token packing with attention masking, packing tokens from multiple clips into a single sequence so each clip is processed at its true length and attention does not cross clip boundaries.

Motion stream – The motion encoder is SkateFormer [51] pretrained on landmark trajectories from DFEW [61]. Following [11], the SkateFormer encoder consists of four blocks with dimensions \((128, 256, 256, 256)\), respectively. We set the temporal length to \(T_m=T_v=64\) and apply zero padding when a clip contains fewer than 64 frames, as SkateFormer requires fixed-length trajectory inputs. We apply random clip reversal as motion augmentation during training.

Fusion and projection layers – We follow [11] to use three cross-fusion blocks, each producing the fused features with the dimension of 256. The visual projector follows the VideoLLaMA3 design, implemented as a two-layer MLP with a GELU non-linearity in between, mapping visual token features to the LLM embedding space (i.e., \(1152 \rightarrow 3584 \rightarrow 3584\)). The motion and fusion projectors use the same MLP design to map the features with the dimension of 256 to the LLM space (i.e., \(256 \rightarrow 3584 \rightarrow 3584\)). To interface the visual stream with the fusion module, we apply a linear video-to-fusion projection (\(1152\rightarrow256\)). For severity scoring, the regression branch consists of a linear layer (\(256\rightarrow256\)) followed by the final linear head (\(256\rightarrow1\)).

Training regime – LoRA [54] is used to adapt the LLM with rank \(\mathrm{lora}_r=64\), \(\mathrm{lora}_\alpha=128\), and dropout \(0.05\). LoRA is applied to attention projections \((q, k, v, o)\) and MLP projections \((up, down)\) in each Transformer layer. Module-wise learning rates are set to \(5\times10^{-6}\) for LoRA parameters and the visual projector, and \(5\times10^{-5}\) for the remaining trainable components, including the video-to-fusion projector, motion projector, fusion projector, fusion modules, and the regression projector/head.

All trainable parameters are optimised with AdamW (weight decay \(0.01\)) using a cosine learning-rate schedule with a warmup ratio of \(0.03\). Training runs for 6 epochs with a per-GPU batch size of 1 on 4 NVIDIA GH200 GPUs. We do not use gradient accumulation (accumulation steps \(= 1\)), resulting in an effective batch size of 4. Mixed-precision training uses BF16, and gradient checkpointing is enabled for memory efficiency. DeepSpeed ZeRO-2 [62] is used to partition optimiser states and gradients across data-parallel workers.

For scoring, clinical severity labels are normalised from \([0,4]\) to \([0,1]\) during training (i.e., \(y \leftarrow y/4\)) and rescaled back at inference. The overall objective weights the regression term (i.e., MSE loss) by \(\lambda=5\). We use SDPA attention for compatibility on the aarch64-based GH200 environment.

5.2 Comparative Results↩︎

We evaluate TraMP-LLaMA on PFED5+ for both severity scoring and report generation. Since TraMP-LLaMA is built upon VideoLLaMA3 [17], the most direct comparison is with this architecturally matched baseline. For severity scoring, we further compare against representative DFER baselines (Former-DFER [63] and S2D [64]) and AQA baselines (USDL [65], CoFInAl [66], QAFE-Net [1], and TraMP-Former [11]) as broader reference methods. For report generation, we additionally include recent general-purpose VLMs, including Chat-UniVi [67], LLaVA-Video [68], and LongVU [36]. These models are used only for report-generation comparison, as they are not designed for score regression.

Unlike previous action quality assessment works, where models are trained and evaluated per expression, we train a single model jointly across all PFED5+ actions to better reflect the target clinical setting. DFER baselines are converted to regressors by replacing the classification head with a three-layer MLP regression head and initialised from their DFEW fine-tuned checkpoints. AQA baselines are initialised from their publicly released pretrained weights. All scoring baselines are then fine-tuned on PFED5+ under the same training protocol. To align with the joint-action training setting, we add an auxiliary cross-entropy objective for predicting the PFED5+ action identity alongside the primary regression loss, weighted equally.

For report-generation baselines, each VLM is initialised from its publicly released fine-tuned checkpoint. For fair comparison, the vision encoder is kept frozen, while the multimodal projector and the LoRA parameters of the LLM remain trainable. All report-generation baselines are fine-tuned using the same instruction format, frame sampling strategy, and decoding settings unless otherwise specified. In all result tables, the best result is highlighted in bold and the second-best is underlined. All metrics are reported in (\(\%\)) except CIDEr [59], which is reported on its original scale because it is non-negative and not upper-bounded, making percentage-style scaling less interpretable.

Score Prediction – Table ¿tbl:tab6:pfed595score? reports the comparative results of severity scoring on PFED5+ across five clinical actions. TraMP-LLaMA achieves the best performance, obtaining \(57.35\%\) in average correlation across actions and \(56.63\%\) in all-action correlation, outperforming all adapted DFER and AQA baselines, as well as the matched VLM baseline VideoLLaMA3.

Compared with QAFE-Net (Avg. \(53.11\%\)) and TraMP-Former (Avg. \(45.04\%\)), TraMP-LLaMA achieves higher average correlation, with gains of \(\uparrow4.24\%\) and \(\uparrow12.31\%\), respectively. These comparisons indicate that the proposed framework remains competitive against representative quality-assessment baselines even under the more challenging joint-action setting. Importantly, compared with the same base VLM (VideoLLaMA3), adding trajectory cues with cross-fusion improves the average correlation by \(\uparrow2.29\%\) and the overall correlation by \(\uparrow5.14\%\). The gains are most evident on ‘Sit at rest’ (\(\uparrow10.40\%\)), ‘Frown’ (\(\uparrow6.83\%\)), and ‘Squeeze eyes’ (\(\uparrow9.76\%\)), indicating that trajectory cues are especially helpful for these actions in the joint-action setting.

For ‘Smile’ and ‘Clench teeth’, TraMP-LLaMA underperforms VideoLLaMA3. This may reflect ambiguity in the mouth region, where subtle smiles and mild clenching can appear visually similar under limited teeth visibility or partial occlusion. In such cases, trajectory cues may be less reliable or less discriminative, especially under joint-action training.

Report Generation – Table 1 compares structured report generation performance on PFED5+ against VLM baselines. The most direct comparison is with VideoLLaMA3, since TraMP-LLaMA is built on the same backbone and uses the same LLM (i.e., Qwen2.5-7B). Under this matched setting, TraMP-LLaMA achieves better results across all metrics, including 67.39% on BERTScore, improving over VideoLLaMA3. Although the absolute differences are modest, the consistent gains across all metrics suggest that the proposed motion-aware design helps improve report generation quality, yielding slightly better alignment with expert references across both semantic-similarity and n-gram–based metrics.

Compared with other generic VLMs, both TraMP-LLaMA and VideoLLaMA3 perform substantially better. This gap may reflect both task mismatch and differences in backbone capability. PFED5+ requires the model to capture subtle, region-level facial motion and express it in a structured report format, which may be less well aligned with VLMs trained mainly for broad video understanding. VideoLLaMA3 provides a stronger starting point in this setting, and TraMP-LLaMA further improves on it by introducing explicit trajectory-based motion cues. Nevertheless, the improvements are small, indicating substantial room to strengthen report reliability. In particular, format deviations (e.g., missing required stages), empty outputs, and unsupported or incorrect region statements may still be observed in a small number of cases, which can limit clinical usability even when overlap-based metrics improve.

5.3 Ablations↩︎

We ablate key design choices in TraMP-LLaMA, including loss balancing, evidence composition for LLM conditioning, temporal fusion design, and the stop-gradient decoupled training strategy. Unless stated otherwise, all ablations follow the same training and evaluation protocol as in Section 5.1.

Impact of Loss Balancing – We vary \(\lambda\in\{1,3,5,7,10\}\) in Eq. 1 (with the stop-gradient routing fixed as described in Section 4.3) to balance the regression and generation objectives. Here, \(\mathcal{L}_{score}\) is computed on normalised labels in \([0,1]\). As shown in Table ¿tbl:tab6:lambda?, \(\lambda=5\) provides the best overall trade-off, achieving the highest overall correlation (\(56.63\%\)), BERTScore (\(67.39\%\)), and CIDEr (\(0.2847\)), while maintaining competitive performance on the remaining metrics. We therefore use \(\lambda=5\) as the default setting in all subsequent experiments.

Gradient Flow Analysis – Table ¿tbl:tab6:stop95gradient? shows the ablation of the gradient-routing choice within the decoupled training scheme (as detailed in Section 4.3). Specifically, we test whether the text-generation objective \(\mathcal{L}_{text}\) should be allowed to backpropagate into the motion-aware representations used for scoring, namely the motion encoder output \(\bar{t}_m\) and the fused representation \(t_f\) in Fig. 3, thereby updating the motion encoder and cross-fusion module, or whether these gradients should be blocked as in our stop-gradient design. We compare two settings while keeping all other configurations identical: (i) enabling \(\nabla \mathcal{L}_{text}\) to update the motion encoder and cross-fusion module, and (ii) blocking \(\nabla \mathcal{L}_{text}\) at \((\bar{t}_m, t_f)\) (ours). Table ¿tbl:tab6:stop95gradient? shows that blocking text gradients yields substantially higher scoring correlation and also improves report quality, confirming the benefit of the proposed stop-gradient routing for stable severity grading. We additionally attempted naive joint optimisation without stop-gradient routing; under the same training setup, optimisation was highly unstable and did not yield reproducible convergent runs.

Effectiveness of Conditioning Tokens – We study whether injecting the trajectory conditioning token \(E_m\) and the fused motion summary token \(E_f\) into the LLM input is redundant or beneficial beyond visual tokens \(E_v\). As can be seen in Table ¿tbl:tab6:evidence95features?, using all conditioning tokens (\(E_v+E_m+E_f\)) achieves the best BERTScore and CIDEr, indicating improved semantic alignment and informative phrase matching. Interestingly, excluding \(E_f\) yields higher ROUGE-L and BLEU-4, but lower BERTScore and CIDEr, suggesting a trade-off between surface-form overlap and semantic matching. Since \(E_m\) and \(E_f\) are each injected as a single token, their overhead is negligible relative to the visual token sequence. Moreover, due to stop-gradient routing, adding these tokens does not affect the scoring pathway (Table ¿tbl:tab6:stop95gradient?). We therefore retain both \(E_m\) and \(E_f\) in the default setup.

Temporal Information Aggregation – We ablate the temporal form of the fused representation by comparing a frame-wise fusion-feature sequence against global average pooling (GAP). To isolate the effect of temporal aggregation in the fusion pathway, we fix the LLM evidence to \(E_v+E_f\) and exclude the landmark token \(E_m\). As shown in Table ¿tbl:tab6:pooling?, GAP yields higher overall scoring correlation and slightly better text metrics. This suggests that pooling can suppress frame-level jitter and provide a more compact conditioning signal under a limited context budget. Notably, retaining the full sequence benefits ‘Smile’ and ‘Clench teeth’, which may indicate that fine-grained temporal dynamics help disambiguate visually similar lower-face actions. Again, since the stop-gradient routing, the score-prediction results under the \(E_v+E_f\) setting are consistent with those reported in Table ¿tbl:tab6:stop95gradient?.

6 Conclusions↩︎

This work studied a joint scoring and reporting method for video-based, fine-grained FEQA. The proposed TraMP-LLaMA model is thus a unified framework that predicts clinically aligned severity scores while generating structured, evidence-based motion reports. To support this evidence-based report generation, we extended PFED5 with expert-guided structured motion descriptions, creating PFED5+. The model leverages a pretrained VLM backbone with instruction tuning for report generation and incorporates trajectory-based motion cues together with a cross-fusion pathway to support fine-grained assessment under clinically realistic joint-action training. Experiments on PFED5+ demonstrated that TraMP-LLaMA achieves SOTA performance for severity scoring and improves report generation quality over strong VLM baselines. Ablation studies further support the effectiveness of the proposed design choices, particularly the evidence-token formulation and the decoupled stop-gradient training scheme for stable joint optimisation.

Although this framework can provide interpretability for FEQA, the joint scoring and reporting approach still has two practical limitations. First, as a generative VLM, TraMP-LLaMA may occasionally hallucinate region-specific details, especially when motion cues are subtle or partially occluded. Second, PFED5+ remains relatively small and imbalanced across severity levels, and report supervision is constrained by single-reference annotations and inter-rater variability. Broader data collection and stronger supervision, for example multi-reference reports or temporally localised annotation, could further improve both measuring reliability and interpretability.

Supplementary Materials↩︎

This supplementary materials provides additional details on the text annotations in the PFED5+ dataset, including the exact templates, slot schema, and refinement prompts used to construct the motion-description labels, as well as the description instructions used for training and inference.

7 Qualitative Comparison of Generated Reports↩︎

Here, we present qualitative visualisations of generated reports of open-source baseline and our proposed method in Figs. 4 and 5. For each example, we show the Ground Truth report alongside outputs from VideoLLaMA3 and TraMP-LLaMA (ours). To keep the comparison concise, we only highlight the differing or most informative parts. Colours identify different methods, and the same colours are used to mark the corresponding key differences.

Despite generally producing well-structured reports, a small number of outputs exhibit formatting and content issues. In particular, the model may omit descriptions for certain facial regions (e.g., missing eyebrows or eyelids while describing only the mouth or cheeks) and occasionally generate incorrect region-specific details that are not supported by the visual evidence.

8 Text Annotation Details for PFED5+↩︎

8.1 Full Templates↩︎

For task-elicited action clips (‘smile’, ‘frown’, ‘squeeze eyes’, ‘clench teeth’), each description follows the same three-phase structure:

Action template (three-phase).
Prior to the onset of the action, [pre-action observations]. During the action, [in-action observations]. Following the completion of the action, [post-action observations].

The ‘sit at rest’ task does not contain a distinct onset, peak, and offset. It is therefore described using a single-phase template that corresponds to the in-action component:

Sit-at-rest template (single-phase).
During this time, [resting-state observations].

8.2 Local-region Structured Annotation Schema↩︎

Before generating the final paragraph-level text, each clip is first represented as a structured record containing (i) phase keys and (ii) region-state pairs.

8.2.1 Region slots↩︎

The structured record uses region-specific slots that reflect clinically relevant facial areas. In PFED5+, the following slots are used:

• Pre-action/Post-action: Eyebrows, Cheeks, Mouth, and optional global indicators such as Blink when visible

• In-action: Eyebrows, Eyelids, Cheeks, Mouth Corners, Mouth

8.2.2 Example for ‘Smile’↩︎

Pre-action: {Eyebrows: Neutral; Cheeks: Neutral; Mouth: Closed}
In-action: {Eyebrows: Neutral; Eyelids: Neutral; Cheeks: Raised with wrinkles; Mouth Corners: Pulled outward and raised; Mouth: Open showing teeth}
Post-action: {Blink: Once; Eyebrows: Neutral; Cheeks: Neutral; Mouth: Closed}

8.3 Annotation Statistics↩︎

We summarise the textual labels to characterise annotation density and consistency, including average description length in words, phase-level description length for pre/in/post components, and the prevalence of explicit no movement statements and [state unobservable] placeholders. Results are reported in Table ¿tbl:tab:annotation95statistic?.

8.4 GPT-4o Refinement Prompt↩︎

You are a language expert refining structured facial expression descriptions for a dataset.

Your task is to improve grammatical fluency and remove redundancy while strictly preserving all factual details and the original order of observations.

Constraints:

1) Do not add any new observations or infer any hidden states (e.g., intent, affect, diagnosis).

2) Do not remove any information about facial details (eyebrows, eyelids, cheeks, mouth corners, mouth).

3) Keep the same template structure. - For ‘sit at rest’, keep the format: “During this time, ...” - For action clips (‘smile’, ‘frown’, ‘squeeze eyes’, ‘clench teeth’), keep the format: “Prior to the onset of the action, ... During the action, ... Following the completion of the action, ...”

4) Preserve all evidence statements in meaning. You may rephrase only for fluency and redundancy reduction.

5) Output a single fluent paragraph.

9 Description Instructions↩︎

9.1 Instruction Diversification and Usage↩︎

To reduce prompt sensitivity and improve robustness to instruction phrasing, we use an instruction diversification strategy during training. We prepare two instruction pools corresponding to the two clip types in PFED5+:

• ‘Sit-at-rest’ clips

• Action clips

For each clip type, we use ChatGPT-4o to create 20 semantically equivalent instruction variants, based on the base instruction and the prompt below.

Please generate 20 instructions that are semantically equivalent to [the base instruction] but vary in wording and structure. Ensure that each new instruction conveys the same core request but uses different phrasing and sentence construction.

During training, one instruction is randomly sampled from the relevant bank for each clip at each iteration. At inference time, we always use the base instruction in the corresponding pool. Since the inference prompt is fixed, the instruction token embedding \(E_t\) is constant during inference.

9.2 ‘Sit-at-rest’ Instructions↩︎

1. Provide a brief, objective summary of the person’s facial state and minor visible movements, focusing on the eyebrows, eyelids, cheeks, mouth corner, and mouth, and noting any additional details such as blinking, chin states, gaze shifts, or wrinkle changes. (Base Instruction)

2. Describe the person’s facial appearance in a concise and neutral way, emphasizing the state of the eyebrows, eyelids, cheeks, mouth corners, and mouth, and include any subtle motions such as blinking, gaze direction, or wrinkle changes.

3. Give a short, factual account of the person’s facial expression and any slight movements, mentioning the eyebrows, eyelids, cheeks, corners of the mouth, and mouth, as well as other minor cues like blinking or shifts in gaze.

4. Summarize briefly and objectively how the person’s face appears, focusing on the position and tension of the eyebrows, eyelids, cheeks, and mouth region, and note if there are tiny movements such as blinks or eye direction changes.

5. Provide an objective and succinct description of the person’s facial condition, paying attention to the eyebrows, eyelids, cheeks, mouth corners, and mouth, and add observations on any minute actions like blinking or wrinkle variations.

6. Write a concise summary of the face’s current state, concentrating on the eyebrows, eyelids, cheeks, and mouth area, and point out any subtle visible movements such as a blink or a change in gaze.

7. Offer a neutral, compact description of how the person’s facial features appear, focusing on key areas-the eyebrows, eyelids, cheeks, and mouth-and mention minor activities like blinking or muscle twitches if visible.

8. Describe, in a brief and impartial tone, the current state of the person’s face, with attention to the eyebrows, eyelids, cheeks, corners of the mouth, and mouth, noting any small or brief movements.

9. Give a short, objective outline of the facial state, including observations about the eyebrows, eyelids, cheeks, and mouth area, and mention any fine motions such as blinking or small shifts in gaze.

10. Provide a compact and factual account of the facial expression, focusing on eyebrow shape, eyelid position, cheek form, and mouth posture, as well as any minimal visible movement like blinking or wrinkle formation.

11. Write a brief, observation-based summary describing the person’s facial appearance-covering the eyebrows, eyelids, cheeks, mouth corners, and mouth-and include minor dynamic details such as gaze movement or blinking.

12. Present an objective and to-the-point description of the individual’s face, noting the positions of the eyebrows, eyelids, cheeks, and mouth, and record any tiny motion such as eye blinks or chin shifts.

13. Offer a concise report of the facial state, describing the eyebrows, eyelids, cheeks, and mouth area in a neutral tone, while also mentioning subtle physical indicators like blinking or wrinkle presence.

14. Provide a short description that objectively summarizes the face’s condition, including eyebrow placement, eyelid openness, cheek tension, and mouth shape, with attention to any faint movements.

15. Summarize the person’s facial appearance clearly and briefly, focusing on static features like eyebrows and cheeks, and any minor visible changes such as blinking or slight mouth movement.

16. Describe succinctly the face’s resting state, detailing the appearance of the eyebrows, eyelids, cheeks, and mouth, and mention any small visible actions like a blink or shift in gaze.

17. Give a clear, unbiased summary of the person’s current facial posture, emphasizing eyebrows, eyelids, cheeks, and the mouth area, and include minor cues such as blink frequency or wrinkle changes.

18. Write a neutral, brief portrayal of the individual’s face, concentrating on the eyebrows, eyelids, cheeks, and mouth region, with mention of tiny or barely noticeable movements.

19. Produce a concise objective description of how the face looks, focusing on the eyebrows, eyelids, cheeks, and mouth corners, and noting any faint physical motion such as blinking or gaze drift.

20. Provide a factual, succinct statement of the facial expression, covering the state of the eyebrows, eyelids, cheeks, and mouth, and mention any delicate or minimal visible movements.

9.3 Action Instructions↩︎

1. Provide an objective, phase-by-phase description of the face-pre-action, during-action, and post-action-highlighting the eyebrows, eyelids, cheeks, mouth corners, and lips, and mentioning any additional cues like blinking, chin movement, wrinkles, or gaze changes. (Base Instruction)

2. Describe the facial movement in up to three phases-before, during, and after the action-focusing on the eyebrows, eyelids, cheeks, mouth corners, and lips, and noting any other visible details such as blinking, chin position, wrinkles, or gaze shifts.

3. Give a concise account of facial changes across up to three stages-before, during, and after the action-covering the eyebrows, eyelids, cheeks, mouth corners, and lips, and including any noticeable features such as blinking, chin motion, wrinkles, or gaze variation.

4. Summarize the face across up to three stages-prior to, during, and following the action-describing the eyebrows, eyelids, cheeks, mouth corners, and lips, and noting other observable elements like blinking, chin shifts, wrinkle formation, or gaze movement.

5. Offer an objective description of facial dynamics in up to three stages-before, during, and after the movement-focusing on eyebrows, eyelids, cheeks, mouth corners, and lips, and adding details like blinking, chin position, wrinkles, or gaze direction.

6. Provide a stage-wise description of the facial state-pre-action, action, and post-action-covering eyebrows, eyelids, cheeks, mouth corners, and lips, and mentioning other visible traits such as blinking, chin adjustments, wrinkle changes, or gaze shifts.

7. Describe the face across up to three sequential phases-before, during, and after the action-highlighting eyebrows, eyelids, cheeks, mouth corners, and lips, and noting other subtle indicators like blinking, chin motion, wrinkles, or eye gaze.

8. Give an objective three-phase overview of the facial expression-before, during, and after the action-detailing eyebrows, eyelids, cheeks, mouth corners, and lips, along with additional observations like blinking, chin posture, wrinkles, or gaze changes.

9. Offer a clear and factual summary of the facial movement in up to three parts-before, during, and after the action-mentioning eyebrows, eyelids, cheeks, mouth corners, and lips, and noting extra cues such as blinking, chin adjustments, wrinkles, or eye direction.

10. Provide a structured account of facial behavior over up to three phases-before, during, and after the action-covering eyebrows, eyelids, cheeks, mouth corners, and lips, and adding any visible signs like blinking, chin movement, wrinkle development, or gaze variation.

11. Describe facial activity in up to three temporal segments-before, during, and after the action-focusing on eyebrows, eyelids, cheeks, mouth corners, and lips, and including any other visible details like blinking, chin motion, wrinkles, or gaze shifts.

12. Summarize facial changes in up to three phases-before, during, and after the action-addressing eyebrows, eyelids, cheeks, mouth corners, and lips, while also noting any other details such as blinking, chin position, wrinkles, or gaze orientation.

13. Provide an objective breakdown of facial movement-before, during, and after the action-describing eyebrows, eyelids, cheeks, mouth corners, and lips, and mentioning any additional visible cues like blinking, chin posture, wrinkles, or gaze movement.

14. Offer a concise, stage-based description of the facial action-pre-action, during-action, and post-action-covering eyebrows, eyelids, cheeks, mouth corners, and lips, and including other observable features such as blinking, chin shifts, wrinkles, or gaze changes.

15. Give a stepwise description of the face through up to three stages-before, during, and after the action-mentioning eyebrows, eyelids, cheeks, mouth corners, and lips, and noting further details like blinking, chin position, wrinkle changes, or eye movement.

16. Provide an objective depiction of facial changes across up to three phases-prior to, during, and following the action-focusing on eyebrows, eyelids, cheeks, mouth corners, and lips, and noting other perceptible elements like blinking, chin adjustments, wrinkles, or gaze shifts.

17. Describe the facial expression over up to three stages-before, during, and after the action-highlighting eyebrows, eyelids, cheeks, mouth corners, and lips, and mentioning additional cues such as blinking, chin motion, wrinkles, or gaze direction.

18. Offer a three-phase summary of facial movements-before, during, and after the action-covering eyebrows, eyelids, cheeks, mouth corners, and lips, and noting any other visible characteristics like blinking, chin shifts, wrinkles, or eye gaze changes.

19. Give an objective overview of the facial sequence-before, during, and after the action-focusing on eyebrows, eyelids, cheeks, mouth corners, and lips, and adding extra observations such as blinking, chin movement, wrinkles, or gaze variation.

20. Provide a factual description of the facial movement in up to three phases-before, during, and after the action-mentioning eyebrows, eyelids, cheeks, mouth corners, and lips, and including any other visible details such as blinking, chin adjustments, wrinkles, or gaze direction.

References↩︎