1 Introduction↩︎

The pursuit of photorealistic audio-driven talking heads remains a central challenge in computer vision and graphics. Early methods [1]–[3] map acoustic features directly to lip movements using generative networks or employ primitive 3D representations [4], [5] for motion modeling. Despite recent remarkable progress [6]–[8], state-of-the-art methods still exhibit temporal instability and local artifacts such as lip jitter.

A milestone work, InsTaG [9] achieves impressive performance by decomposing the identity-specific motion into universal priors and personalized adaptation fields, revealing that high-fidelity talking-head synthesis could benefit from combining generalizable motion structures with individual expressive dynamics. Consequently, a fundamental tension exists: an ideal model must learn a hybrid motion field that captures identity-specific facial features and speaking style while generalizing robustly to unseen, identity-agnostic audio.

Balancing personalization and generalization, however, is non-trivial. Implicit models [10]–[12] learn a direct audio-to-motion mapping. Despite various technical refinements such as enhanced feature encoding [11], [12], these methods inherently lack structural constraints, often producing unstable or distorted facial dynamics (1 (a)). Conversely, explicit methods that incorporate geometric or anatomical priors introduce other limitations. Priors derived from statistical multi-person models, such as 3D Morphable Models [13], [14] (3DMMs), improve generalizable animation [15], [16] but often sacrifice person-specific expressiveness (1 (b)). In contrast, priors extracted directly from a target video, like per-frame Action Units [17] (AUs), provide rich stylistic cues but risk overfitting and poor generalization. Even recent hybrid methods, such as TalkingGaussian [8], only partially resolve this issue. These approaches often use explicit priors to stabilize the upper face, but rely on an unconstrained implicit model for the highly articulated lower face. By omitting structural guidance where it is most critical, this compromise leads to a loss of per-frame precision, which accumulates over time as temporal artifacts like lip jitter (1 (c)).

This analysis yields our key insight: photorealistic talking head generation requires leveraging person-specific (geometric and anatomical) priors while ensuring generalization to unseen (cross-identity) audio. To this end, we propose HM-Talker, a hybrid motion modeling framework for universal audio-driven facial animation. Unlike prior methods that rely solely on image-derived explicit features or audio-derived implicit features for lower face motion, HM-Talker harmonizes both merits to enable anatomical grounding-based reconstruction (prosodic features) while enabling audio-driven generalization.

Our framework achieves this balance through two synergistic components. First, the Cross-Modal Mapping Module (CMMM) constructs a comprehensive motion vocabulary by extracting implicit prosodic features from audio and explicit articulatory features from the reference video, while projecting audio features into the visual articulatory space to align with image-based representation via an Audio-to-Visual Mapper. This enables grounding-based reconstruction even under audio-driven conditions. Second, the Hybrid Motion Modeling Module (HMMM) employs a Stochastic Feature Pairing (SFP) strategy to adjust input features. Specifically, this training scheme dynamically alternates between fusing implicit audio features with (1) video-derived explicit priors to enforce personalization, and (2) audio-predicted explicit features to alleviate identity-dependent biases in explicit motion and enhance cross-identity generalization under audio-driven conditions.

Our main contributions are threefold:

• We introduce HM-Talker, a novel paradigm for audio-driven talking head generation by strategically reconciling the fundamental trade-off between personalization and generalization.

• We propose the Cross-Modal Mapping Module (CMMM), which prepares a comprehensive vocabulary of motion cues from both audio and video sources, enabling the model to synthesize anatomically plausible facial movements.

• We design the Hybrid Motion Modeling Module (HMMM) with a Stochastic Feature Pairing strategy, where dynamical optimization alternately reinforces personalization and generalization to learn a robust and adaptable facial motion synthesizer.

2 Method↩︎

2.1 Preliminary: Deformable Gaussian Avatars↩︎

Our framework builds upon the deformable 3D Gaussian Avatars paradigm, specifically adopting TalkingGaussian [8] as the foundational backbone. This approach offers high-fidelity rendering and an explicit decomposition of the talking head into a static identity representation and a dynamic motion field.

Canonical Representation. A subject’s identity and geometry are encoded by a set of Persistent 3D Gaussians, parameterized by \(\theta = \{\mu, s, q, \alpha, f\}\), which represents 3D position, scale, rotation, opacity, and spherical harmonics coefficients, respectively. The Gaussian centers \(\mu\) in this canonical space collectively form a deformable template capturing the subject’s unique facial structure.

Dynamic Motion Representation. Facial animation is driven by a motion field, which predicts a per-Gaussian 3D displacement \(\delta = \{\Delta \mu, \Delta s, \Delta q\}\) from the canonical template. This displacement is conditioned on the Gaussian’s canonical position \(\mu\) and a condition vector \(\mathbf{C}\): \[\delta = \text{MLP}(\mathcal{H}(\mu) \oplus \mathbf{C}),\] where \(\mathcal{H}(\cdot)\) is tri-plane hash encoding [12] for high-frequency spatial localization, and \(\oplus\) indicates feature concatenation. The condition vector \(\mathbf{C}\) incorporates features from the driving signal (e.g., audio) alongside explicit parameters such as expression coefficients. The objective is to learn a mapping to \(\delta\) so that the resulting motion field produces realistic and subject-specific facial animations. Limitation of TalkingGaussian. In TalkingGaussian, the motion field \(\mathbf{C}\) is derived via a spatially partitioned scheme that combines implicit audio features with explicit expression parameters. Specifically, the upper face is governed primarily by explicit priors, whereas the lower face is driven predominantly by audio cues. Although this design effectively disentangles facial regions, it compromises stability. Specifically, the absence of explicit structural guidance for the lower face often results in motion jitter and imprecise lip synchronization. To resolve this instability, we introduce a hybrid motion formulation by integrating implicit and explicit cues for the lower face reconstruction while the dynamical optimization alternately reinforces personalization and generalization.

2.2 Overview of HM-Talker↩︎

To overcome these limitations, we propose HM-Talker, a hybrid motion modeling framework designed to learn a motion field that balances generalization accuracy and personalized modeling. As illustrated in 2, our design focuses on improving motion learning specifically for the lower face region—where speech-related dynamics are most complex—while preserving explicit, prior-driven guidance for the upper face. A key feature of HM-Talker is its prior-flexible design: while we instantiate explicit motion representations using Action Units (AUs) for concreteness, the framework is readily adaptable to other motion priors, such as 3D Morphable Models (3DMMs) and BlendShapes [18].

HM-Talker comprises two synergistic components. The Cross-Modal Mapping Module (CMMM) first extracts explicit facial representations from video, distinguishing between the upper (\(c^e_{v,u}\)) and lower (\(c^e_{v,l}\)) regions. It also derives implicit motion features from audio (\(c^i_{a,l}\)) and projects them into a visual articulatory space (\(c^e_{a,l}\)) to establish cross-modal alignment. The Hybrid Motion Modeling Module (HMMM) subsequently fuses these implicit and explicit cues via a gated attention. Crucially, during training, we employ a stochastic pairing of feature sets to promote identity-specific personalization while simultaneously enhancing generalization to unseen speakers and audios.

HM-Talker is trained in a multi-stage paradigm, encompassing static identity initialization, motion learning, and fine-tuning. The overall objective is defined as: \[\mathcal{L}_{\text{total}} = \mathcal{L}_{1} + \lambda_1 \mathcal{L}_{\text{D-SSIM}} + \lambda_2 \mathcal{L}_{\text{LPIPS}} + \lambda_3 \mathcal{L}_{\text{align}},\] where \(\mathcal{L}_{1}\), \(\mathcal{L}_{\text{D-SSIM}}\), and \(\mathcal{L}_{\text{LPIPS}}\) ensure pixel-level and perceptual fidelity. The alignment term is depicted as \[\mathcal{L}_{\text{align}} = \mathcal{L}_1(\mathbf{c}_{a,l}^{e}, \mathbf{c}_{v,l}^{e}), \label{eq:align}\tag{1}\] enforcing consistency between audio-derived and visual articulatory embeddings, thereby grounding the audio-driven motion in the subject’s physical expression. For ease of exposition, we focus our description on the Face Branch; the Inside Mouth Branch follows an analogous hybrid-driven strategy as the lower facial motion pathway.

2.3 Cross-Modal Mapping Module (CMMM)↩︎

CMMM establishes a motion vocabulary by projecting implicit audio features into explicit articulatory representations. This explicit cue space serves as a reliable guide for subsequent motion modeling.

Target Space: Video-Derived Explicit Representation. The target space is constructed from the ground-truth video data, providing a high-fidelity supervisory signal and motion cues for learning personalized motion. We extract frame-wise Facial Action Units (AUs) and divide them into upper face (\(\mathcal{A}_u\)) and lower face (\(\mathcal{A}_l\)) subsets. For the upper face, we form the representation \(\mathbf{c}_{v,u}^{e}\) via a straightforward concatenation of AUs, which is sufficient to capture the relatively rigid and less articulated dynamics of this region. The upper face representation \(\mathbf{c}_{v,u}^{e}\) is directly passed to the motion field decoder without modification, following the original TalkingGaussian framework.

For the lower face AUs, which govern complex articulatory motions, we employ a residual MLP to enhance expressiveness: \[\mathbf{c}_{v,l}^{e} = \mathrm{MLP}(\mathcal{A}_l) \oplus \mathcal{A}_l \in \mathbb{R}^{32}.\] The output dimension of \(32\) is chosen to match that of the audio-derived features, facilitating subsequent fusion. The residual connection ensures that the original anatomical information is preserved, while the non-linear branch captures complex correlations between AUs. This explicit vector \(\mathbf{c}_{v,l}^{e}\) encodes the subject’s personalized articulatory style, serving as the primary supervision for the audio-to-motion projection and as a direct motion cue in our hybrid model.

Source Space: Audio-Derived Implicit Representation. The source space is derived from the input audio using a pre-trained Audio-Visual Encoder (AVE) [6], which outputs a prosody-rich feature \(a \in \mathbb{R}^{512}\). These features are then processed by a two-stage network designed to distill articulatory information. First, AudioNet acts as a feature extractor, performing hierarchical temporal compression to produce compact embeddings. Subsequently, AudioAttNet refines these embeddings using an attention mechanism to focus on perceptually salient temporal regions, yielding the final implicit feature \(\mathbf{c}_{a,l}^{i} \in \mathbb{R}^{32}\). The implicit feature \(\mathbf{c}_{a,l}^{i}\) captures prosodic and temporal audio cues, thereby complementing the explicit visual features to generate accurate and personalized lower face motion. (Architectural details for AudioNet and AudioAttNet are provided in the Appendix).

Cross-Modal Projection. To project audio-derived features into the explicit articulatory space, we introduce Audio-to-Visual Mapper (A2VM), a lightweight MLP defined as \[\mathbf{c}_{a,l}^{e} = \mathrm{A2VM}(\mathbf{c}_{a,l}^{i}).\] This projection is supervised by the alignment loss \(\mathcal{L}_{\text{align}}\) (1 ), which encourages the predicted \(\mathbf{c}_{a,l}^{e}\) to match the video-derived target \(\mathbf{c}_{v,l}^{e}\). By learning this mapping, the CMMM produces lower face features that integrate audio-derived motion cues with explicit visual priors, subsequently serving as inputs to the HMMM.

2.4 Hybrid Motion Modeling Module (HMMM)↩︎

HMMM is designed to resolve the fundamental trade-off between personalization and generalization in talking-head synthesis. It consumes the vocabulary of motion cues from the CMMM and integrates our key technical contribution, the Stochastic Feature Pairing (SFP) strategy, with a dynamically-gated fusion mechanism.

Dual-Objective training in SFP. SFP operates by stochastically alternating between distinct feature-pairing paths during training, thereby balancing two primary objectives: capturing personalized motion and generalizing to unseen audio. At each iteration, one of the following paths is selected:

• Path 1 (\(\mathcal{P}_{\text{style}}\)) - Supervised Personalization: Pairs the implicit audio feature \(\mathbf{c}_{a,l}^{i}\) with the video-derived explicit representation \(\mathbf{c}_{v,l}^{e}\). Conditioned on the referenced motion cues, this path anchors the learned motion field to the subject’s unique articulatory style.

• Path 2 (\(\mathcal{P}_{\text{gen}}\)) - Audio-Driven Generalization: Pairs \(\mathbf{c}_{a,l}^{i}\) with the audio-predicted explicit feature \(\mathbf{c}_{a,l}^{e}\). This path, deprived of ground-truth video priors, compels the model to generate accurate motion from audio alone, ensuring a generalizable audio-to-motion mapping.

• Path 3 (\(\mathcal{P}_{\text{robust}}\)) - Auxiliary Robustness: Pairs a masked audio feature \(\mathbf{c}_{a,l}^{i\text{-mask}} = \mathcal{M}_a \cdot \mathbf{c}_{a,l}^{i}\) with \(\mathbf{c}_{v,l}^{e}\). \(\mathcal{M}_a\) is an element-wise random mask sampled uniformly (e.g., \([0.1,0.3]\)), scaling feature dimension to improve robustness and prevent over-reliance on the audio stream.

Unlike conventional multi-task learning that computes all losses simultaneously, our stochastic path selection is highly memory-efficient, as only one path is active per iteration. During inference, only the generalization path (\(\mathcal{P}_{\text{gen}}\)) is used, ensuring fully deterministic, audio-driven motion synthesis.

Dynamically-Gated Fusion. The explicit and implicit features selected by SFP are fused through a learnable gating mechanism: \[\mathbf{c}_f = \alpha \odot \mathbf{c}_{\cdot,l}^{e} + (1 - \alpha) \odot \mathbf{c}_{a,l}^{i*},\] where the gating weight is predicted via a lightweight MLP with a sigmoid activation: \[\alpha = \mathrm{sigmoid}(\mathrm{MLP_g}(\mathbf{c}_{a,l}^{i*} \oplus \mathbf{c}_{\cdot,l}^{e})),\] where \(\mathbf{c}_{a,l}^{i*}=\{\mathbf{c}_{a,l}^{i},\mathbf{c}_{a,l}^{i-mask}\}\) and \(\mathbf{c}_{\cdot,l}^{e}=\{\mathbf{c}_{a,l}^{e},\mathbf{c}_{v,l}^{e}\}\). This mechanism allows the model to adaptively balance the influence of structured explicit cues and prosodic implicit cues on a frame-by-frame basis.

Final Deformation Prediction. The fused lower face feature \(\mathbf{c}_f\) and the upper face feature \(\mathbf{c}_{v,u}^e\) are modulated by spatially-aware attention maps [19] to enable fine-grained, region-specific control. These maps are predicted by two lightweight MLPs (\(\text{MLP}_f, \text{MLP}_u\)) from the positional encoding \(\mathcal{H}(\mu)\), depicted as \[\mathbf{C}_f = \mathbf{c}_f \odot \text{MLP}_\text{f}(\mathcal{H}(\mu)), \mathbf{C}_u = \mathbf{c}_{v,u}^e \odot \text{MLP}_\text{u}(\mathcal{H}(\mu)).\] The final per-Gaussian deformation \(\delta_{\text{face}}\) is predicted by a deformation MLP that integrates all conditioned features: \[\delta_{\text{face}} = \text{MLP}(\mathcal{H}(\mu) \oplus \mathbf{C}_u \oplus \mathbf{C}_f).\] This hierarchical and spatially-attentive conditioning ensures the generated motion is both holistically coherent and preserves the detailed, personalized articulatory style learned.

3 Experiments↩︎

3.1 Experimental Settings↩︎

Dataset. Following established protocols in [12], [20], we evaluate our method on five publicly available portrait videos to ensure fair and impartial comparisons. The dataset comprises three male subjects (“Lieu",”Jae-in", and “Obama") and two female subjects (”May" and “Shaheen"), with an average duration of 7,637 frames captured at 25 FPS. All recordings maintain portrait-centered composition, predominantly at 512\(\times\)​512 resolution except for 450\(\times\)​450 resolutions for”Obama" and “Jae-in".

Baselines for Comparison. Our comparative analysis encompasses three distinct categories of contemporary approaches: 2D generative models (IP-LAP [2], TalkLip [21], DINet [3]), Neural Radiance Field (NeRF) based methods (AD-NeRF [10], RAD-NeRF [11], ER-NeRF [12], SyncTalk [6]), and 3D Gaussian Splatting (3DGS) based techniques (GaussianTalker [7], TalkingGaussian [8]). To further ensure a fair comparison, we re-implemented the TalkingGaussian baseline using our data preprocessing pipeline like SyncTalk’s, which is denoted as TalkingGaussian in our results.

Static Image Quality. We employ Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [22], and Learned Perceptual Image Patch Similarity (LPIPS) [23] metrics for evaluation.

Dynamic Motion Quality. Lip synchronization is evaluated using SyncNet [24], [25], which provides a Confidence Score (Sync-C) and an Error Distance (Sync-D). We also measure the Landmark Distance (LMD) [26] between generated and reference facial expressions. To further analyze the articulatory precision, we use Action Units (AUs) [27] extracted via OpenFace [28] and report the error for the lower facial region (AUE-L) and upper facial region (AUE-U).

Efficiency. We report the total Training Time (in hours) for model convergence on a single subject and the inference speed in Frames Per Second (FPS) on a 512\(\times\)​512 sequence.

Implementation Details. For each subject, we first train the Face Branch and Inside Mouth Branch in parallel for 50,000 iterations. During this stage, the Face Branch is driven by hybrid motion features selected according to the HMMM’s three-path strategy, with path proportions set to \(\mathcal{P}_{\text{gen}}:\mathcal{P}_{\text{robust}}:\mathcal{P}_{\text{style}} = 4{:}4{:}2\) and the audio masking ratio \(\mathcal{M}_a\) sampled uniformly from 0.1 to 0.3. Both branches are then jointly fine-tuned for an additional 15,000 iterations to stabilize articulation consistency. We use Adam [29] and AdamW [30] optimizers with a learning rate of \(5\mathrm{e}{-4}\), and set loss weights to \(\lambda_1 = 0.2\), \(\lambda_2 = 0.5\), and \(\lambda_3 = 1\mathrm{e}{-3}\). All experiments are conducted on a single NVIDIA RTX 3090 GPU. During inference, upper-face motion is driven by image-based explicit features, while lower-face and inner-mouth regions are generated from audio-only inputs.

3.2 Comparison with SOTA↩︎

[!b]
\setlength\tabcolsep{5pt}
\begin{center}
\vspace{-4mm}
\resizebox{0.9\textwidth}{!}{
\begin{tabular}{@{}c@{\hspace{12pt}}>{\hspace{-8pt}}l|ccc} 
\toprule
\addlinespace[0mm]
\rowcolor{gray!50} 
\multicolumn{2}{l|}{\multirow{2}{*}{}}
& \multicolumn{3}{c}{\textbf{Motion Quality}}  \\ 
\addlinespace[-0.1mm]
\rowcolor{gray!50} 
\multicolumn{2}{l|}{\multirow{-2}{*}{\textbf{Methods}}}       & \textbf{LMD} $\downarrow$ & \textbf{AUE-(L/U)} $\downarrow$ & \textbf{Sync-C} $\uparrow$  \\
\addlinespace[-0.6mm]
    \midrule
    \addlinespace[0mm]
    \multirow{3}{*}{\rotatebox[origin=c]{90}{GAN}}
    & Wav2Lip~\cite{chung2017lip}  & 2.948 & 0.70/- & \textbf{8.755}  \\
                                                  & IP-LAP~\cite{zhong2023identity}  & 3.161 & 1.00/- & 7.040  \\
                                                  & DINet~\cite{zhang2023dinet}   & 3.230 & 1.09/- & 7.455  \\
                                                  & TalkLip~\cite{wang2023seeing}   & 3.285 & 0.82/- & 6.657  \\
    \addlinespace[-0.6mm]
    \midrule
        \addlinespace[0mm]
    ~    &\cellcolor{gray!15}HM-Talker (Ours)  &  \cellcolor{gray!15}\textbf{2.514} & \cellcolor{gray!15}\textbf{0.53}/\textbf{0.22} & \cellcolor{gray!15} 7.807  \\
    \addlinespace[-0.6mm]
    \bottomrule
\end{tabular}}
\end{center}
\vspace{-6mm}
\caption{Motion quality comparison with 2D generative methods. HM-Talker achieves significantly better results across all metrics. The `-' for upper-face AU error (AUE-U) indicates that a reliable measurement could not be obtained, as 2D methods often generate motion only for the mouth region, while keeping the rest of the face from the original frame unchanged.}
\label{tab:sr2}

Self-Reconstruction. We evaluate HM-Talker under a 10:1 train–validation split across all datasets. As shown in [tab:sr1] [tab:sr2], our method achieves leading performance in both visual quality and motion fidelity. It outperforms SyncTalk (34.51 dB) and TalkingGaussian* (33.75 dB) with a PSNR of 35.15 dB, while achieving the lowest motion errors (LMD =2.514, AUE-L =0.53) and the highest synchronization confidence (Sync-C =7.807). These results support the effectiveness of explicitly learning a personalized motion field for achieving more stable and accurate facial dynamics. HM-Talker’s hybrid implicit–explicit framework captures individual articulatory patterns more faithfully, helping to mitigate lip jitter and improve synchronization. Although 2D generative models specialize in synchronization, HM-Talker surpasses top-performing baselines such as Wav2Lip on most motion metrics, while preserving superior 3D realism. Furthermore, it maintains real-time rendering speed (120 FPS) and fast convergence (0.6 h), matching the efficiency of 3DGS-based pipelines.

Generalization to Unseen Audio and Speakers. To assess generalization, we drive models trained solely on the “May” dataset using out-of-domain audio from unseen speakers—“Shaheen” (gender-matched) and “Lieu” (gender-mismatched). As shown in 1, HM-Talker achieves superior lip-synchronization accuracy across these settings, outperforming SyncTalk by a noticeable margin. These results indicate that the Hybrid Motion Modeling Module effectively disentangles phonetic content from speaker-specific acoustics, thereby facilitating a robust content-to-style mapping. Furthermore, the Stochastic Feature Pairing (SFP) strategy acts as a regularizer, explicitly encouraging generalization beyond the training speaker’s voice. The t-SNE visualization in 3 reveals that while implicit audio and explicit visual features reside in distinct subspaces, their fused representations form a continuous manifold bridging the two modalities, indicating effective cross-modal alignment and robust feature fusion.

Qualitative Comparison. We conduct qualitative, frame-level comparisons against state-of-the-art temporal modeling approaches: SyncTalk, TalkingGaussian*, and our HM-Talker. Key video frames corresponding to target phonemes are selected to critically assess phoneme-viseme alignment. As illustrated in 4, our approach generates the most visually consistent results with reference frames across various phoneme categories. For instance, during the articulation of wide-mouth phonemes (e.g., ) or subtle ones (e.g., ), our model maintains precise lip closure and shape, whereas competing methods exhibit noticeable misalignments, as indicated by the red boxes. For articulations such as , while baseline methods capture a broadly similar mouth aperture, our method reconstructs more fine-grained intra-oral details (highlighted in yellow), achieving superior perceptual realism. These observations demonstrate the efficacy of applying our hybrid motion modeling to the complete articulatory system. By jointly modeling external facial expressions and internal oral structures with a unified, hybrid-driven strategy, our framework ensures cohesive and synchronized movement across all components, resulting in realistic talking head synthesis.

User Study. We conduct a user study with 30 non-expert participants evaluating 35 videos (5 identities \(\times\)​7 methods including ground truth), each 20 seconds in duration. Participants rate Video Realness, Image Quality, and Lip-Sync Accuracy using a 5-point scale. Our method consistently outperforms all competitors across all metrics (5). In particular, our method achieves 4.31 score in Video Realness and 4.08 in Image Quality, surpassing the second-best method by margins of 17% and 23%, respectively. Moreover, our approach gains 4.10 score (Lip-Sync Accuracy), significantly narrowing the gap to the ground truth (4.36).

3.3 Ablation Studies on Self-Reconstruction↩︎

We perform comprehensive ablations to analyze the individual influence of fusion strategy and component configurations on the reconstruction stability and fidelity.

Fusion Strategy. We first ablate the fusion mechanism, which is responsible for integrating explicit structural priors with implicit prosodic cues to form a stable and expressive personalized motion field. We compare four variants: (a) Purely Implicit: setting \(\alpha=0\) during training, similar to the lower-face configuration in TalkingGaussian*; (b) Purely Explicit: setting \(\alpha=1\) during training, making the model depend solely on the explicit motion predicted by CMMM; (c) MLP Fusion, directly concatenating the two features, followed by an MLP; (d) Gated Fusion (ours), our proposed adaptive fusion. The results reveal a clear hierarchy (2). The unimodal baselines (a, b) perform poorly, confirming the necessity for hybrid modeling. While MLP Fusion (c) improves consistency but still lacks adaptive control for optimal performance. In contrast, our Gated Fusion (d) achieves the best performance, with an LMD of 2.514 and Sync-C of 7.807. This result validates that an adaptive gating mechanism is crucial for effectively combining personalized structural style with dynamic prosodic information.

Component Robustness. We next evaluate the robustness and contribution of individual components by ablating the following: (1) replacing the Action Unit prior with 3DMM or BlendShape [18] coefficients; (2) replacing the learnable Audio-to-Visual Mapper with a pretrained ExpNet encoder from SadTalker [31]; (3) using a different audio encoder (DeepSpeech [32]); (4) replacing the explicit motion stream with Gaussian noise.

Results reveal three key observations (2). Replacing the AU prior with 3DMM or BlendShape yields comparable results, showing that HMMM is prior-flexible. Replacing our A2VM with the fixed ExpNet encoder causes a notable drop in lip-sync accuracy, underscoring the benefit of our identity-specific, learnable audio-to-visual mapping. When using the less suitable DeepSpeech features, the fusion gate \(\alpha\) converges toward 1, and performance drops. This indicates that HMMM intelligently down-weights unreliable or noisy audio streams, relying more heavily on the trustworthy explicit pathway. Replacing the explicit stream with noise causes \(\alpha\) to drop to 0.2 (compared to  0.6 in the default setting). As illustrated in 6, such incomplete variants often produce inaccurate lip shapes or stiff motion. Furthermore, as shown in 7, \(\alpha\) exhibits dynamic, word-dependent fluctuations: for “The" and”jumps", it rises then falls in our model but stays flat or inverted under noise; for “lazy", the trend reverses. These patterns confirm that HMMM adaptively balances the two streams based on the reliability of the input cues.

3.4 Ablation Studies on Audio Generalization↩︎

To verify our proposed generalization mechanisms, we perform ablation studies on two components: the Audio-to-Visual Mapper (A2VM) and the Stochastic Feature Pairing (SFP) strategy. Experiments are conducted under increasing distribution shifts, including cross-identity and cross-lingual scenarios. The results are tabulated in 3.

Ablation Setup. We define four variants to isolate the effects of each component: (i) ablating A2VM by removing the module and its alignment loss, leaving SFP’s generalization path fully implicit; (ii) ablating SFP by disabling the generalization-focused paths (\(P_{\text{gen}}\), \(P_{\text{robust}}\)) and training only with the personalization objective (\(P_{\text{style}}\)); (iii) ablating both components, reducing the model to a fully implicit baseline; (iv) the full model with all components enabled.

Analysis of Stochastic Feature Pairing (SFP). Ablation studies confirm the indispensable role of SFP in enabling model generalization. Removing SFP results in a severe performance drop in both cross-identity (Sync-C: 8.663 \(\rightarrow\) 3.727) and cross-lingual (6.501 \(\rightarrow\) 1.871) settings. This decline is attributable to severe over-fitting on the source speaker’s identity and linguistic patterns when the model is trained solely on the personalization objective. These results collectively underscore that SFP’s multi-objective, stochastic training strategy is essential for learning a generalizable audio-to-motion mapping.

Analysis of Audio-to-Visual Mapper (A2VM). The role of A2VM is nuanced and task-dependent. In in-distribution data (S-to-S), removing A2VM slightly decreases performance (Sync-C: 10.269 \(\rightarrow\) 9.683), confirming its role as a structural regularizer that enhances articulatory precision. Conversely, in the cross-lingual task (M-to-S), A2VM ablation yields an improvement in Sync-D (lower is better) yet a degradation in Sync-C (6.501 \(\rightarrow\) 6.183). This indicates a trade-off: relying on a purely implicit mapping improves temporal consistency (as reflected in Sync-D) but at the expense of phonetic accuracy (Sync-C). Nevertheless, even under unseen language conditions, A2VM provides crucial structural guidance that enforces more precise viseme shapes. In summary, these findings indicate that cross-domain generalization is primarily enabled by the robust audio encoder, while A2VM serves as a precision-enhancing module, the effectiveness of which is modulated by the degree of domain shift.

Summary. Our analysis reveals that SFP provides a robust, domain-agnostic foundation for generalization, while A2VM offers a powerful but potentially domain-specific refinement. The full model, combining both, achieves the best overall trade-off between high-fidelity reconstruction and robust generalization across diverse speech conditions.

4 Conclusion↩︎

This paper presents HM-Talker, a novel hybrid motion modeling framework that solves the fundamental challenge in talking head generation: the reconciliation of person-specific anatomical priors with robust cross-identity audio generalization. Our key insight is to hybridize implicit audio features with explicit visual priors. This is realized through two components: a Cross-Modal Mapping Module (CMMM) that constructs a comprehensive, anatomically grounded motion vocabulary by aligning audio prosody with visual articulation, and a Hybrid Motion Modeling Module (HMMM) that uses a Stochastic Feature Pairing (SFP) strategy to dynamically train the model for both personalization and generalization. Extensive experiments confirm that HM-Talker achieves state-of-the-art performance, consistently producing high-fidelity, lip-synchronized, and identity-preserving talking head videos. By unifying previously competing objectives of personalization and generalization, our work establishes a new paradigm for universal facial animation.

5 Acknowledgement↩︎

This work was supported by the National Natural Science Foundation of China (62501189), the Natural Science Foundation of Heilongjiang Province of China for Excellent Youth Project (YQ2024F006) and Guangdong Basic and Applied Basic Research Foundation (2026A1515010184).

References↩︎