1 Introduction↩︎

Facial animation plays a crucial role in creating immersive experiences in games and animated films. Traditional production pipelines heavily rely on high-end 4D capture systems, professional actors, and skilled animators to achieve high-fidelity animations [1]. This enables directors and designers to follow a precise creative vision by leveraging the organic nuances of real-world speech and expressions. Despite these advantages, this approach is costly, labor-intensive, and inherently rigid, limiting post-capture flexibility and making it impractical for animating large casts of characters or producing content in multiple languages (e.g., dubbing).

Speech‑driven 3D facial animation has emerged as a scalable alternative to traditional capture‑based pipelines, with approaches broadly falling into two categories. Artist‑centric systems (e.g., viseme‑ or curve‑based rigs) [2]–[4] offer interpretability and fine control but require extensive manual tuning to achieve high‑quality expressive motion. In contrast, data‑driven methods [5]–[7] learn facial dynamics directly from real performances, offering a significantly higher ceiling for realism. These methods typically operate in vertex space rather than rig‑specific parameter spaces, which limits direct artistic controllability but provides a practical advantage: the generated motion is not tied to a rig parameterization, facilitating integration into different production setups.

A key challenge, therefore, lies in bridging the gap between realism and controllability. This motivates recent work to explore controllable generative frameworks, often conditioning on discrete labels concept tags such as emotions [8]–[11]. While effective for coarse adjustments, this paradigm is inherently rigid: models are trained on a fixed vocabulary of styles, requiring retraining to introduce new expressions. Moreover, expanding the label set significantly increases annotation cost, encouraging the use of broad categories (e.g., “angry”) that collapse expressive variability into caricature-like behaviors. Alternative approaches introduce control via text or image prompts [11]–[15]. However, such modalities struggle to capture subtle behavioral traits, speaker-specific dynamics, and temporal nuances of facial motion, due to language ambiguity and the static nature of images. As a result, achieving precise, expressive, and production-oriented controls over speech-driven facial animation remains an open challenge.

Example-based strategies [16]–[19] offer a compelling alternative by allowing reference videos or animations to guide the global delivery style, greatly simplifying the specification of the desired intent. However, these methods face two key limitations. First, they rely on large-scale datasets derived from 3D reconstructions [20], [21] of in-the-wild monocular videos, which significantly compromises quality in two ways: the source videos themselves are often of poor quality (e.g., low resolution, challenging lighting conditions, occlusions), and the parametric morphable models [22] used for reconstruction have limited expressivity. This combination results in imprecise facial motion and poor lip sync. While high-end 4D capture systems would address these issues, their cost makes large-scale collection impractical. Second, they lack the fine‑grained control mechanisms necessary for applications such as dubbing (used interchangeably with dialogue localization throughout this work), where lip-synchronization must be adapted to new audio while preserving the original eyebrow raises, eye closures, and general expressions [23]. MeshTalk [24] partially addresses dubbing by preserving upper-face motion while generating new lip motion from audio, but it lacks a global style mechanism for ensuring coherent integration of upper- and lower-face motion. In summary, existing methods face a fundamental trade-off between animation quality and controllability, failing to meet production requirements where artists need both high-fidelity and flexible control.

In this paper, we present KM-Speaker (Keypoint Matching Speaker), a flow-based framework [25] for speech-driven facial animation providing both example-based global control and fine-grained temporal controllability (as illustrated in [fig:teaser]). Unlike prior example-based approaches, our model operates in a realistic data-constrained setting using a curated, high-quality expressive dataset, reflecting practical data acquisition and usage constraints, while producing accurate and synchronized facial motion. We condition generation on sparse facial keypoints from a reference animation, encoding them as global and temporal style signals, and design a disentanglement strategy to separate lower-face motion from audio and upper-face motion from temporal style. The explicit temporal signal, together with this disentanglement strategy, enables dubbing applications. We also propose a style context preservation mechanism based on modality dropout that enforces reliance on global style to maintain full-face coherence. This enables example-based generation, where the animation adheres to the target’s overall expressive intent.

The contributions of our paper are as follows:

• We introduce KM-Speaker, a keypoint-conditioned framework for speech-driven 3D facial animation. Our model achieves high-fidelity generation with controllable style in data-constrained settings using high-quality expressive data.

• Our framework enables both example-based global style control and fine-grained frame-level temporal control from reference performances, supporting production workflows such as dialogue localization.

• We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, combined with a global style context preservation mechanism that maintains coherent full-face expressiveness.

• Extensive evaluations and user studies demonstrate that our method outperforms state-of-the-art approaches in lip-sync accuracy, style adherence, and dialogue localization.

2 Related work↩︎

Example-based facial animation generation. Recent literature explores the benefits of example-based generation, aiming to capture the global delivery style of a short target animation sequence (e.g., speaker‑specific idiosyncrasies or characteristic speaking patterns) [16]–[19]. DiffPoseTalk [16] extracts style through a temporally agnostic full‑face encoder–decoder trained with contrastive learning. MIMIC [17] also derives global style from full‑face motion but uses an end‑to‑end approach with a speaker‑classification regularizer, effectively constraining the style space to speaker‑dependent clusters. DisenEmo [18] further restricts the latent style representation through emotion‑aware supervision. MSMD [19] infers style from implicit latent expression codes from SEREP [26] instead, using a VAE to structure the style manifold while relaxing constraints to preserve both inter‑speaker and inter‑emotion variability. Despite their strengths, current example-based methods [16]–[19] rely on large-scale, reconstruction‑based datasets, whose limited fidelity inherently constrains the realism of generated motion. In contrast, our method operates in a challenging yet high-quality data setting. Furthermore, we introduce a temporal style signal, enabling additional controls through dialogue localization, a capability not addressed in prior example-based work.

Dialogue localization. Dialogue localization adapts spoken content in films and video games across languages and cultures, primarily through dubbing. The goal is to preserve the original actor’s performance, such as eye blinks, eyebrow motion, and expressions, while seamlessly adapting lip motion to a new audio [23]. Often referred to as video dubbing in 2D, existing methods fall into either end‑to‑end pipelines [27], [28], which map source frames and audio directly to the output, or structural approaches [29] that rely on interpretable intermediate representations, such as facial keypoints. Although structural methods require extra preprocessing (e.g., detection and landmark extraction), they offer improved interpretability and higher‑fidelity expression synthesis [23].

Trained on a large-scale capture dataset of 250 individuals, Mesh-Talk [24] is the first speech‑driven 3D facial animation framework explicitly designed for dubbing. It conditions a decoder on audio features fused with full‑face temporal embeddings and employs a disentanglement mechanism to separate audio‑driven motion from the target upper‑face dynamics. However, MeshTalk does not enforce full facial expressive coherence, which is essential when the audio alone cannot convey the intended emotional delivery. To address this limitation, we introduce a global style signal that ensures coherent full‑face expression, complementing both audio and target temporal cues. Furthermore, inspired by the demonstrated benefits of keypoints in video dubbing and their effectiveness in 3D settings [30], [31], we introduce a lightweight and explicit style representation based on sparse, semantically meaningful 3D facial landmarks, well‑suited for training in our realistic data constrained setting.

3 Method↩︎

Our goal is to generate speech-driven facial animation that flexibly matches a reference style. Given input audio and keypoints from a reference animation, we extract two style signals: a time-invariant global style from full-face keypoints captures general style, and a frame-wise temporal style from upper-face keypoints that enables precise upper-face matching. This allows two generation modes: dialogue localization, where all signals condition the model to re-synchronize lip motion while preserving upper-face dynamics, and example-based generation, where only audio and global style condition the model to synthesize full motion matching the reference’s overall style. We first provide a general description of our model, including its components and formulation within the flow-matching paradigm (3.1). We follow with our audio–keypoint disentanglement strategy (3.2) and finally introduce our global style context preservation approach (3.3).

3.1 Model architecture↩︎

The overall architecture of KM-Speaker is illustrated in 1. Our flow-based transformer model generates facial motion sequences for fixed window of length \(N\) frames conditioned on three features signals: audio context \(\mathcal{A}\), global style \(\mathcal{S}_\text{glob}\), and temporal style \(\mathcal{S}_\text{temp}\). We describe how each feature vector is obtained below.

Audio encoder. Input speech is encoded using a pretrained audio encoder. We average its hidden-layer features, project them to our model’s latent dimension \(d\), and resample to match the animation frame rate. The processed audio features are denoted \(\mathcal{A} \in \mathbb{R}^{N \times d}\).

Global style encoder. The global style encoder \(E_\text{glob}\) takes a sequence of full-face 3D keypoints \(\mathcal{K} \in \mathbb{R}^{N \times 3K}\), where \(K\) is the number of keypoints per frame. The encoder first encodes the keypoint signal \(\mathcal{K}\) to a sequence of framewise feature embeddings using convolutional layers followed by a transformer encoder, and then applies a temporal average pooling to form a compact global style representation \(\mathbf{z}_\text{g}\in \mathbb{R}^{d}\) which captures the reference animation’s overall style. This vector is then replicated across all frames to form the global style representation \(\mathcal{S}_{glob} = [\mathbf{z}_g]_{n=1}^N \in \mathbb{R}^{N \times d}\). We implement \({E}_\text{glob}\) as a VAE-based encoder following [19], which predicts a mean and variance regularized via KL divergence.

Temporal style encoder. The temporal style encoder \(E_{\text{temp}}\) encodes frame-wise upper-face dynamics from an upper-face keypoint sequence \(\mathcal{K}_\text{u} \in \mathbb{R}^{N \times 3K_\text{u}}\), where \(K_\text{u}\) is the number of upper-face keypoints. The encoder produces a temporal style sequence \(\mathcal{S}_\text{temp} \in \mathbb{R}^{N \times d}\). This signal enables precise frame-level matching of upper-face motion, essential for dialogue localization where lip motion is re-animated while preserving eyebrow raises, gaze shifts, and other upper-face expressions. We implement \(E_\text{temp}\) as a two-layer MLP that projects each keypoint frame to the model latent dimension \(d\).

Flow transformer model. We represent the target facial animation using PCA-compressed vertex offsets. Specifically, we project the per-frame mesh deformations into a low-dimensional latent space of dimension \(l\), resulting in a target latent sequence \(\mathbf{x} \in \mathbb{R}^{N \times l}\) of latent codes. We employ a transformer-based flow model \(\mathbf{\Psi}\) parameterizes a continuous transformation from a standard normal prior \(p_0\) to plausible facial motion trajectories, inspired by conditional flow matching [32]. This transformation is governed by the ODE \(\frac{d\mathbf{x}_t}{dt} = u_t(\mathbf{x}_t)\), where \(\mathbf{x}_t \in \mathbb{R}^{N \times l}\) denotes the latent sequence at flow time \(t \in [0,1]\). We construct trajectories using conditional optimal transport paths [33], with linear interpolation \(\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\mathbf{x}_1\), where \(\mathbf{x}_0 \sim \mathcal{N}(0, \mathbf{I})\). Rather than directly regressing velocities, we adopt an \(\mathbf{x}_1\)-prediction parameterization: \(\mathbf{\Psi}(\mathbf{x}_t, t, \mathbf{c})\) predicts the clean target \(\mathbf{x}_1\) from \(\mathbf{x}_t\), conditioned on \(\mathbf{c} = (\mathcal{A}, \mathbf{S}_\text{glob}, \mathbf{S}_\text{temp})\). This enables supervision through vertex-based losses without ODE integration during training. At inference, the predicted \(\hat{\mathbf{x}}_1\) is converted to a velocity estimate \(\hat{u}_t = (\hat{\mathbf{x}}_1 - \mathbf{x}_t)/(1-t)\), and the ODE is solved (midpoint, 50 steps) to produce latent coefficients \(\hat{\mathbf{x}} \in \mathbb{R}^{N \times l}\). Latents are decoded to vertex space via the PCA basis \(\mathcal{R}_{\text{PCA}}: \mathbb{R}^{N \times l} \rightarrow \mathbb{R}^{N \times 3V}\), where \(V\) is the number of mesh vertices, yielding the final animation \(\hat{\mathcal{X}} = \mathcal{R}_{\text{PCA}}(\hat{\mathbf{x}})\).

3.2 Audio-keypoint disentanglement↩︎

We propose a disentanglement strategy to achieve three objectives: (1) audio must control lip synchronization and mouth shapes for natural delivery, (2) temporal style must precisely drive upper-face motion frame-by-frame according to the target animation, enabling dialogue localization, and (3) global style must govern overall delivery intent and enforce coherent full-face motion.

Inspired by MeshTalk [24], we define per-vertex weight masks \(\mathcal{M}_{\text{upper}}, \mathcal{M}_{\text{lower}}\) with weights in \([0,1]\) that partition the face into upper and lower regions. We extend their approach by incorporating style conditioning to ensure facial coherence (2).

During training, for each sample \(i\) in a batch of size \(B\), we define a conditioning tuple \(\mathbf{c}^i = (\mathcal{A}^i, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^i)\) corresponding to target animation \(\mathcal{X}^i\). To enforce disentanglement, we randomly select another sample \(j \neq i\) from the batch and create two mismatched conditioning variants: 1) Audio conditioning: \(\mathbf{c}^i_{\text{audio}} = (\mathcal{A}^i, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^j)\): keeps audio from \(i\), swaps temporal style from \(j\). 2) Temporal conditioning: \(\mathbf{c}^i_{\text{temp}} = (\mathcal{A}^j, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^i)\): swaps audio from \(j\), keeps temporal style from \(i\). Crucially, the global style \(\mathcal{S}_{\text{glob}}^i\) always matches the target animation \(\mathcal{X}^i\), ensuring coherent integration of upper- and lower-face motion even when audio and temporal style intent is different (objective (3)).

We generate predictions for both conditioning variants: \(\hat{\mathcal{X}}_\text{audio}^i=\mathcal{R}_\text{PCA}(\Psi(\mathbf{x}_t^i, t, \mathbf{c}^i_\text{audio}))\) enforces to match \(\mathcal{X}^i\) in the lower face (audio drives lips, objective (1)) and \(\hat{\mathcal{X}}_\text{temp}^i=\mathcal{R}_\text{PCA}(\Psi(\mathbf{x}_t^i, t, \mathbf{c}^i_\text{temp}))\), which enforces to match \(\mathcal{X}^i\) in the upper face (temporal style drives upper face, objective (2)).

Training losses. We train the model using a disentangled reconstruction loss:

\[\begin{align} \mathcal{L}_{\text{mse}} = &\lVert\mathcal{M}_{\text{lower}} \odot (\mathcal{X}^i - \hat{\mathcal{X}}_{\text{audio}}^i)\rVert^2_F \\ &+ \lVert \mathcal{M}_{\text{upper}} \odot (\mathcal{X}^i - \hat{\mathcal{X}}_{\text{temp}}^i) \rVert^2_F \end{align}\]

where \(\odot\) denotes the Hadamard product and \(\lVert \cdot \rVert_F\) is the Frobenius norm.

To regularize temporal smoothness, we add a velocity loss [34] adapted for our disentanglement strategy:

\[\begin{align} \mathcal{L}_{\text{vel}} = & \mathcal{M}_{\text{lower}} \odot (\Delta\mathcal{X}^i - \Delta\hat{\mathcal{X}}_{\text{audio}}^i) \rVert^2_F \\ & + \lVert \mathcal{M}_{\text{upper}} \odot (\Delta\mathcal{X}^i - \Delta\hat{\mathcal{X}}_{\text{temp}}^i) \rVert^2_F \end{align}\]

where \(\Delta\mathcal{X} = \mathcal{X}_{1:N} - \mathcal{X}_{0:N-1}\).

Finally, we regularize the global style encoder’s latent space with KL divergence: \(\mathcal{L}_{\text{KL}} = \frac{1}{2}\sum_{i=1}^{d}\left(\mu_i^2 + \sigma_i^2 - \log(\sigma_i^2) - 1\right).\)

The total training loss is: \[\mathcal{L} = \mathcal{L}_{\text{mse}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}} + \lambda_{\text{KL}} \mathcal{L}_{\text{KL}},\] where \(\lambda_{\text{vel}}\) and \(\lambda_{\text{KL}}\) are hyperparameters balancing each loss term.

3.3 Global style preservation↩︎

The disentanglement strategy described above creates an asymmetry in conditioning strength: the temporal style signal \(\mathcal{S}_{\text{temp}}\) provides rich, frame-wise upper-face motion information that can dominate the generation, while the global style \(\mathcal{S}_{\text{glob}}\) encodes only high-level information. This imbalance poses a problem for example-based generation. At inference, when temporal style is absent (i.e., when we want to match the reference’s style), the model may fail to leverage \(\mathcal{S}_{\text{glob}}\) effectively for upper-face generation as it has learned to rely primarily on the stronger temporal signal during training.

To address this, we randomly zero out the temporal style signal while always retaining the global style. This forces the model to learn to capture the overall style from \(\mathcal{S}_{\text{glob}}\) alone when temporal information is unavailable, ensuring that global style remains influential even in the presence of strong temporal conditioning. Combined with our disentanglement strategy, this dropout naturally creates training samples with conditioning tuples \((\mathcal{A}^j, \mathcal{S}_{\text{glob}}^i, \varnothing)\) paired with target \(\mathcal{X}^i\). This configuration encourages the model to maintain the target’s global style across arbitrary audio inputs, including those with conflicting emotions.

4 Experimental protocol↩︎

Our experimental protocol is designed to assess lip‑motion accuracy, global style transfer fidelity, and the model’s dubbing capabilities.
Dataset. Our dataset consists of 2.6 hours of 4D captured data in consistent topology, evenly distributed across 12 professional actors and recorded at 60 FPS. Each actor performs a personalized set of sentences spanning 8 emotional states and 2 intensities (see capture protocol details in Appendix 9). We do not leverage label information during training to preserve diversity within each emotional state (1). We use data from 8 actors for training, 1 for validation, and 3 for testing.

Baseline models. For global stylization evaluation, we compare our method to the state‑of‑the‑art example‑based models MIMIC [17] (deterministic) and MSMD [19] (diffusion‑based). Both methods are originally learned on large-scale datasets, encode style from full-face, and accept an audio and a target window animation for style extraction. We compare against their original released models, as well as their retrained and finetuned versions on our small high-quality dataset (respectively noted _retrain or _finetune) for fair comparison. These variants assess the impact of data sources on style control and lip synchronization. For dialogue localization evaluation, we compare against the original MeshTalk [24] model, which to our knowledge is the only prior work explicitly addressing this task. We also evaluate the impact of the style input through training with a dense set of 1660 keypoints derived from [30] (denoted Ours_1660_keypoints) and a sparse set of 68 keypoints commonly used for facial tasks [35] (noted Ours_68_keypoints) visually presented 5. In all experiments, bold entries indicate statistically significant improvements (Wilcoxon signed rank test, \(\rho<0.01\)).

Implementation details. Our audio encoder relies on w2v-BERT 2.0 [36]. Our global style encoder is formed of 1D convolutions followed by transformer layers, following MIMIC and MSMD style encoder architectures [17], [19]. The temporal style encoder is an MLP, while our flow model is a transformer-based model conditioned using style-adaptive layer normalization [37]. We trained our full model using Adam optimizer [38] for 2.3 days on a single RTX 8000 GPU. More details on the training, architecture, and mask construction are provided in Appendix 10.1 (training), 10.2 (architecture), and 10.3 (masks).

We preserved the original MIMIC and MSMD protocols (25 FPS training) when training and finetuning those models on our dataset. MSMD and MIMIC use 100-frame style context windows at 25 FPS, whereas our method uses 200-frame context windows at 60 FPS. MIMIC_finetune is trained in the FLAME topology [22] and the output animation is converted back to our topology for evaluation, whereas MIMIC_retrain is directly trained in our own topology. For the MeshTalk model, we convert the input neutral mesh and target style animations to their topology prior to generation, and revert the topology to our own for evaluation. Topology conversion details are provided in Appendix 10.4. For fair comparison, example-based experiments are conducted at 25 FPS, and dialogue localization experiments at 30 FPS.

Unless stated otherwise, rendered animations are augmented with textures, eyes, and teeth to improve perceptual realism. Details on assets are provided in Appendix 10.5. We provide additional geometry-only visualizations in Appendix 11.

5 Results↩︎

5.1 Matching audio-style context↩︎

We first quantitatively evaluate example-based models under matching audio and style contexts (e.g., same emotion), measuring lip-sync and motion accuracy. Using all sequences from our test set, we condition on the previous window as global style and the current window as audio, with no overlap to prevent leakage [19]. Note that no temporal style is used for example-based generations.

We evaluate generation accuracy using standard metrics: vertex MSE (global motion), LVE [24] (lip accuracy), MOD [16] (mouth aperture), and FDD [5] (upper-face dynamics). Results for all baselines and related ablations are reported in 1.

Baseline results for the MIMIC and MSMD variants highlight the benefits of finetuning and training on higher‑quality data, yielding consistent improvements across both lip‑related metrics (LVE, MOD) and global motion (MSE) compared to the author‑released versions. Upper‑face dynamics (FDD) also improve for MSMD, whereas they remain comparatively stable for MIMIC, which we attribute to MIMIC’s deterministic nature that constrains stochastic events such as blinks and brow motion.

Overall, our model achieves significant improvements over baseline methods thanks to the efficiency of our flow‑based formulation and the use of a sparse yet explicit global‑style keypoint signal. This is supported by ablation results (1) showing that substituting our keypoint representation with implicit embeddings or denser keypoint configurations reduces general performance. For instance, using implicit representations derived from the MeshTalk encoder as input to our global style encoder yields the lowest scores across all ablation metrics. Then, reducing the keypoint density from 1660 to 68 and our 55 keypoints (as detailed 4) while preserving semantically meaningful facial locations or refining them by introducing nasolabial folds keypoints produces a more explicit and informative style cue, consistently benefiting the model. This likely facilitates encoding, yielding a compact representation better suited to our challenging data setting. Notably, MSMD derives its style features from implicit expression codes computed via SEREP [26], while MIMIC operates on full‑face inputs, both of which are conceptually aligned with the implicit and dense alternatives tested in our ablations.

5.2 Cross audio-style context↩︎

We qualitatively evaluate our model against baselines trained on the our high‑quality dataset (MSMD_retrain and MIMIC_retrain) as well as an ablated version of our method without the context‑preservation mechanism (3.3). The goal is to assess how well each model adheres to a target animation’s global style when the audio does not match the target intent. Such a scenario is common in production, where style nuances (e.g., suspicion, doubt, or irony conveyed through eyebrow) are absent from the audio. 3 illustrates this by using a desired angry target style paired with a neutral audio line. We provide additional qualitative examples in supplementary video (3:33 to 4:31).

By balancing the importance of conditioning signals and training with conflicting audio–style pairs (3.3), our model produces upper‑face motion that more faithfully reflects the target style. This improvement is especially visible when comparing our model’s expressive upper-face intent to its less expressive ablated variant. Similarly, MIMIC_retrain and MSMD_retrain remain mostly neutral, which we believe stems from the challenging size of our dataset that hinders their ability to capture style.

5.3 User study↩︎

We conducted two user studies to evaluate (1) lip synchronization quality and (2) global style adherence across three models: Ours, MIMIC_retrain, and MSMD_retrain. We include textures, as well as eye and teeth assets for more natural assessment and ensure fair comparison by comparing models trained on our dataset (best-performing variants in our quantitative evaluation 5.1).

Each study comprises five sets of ten randomly selected, non-overlapping videos covering all test actors. Samples from the lip-sync and style-adherence studies are drawn from pools of 83 lip-sync and 200 style examples, respectively, with randomized model ordering. We adopt a MUSHRA-like protocol [39], where participants rate samples on a 0–100 scale, and a negative anchor is hidden in each set. Lip-sync evaluation includes audio, whereas style evaluation is muted, following [19], to improve focus on style consistency. Studies were conducted with researchers from the video‑game industry and professionals in animation, yielding 25 and 23 responses for the lip-synchronization and style-adherence tasks, respectively. We report the mean rating (\(95\%\) CI) in 4. Study guidelines, structure, visuals, anchor selection, and additional analyses (i.e., mean rankings and agreement measures) are provided in Appendix 12.

Results show our method as preferred for lip-synchronization, with MIMIC_retrain lagging significantly behind, and MSMD_retrain scoring closely behind. This ranking differs from the quantitative close-context scenario, where lip-related metrics (LVE) show MIMIC_retrain performing better than MSMD_retrain. We attribute this inversion to the limitations of LVE, which measures strict geometric distance to a single ground-truth reference. Because speech-to-motion is a one-to-many mapping, generative models can produce plausible, high-quality lip articulations that diverge from the specific actor’s performance, yet are penalized by this metric. Visually, MIMIC_retrain presents a less organic, expressive, and more damped motion than MSMD, further supported by qualitative results in 6. While this conservative behavior benefits MIMIC_retrain regarding LVE (by remaining closer to the mean), it degrades perceptual quality. In comparison, our model achieves better lip-synchronization than all competitors in both the user study and quantitative measures, confirming the benefits of the flow-based approach in delivering both geometrically accurate and perceptually convincing results.

Across the style-adherence evaluation, MIMIC_retrain and MSMD_retrain underperform markedly, consistent with our qualitative observations in 5.2 that these models struggle to capture style when trained on small datasets (see additional examples in supplementary video 3:33). Upon inspection, we observe that MIMIC_retrain and MSMD_retrain frequently produce either near-neutral or aligned with the speech emotion deliveries, both of which deviate from the target style and are therefore similarly penalized.

5.4 Dialogue localization↩︎

We evaluate dialogue localization: matching target expressions while resynchronizing lip motion to new audio. For quantitative evaluation, we randomly sample 100 pairs of audio and target facial animations of the same length. We report the upper-face mean squared error (U-MSE) measured with respect to the target ground-truth animation and the Lip Vertex Error (LVE) with respect to the corresponding audio ground-truth animation in 2. We compare MeshTalk [24] against our model and three ablations: using the MeshTalk full-face encoder instead of keypoints, using MeshTalk disentanglement masks (that allow the target animation to affect jaw motion 5), and our model trained with no disentanglement strategy.

Our model achieves the lowest LVE while best preserving target upper-face motion, indicating superior dialogue localization performance. The temporal keypoint signal provides strong supervision: without disentanglement (Ours_no_disentangle), it leaks into lip motion, degrading lip-sync quality. MeshTalk’s disentanglement partially mitigates this effect but still allows upper-face dynamics to influence lip articulation via jaw motion, which is detrimental in expressive scenarios. MeshTalk also exhibits lower fidelity to ground-truth motion, likely due to less expressive training data and the absence of mechanisms enforcing coherent full-face expressiveness. Replacing explicit keypoint conditioning with a learned full-face embedding (Ours_meshtalk_encoder) also degrades performance, highlighting the benefits of our explicit signal for robust style encoding in constrained data settings.

We complement these findings with qualitative results in 7, showing raw geometry outputs without eyes, teeth, or textures. This setup follows the MeshTalk setup and provides a complementary perspective to our previous renderings by isolating predicted geometry motion from rendering assets. The issues described above are clearly visible in dynamic sequences (see supplementary video 5:45–7:07) and, to a lesser extent, in 7. Overall, these results highlight the importance of explicit keypoint conditioning and principled disentanglement for coherent localization.

We provide additional qualitative analysis comparing against a baseline that deterministically blends mouth motion, with or without global style conditioning, with a fixed upper-face animation (see Appendix 13). This setup is inspired by common practices in artist-driven pipelines (e.g., viseme-based systems such as JALI [2]), in which mouth motion is often edited or blended independently of upper-face animation [2]. The comparison highlights inherent limitations of such strategies, which often require time-consuming manual editing and struggle to generalize across diverse expressions and geometries. These observations further support our unified approach, which enables both example-based generation and dialogue localization within a single framework.

6 Limitations and future works↩︎

While effective in our high-fidelity setting, training on 8 actors with production-quality audio limits the diversity of the learned facial prior and the range of styles and acoustic conditions, potentially introducing bias toward specific morphologies, expressions, or speech characteristics. This limitation is observed in our generalization experiment (Appendix 14 and supplementary video 7:20), where certain facial morphologies lead to imperfect mouth closure or inner-mouth mesh intersections. Addressing these limitations calls for exploring few-shot adaptation to better capture actor-specific attributes. Despite this, our model shows reasonable robustness to in-the-wild styles and audio. Beyond these limitations, our method raises ethical considerations related to example-based generation and dubbing. Such systems should only be used with appropriate consent from the individuals whose likeness or performance is reproduced.

7 Conclusion↩︎

In this paper, we present KM-Speaker, a flow-based generative framework with the explicit goal of making data-driven models more amenable to artistic direction. To this end, we introduce the first data-driven method enabling both example-based generation and dialogue localization, combining captured realism with flexible control. Leveraging a keypoint-conditioned architecture and a disentanglement strategy, KM-Speaker achieves accurate lip synchronization while capturing global and temporal cues from reference performances. We view this approach as a step toward production-ready solutions that give artists full control over performance while leveraging the realism learned from captured data at scale.

8 Acknowledgements↩︎

This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and MITACS through the Alliance-Mitacs program (Grant No. ALLRP 589317 - 23).

We would also like to thank the authors of MeshTalk [24], MIMIC [17], and MSMD [19] for releasing their code and advancing research in the field.

We also thank the actors who consented to the use of their likenesses for the qualitative results presented in this paper.

Appendices↩︎

9 Data collection protocol↩︎

We built our dataset by recording synchronized speech audio and high-fidelity facial performances from 12 professional actors. To ensure broad demographic coverage, we conducted a targeted casting aimed at maximizing diversity in age, gender, and ethnicity within the limited number of actors we captured. Our final cast includes six male and six female actors, with four identifying as White, three as Asian, three as Black, one as Middle Eastern, and one as Latin, spanning an age range from 18 to 64.

Data was captured in two configurations: (1) a seated light-stage setup consisting of six synchronized cameras, and (2) a head-mounted rig containing four cameras aimed at the actor’s face.

Actors performed a curated set of actor-specific sentences across eight target emotional states: Neutral, Angry, Fear, Happy, Disgust, Sad, Surprise, and Authority. Except for Neutral, each emotion was recorded at two intensity levels: Moderate and Heightened. The Moderate condition corresponds to a controlled, conversational delivery appropriate for everyday dialogue, whereas the Heightened condition reflects amplified facial expressiveness suited for dramatic or high-impact narrative moments commonly found in digital media. All the performances were conducted in English.

Because actor expressiveness was critical to the overall quality of the dataset, we designed emotion-specific sentences to help actors deliver consistent, contextually coherent performances. Example sentences associated with each emotion are provided in Table 3.

Following capture, we followed the standard cinematic‑quality procedure of the Faceform 4D‑capture system². More precisely, all footage was processed through a standard photogrammetry pipeline, in which each video frame was 3D-reconstructed using RealityCapture, a commercial photogrammetry software. Then, raw mesh sequences were aligned to a common mesh topology of 13,473 vertices using Faceform Wrap 4D, with additional manual refinements by an experienced facial motion-capture supervisor to achieve accurate and stable results. Processing a single minute of footage under this full pipeline takes approximately 8 hours.

10 Additional implementation details↩︎

10.1 Training details↩︎

Importance weights. The following importance weights are used in eq. 4 of the main manuscript: \(\lambda_\text{mse}=100\), \(\lambda_\text{vel}=100\), and \(\lambda_\text{KL} = 1e-6\).

Optimization hyperparameters. We trained with a batch size of 32, a learning rate of \(1e-4\), and a weight decay of \(1e-06\).

Temporal-style dropout. We randomly drop the temporal style conditioning signal with a probability of 0.1 during training, analogous to classifier-free guidance strategies. This encourages the model to rely on global style for full-face coherence when temporal information is absent. We experimented with higher dropout rates (0.2–0.3) and observed no significant qualitative differences, leading us to adopt 0.1 as a stable default.

10.2 Model and Representations↩︎

Model architecture. We detail our model architecture in 4, including audio encoder, global style encoder, temporal style encoder, and our flow transformer model.

PCA construction and dimensionality. We construct the PCA basis from the full training set of eight identities using vertex displacements computed with respect to each identity’s neutral face mesh in rest pose. We retain \(l=128\) principal components, which preserve approximately 99.5% of the total variance. This dimensionality provides a good balance between compression and reconstruction fidelity, while enabling efficient sequence modeling. We did not further reduce \(l\), as maintaining high-fidelity facial motion is critical for our target applications.

Keypoint signal. We use a fixed set of \(K=55\) full-face keypoints and \(K_\text{u}=47\) upper-face keypoints. The upper-face set is a subset of the full-face set, as illustrated in Fig.6 of the main paper.

Keypoint extraction and consistency. Since all meshes share a fixed topology (see supplementary section A), keypoints are defined via direct vertex correspondences rather than learned detectors. This ensures consistent spatial alignment across identities and sequences.

Keypoint preprocessing. Keypoint signals are computed as vertex displacements with respect to each identity’s neutral face mesh in rest pose. These displacements are standardized per keypoint using training-set statistics before being fed to the model. This consistent parameterization ensures robustness across identities while improving conditioning stability and generalization.

10.3 Facial region masks↩︎

The facial masks \(\mathcal{M}_{\text{lower}}\) and \(\mathcal{M}_{\text{upper}}\) are vectors of length \(13{,}473\), with all elements in the range \([0,1]\). To construct those, we first define two disjoint vertex sets on the template mesh: a lower-face region \(\mathbf{R}_{\text{lower}}\) and an upper-face region \(\mathbf{R}_{\text{upper}}\), leaving a middle band of vertices unassigned to either region to enable a smooth integration of lower-and upper-face.

To construct \(\mathcal{M}_{\text{lower}}\), which emphasizes audio-driven deformations on the lower part of the face, we set:

\[\mathcal{M}_{\text{lower}}(v) = \begin{cases} 1, & \text{if } v \in \mathbf{R}_{\text{lower}},\\[3pt] 0, & \text{if } v \in \mathbf{R}_{\text{upper}}. \end{cases}\]

For vertices \(v\) in the middle band, we define a smooth transition using a distance-based inverse decay from the lower region. We define a point-to-region distance operator \(d\) based on Euclidean distance: \[d(v,\mathbf{R}_{\text{lower}}) = \min_{u \in \mathbf{R}_{\text{lower}}} \| v - u \|_2,\] and define a lower-face mask as: \[\mathcal{M}_{\text{lower}}(v) = \frac{1}{1 + \beta \, d(v,\mathbf{R}_{\text{lower}})}, \quad \text{for } v \notin \mathbf{R}_{\text{lower}} \cup \mathbf{R}_{\text{upper}},\] where \(\beta > 0\) controls the rate of inverse decay (we use \(\beta = 1.0\)).

The upper-face mask is then defined as the complement: \[\mathcal{M}_{\text{upper}} = 1 - \mathcal{M}_{\text{lower}},\] which, analogously, enforces temporal-style keypoints to drive the upper face.

10.4 Topology conversion↩︎

To convert mesh data from a source topology to a target topology, we compute a sparse conversion matrix that maps each vertex of the target mesh to a barycentric combination of the closest triangle in the source mesh. Meshes must be spatially aligned and scaled before computing the matrix. The process is as follows: (1) For each vertex in the target topology, we find the closest triangle in the source topology using a point-to-triangle distance metric. (2) We compute the barycentric coordinates of the target vertex with respect to this closest triangle. (3) These barycentric weights are used to populate a sparse matrix, where each row corresponds to a target vertex and each column to a source vertex.

The resulting matrix enables efficient conversion: multiplying the source vertex data by it yields the corresponding data in the target topology. This method ensures that geometric information is transferred as accurately as possible between different mesh topologies, while the conversion is performed via a single matrix multiplication.

10.5 Auxiliary Components for Rendering↩︎

Textures. We apply static per-actor textures to the animated meshes to enhance visual realism. Modeling dynamic or view-dependent textures could further improve fidelity, as highlighted by Li et al. [40], but such aspects are not the focus of this work. We leave these extensions to future research.

Eye and jaw processing. For all retrained models, we predict eye motion and jaw articulation using lightweight regressors trained directly on top of the model’s latent motion features. Given the predicted motion feature sequence \(\hat{\mathbf{x}} \in \mathbb{R}^{N \times l}\), where \(N\) denotes the number of frames and \(l\) the feature dimension, we train a small MLP to infer (i) eye gaze and (ii) jaw rotation.

Eye motion is represented by four parameters corresponding to yaw and pitch for the left and right eyes, while jaw articulation is represented as a \(3 \times 3\) rotation matrix parameterized as a 9-dimensional vector.

For each target sequence \(y_{\text{eyes\_jaw}} \in \mathbb{R}^{N \times 13}\), the MLP predicts \(\hat{y}_\text{eyes\_jaw} = g(\hat{\mathbf{x}})\) and is optimized using \({L_{\text{eyes\_jaw}}}\) defined with a combination of mean-squared error (MSE) and velocity consistency losses: \[\begin{align} \mathcal{L}_{\text{eyes\_jaw}} = &\sum_{n=0}^{N} \left\| \hat{y}_{\text{eyes\_jaw}, n} - y_{\text{eyes\_jaw}, n} \right\|_2^2 \\ &+ \sum_{n=1}^{N} \left\| \left(\hat{y}_{\text{eyes\_jaw}, n} - \hat{y}_{\text{eyes\_jaw}, n-1}\right) \right. \\ &\qquad\left. - \left(y_{\text{eyes\_jaw}, n} - y_{\text{eyes\_jaw}, n-1}\right) \right\|_2^2. \end{align}\]

where both terms are weighted equally.

This post-hoc regression strategy enables satisfying estimation of eye and jaw motion while keeping the core facial-motion model unchanged. We emphasize that these components are included primarily to enhance visual realism rather than as a core contribution of our method.

While such components could be integrated directly into the mesh topology for end-to-end prediction, most production pipelines treat eyes and teeth as separate assets and animate them independently to meet scene-specific requirements. Our design, therefore, aligns with common practices while maintaining architectural simplicity.

11 Raw geometry results↩︎

This section presents additional qualitative results showing facial geometry only (without textures, eyes, or teeth). These visualizations correspond directly to the model’s predictions, while rendering assets are used in the main paper solely to improve perceptual realism.

Fig. 8 presents additional geometry-only results for the example-based style generation task, where neutral audio is paired with an angry target style, similar to fig. 4 of the main paper. Fig. 9 presents results for the lip-synchronization experiment, similar to fig. 7 in the main paper. The same observations apply.

12 Perceptual study↩︎

12.1 Perceptual study construction details↩︎

We present screenshots of the user‑study guidelines for both the lip‑synchronization and the style‑adherence in 10 (a). For the lip‑synchronization study, raters were shown videos containing the four model outputs, displayed in random order, and asked to rate each generation individually on a scale from 0 to 100. For the style‑adherence study, the videos were muted to ensure raters focused solely on the delivery style, while the target-style animation was displayed in a separate upper row, playing simultaneously with the generated outputs. See 10 for screenshots of both studies videos and rating display. Participants were allowed to replay the videos as many times as needed and encouraged to submit their ratings only when confident in their assessments. They were also asked to ignore eyes and teeth motion at all time, given those assets are added to improve realism and not part of the evaluated methods.

Negative anchor For the lip‑synchronization study, the negative anchor is randomly sampled from other available sequences of the same actor.

In the style‑adherence study, constructing a meaningful negative anchor is more challenging, as some emotion categories are semantically and behaviorally close (e.g., surprise vs. happiness, or authority vs. anger). To obtain more robust negatives, we represent each animation clip using summary statistics of its facial landmark dynamics, computed from the KM‑Speaker set of 55 keypoints. For each frame, we concatenate the flattened 3D landmark positions with their first‑order temporal velocities, forming a per‑frame descriptor. For each clip, we then compute the temporal mean and covariance of these descriptors.

To select a negative emotion class for a given target emotion, we compute an inter‑emotion distance based on the centroid statistics of each emotion class. Given a target animation, the negative sample is drawn from the emotion class that maximizes this distance. All centroid statistics and distance computations are performed within the test actor’s data distribution only, as different actors exhibit distinct styles and interpretations when conveying emotions.

12.2 Perceptual study additional analysis↩︎

To complete the analysis of the pursued user studies, we additionally compute the mean ranking for each method (5), including the negative anchor, and ensured statistical significance (\(\rho < 0.01\)) using a Wilcoxon test between our methods’ performances and each baseline and negative.

Across both user studies, our method consistently achieves the lowest mean ranking, demonstrating its consistency across diverse generations and styles. Importantly, we observe that the mean ranking of the negative in both studies consistently comes behind other methods, while the ordering of MSMD_retrain and MIMIC_retrain follows the main paper results.

We further assess inter-rater agreement to evaluate the reliability of the collected perceptual judgments. For lip synchronization, agreement is strong at the pairwise level, with a mean Pearson correlation of \(0.88\), indicating a high degree of consistency in relative comparisons across raters. This level of pairwise agreement suggests that lip synchronization is generally perceived as a more objective task, leading to stable and reproducible evaluations. For style adherence, we likewise observe consistent inter-rater agreement, with a mean pairwise Pearson correlation of \(0.57\). This suggests that, while individual assessments of style may vary due to subjective interpretation, raters nonetheless tend to converge toward a consistent consensus when judgments are considered collectively.

13 Dialogue localization: Blending baseline and global style↩︎

In this section, we evaluate a blending-based baseline inspired by common practices in artist-driven pipelines, where mouth motion is edited or composed independently from upper-face animation. This setup serves as a strong deterministic reference for dialogue localization.

Specifically, we construct a blended animation by combining target upper-face motion with lip-sync animation generated by our model. The blended animation \(\mathcal{X}_{\text{blend}}\) is defined as: \[\mathcal{X}_{\text{blend}} = \mathcal{M}_{\text{upper}} \odot \mathcal{X}_{\text{target}} + \mathcal{M}_{\text{lower}} \odot \mathcal{X}_{\text{generated}},\] where \(\odot\) denotes element-wise multiplication, \(\mathcal{M}_{\text{upper}}\) and \(\mathcal{M}_{\text{lower}}\) are blending masks (Supplementary Section B), and \(\mathcal{X}_{\text{target}}\), \(\mathcal{X}_{\text{generated}}\) denote the target and generated animations, respectively.

We evaluate this baseline both without global style conditioning (Blend (no global)) and with it (Blend), to assess its impact on overall coherence.

Qualitative results are shown in 11. While blending preserves upper-face motion, it introduces geometric inconsistencies in transition regions, particularly near mask boundaries (see zoomed-in views), even when global style is applied. These discontinuities affect both surface geometry and normals, and would lead to shading artifacts in rendered animations.

More fundamentally, such compositing strategies remain sensitive to identity-specific geometry and expression variation. Although careful tuning of blending parameters can improve results in constrained cases, this process is time-consuming and does not generalize well across diverse conditions. Also, while global style conditioning improves coherence between upper- and lower-face motion (particularly visible for the left target style), it does not address the fundamental limitations of compositing-based strategies. In contrast, our model directly generates coherent full-face animations, avoiding boundary artifacts and generalizing across identities without manual intervention. In addition, our approach supports both dialogue localization and example-based generation within a unified framework.

14 Generalization↩︎

We conduct a qualitative generalization experiment in the example-based setting to evaluate our model beyond the controlled high-fidelity setting, considering variations in facial geometry, non production quality audio, and styles extracted from off-the-shelf face expression capture method [26] rather than 4D capture. To this end, we sample three audio clips from CelebV-HQ [41], three target videos serving as styles, and three neutral meshes with diverse facial traits from Triplegangers [42].

We present those qualitative results in 12, with corresponding dynamic results visible in our supplementary video (7:20 to 7:50). Despite these challenging conditions, our model demonstrates reasonable robustness: it captures global style from reconstructed expressions and produces convincing lip motion with accurate synchronization. For instance, expressive styles such as anger (first row) are clearly conveyed through both facial dynamics and motion intensity, whereas more neutral styles (last row) yield correspondingly restrained animations without introducing artifacts.

At the same time, this experiment reveals limitations in geometric generalization. For certain identities, we observe imperfect mouth closure (last row) and inner-mouth mesh intersections (middle row), particularly for challenging facial morphologies. These failure cases highlight the limits of the learned facial prior when applied to out-of-distribution geometries.

Despite this, the observed artifacts remain moderate. Inner-mouth intersections could likely be mitigated through appropriate regularization, whereas imperfect lip closure remains a more fundamental limitation. In both cases, incorporating identity-specific modeling, such as few-shot adaptation or explicit identity encoding/decoding (e.g., SEREP [26] or [43]), would likely improve robustness and reduce these effects.

References↩︎