1 Introduction↩︎

Talking head generation aims to synthesize realistic videos of a person speaking, typically from a still image, a reference video, an audio signal, text, or a combination of these inputs. The task sits at the intersection of computer vision, graphics, speech processing, multimodal learning, and human-computer interaction. A convincing talking head must preserve identity, synchronize lip movements with speech, model facial expression and head motion, remain temporally stable, and generate visually plausible details under changes in pose, emotion, language, and speaking style. These requirements make talking head generation a challenging problem, but also a practically important one, with applications in digital avatars, video conferencing, film and game production, virtual reality, assistive communication, education, and embodied conversational agents.

Early work in this area relied on rule-based systems and explicit mappings from speech or text to lip motion [1], [2]. These methods demonstrated the promise of speech-driven facial animation, but were limited in visual realism, expression diversity, and generalization to unconstrained identities and environments. The widespread adoption of deep learning changed the trajectory of the field. Convolutional networks, recurrent models, generative adversarial networks, attention mechanisms, transformers, and more recently diffusion and neural rendering models have enabled data-driven systems that can synthesize increasingly natural facial motion and high-quality portrait videos [3]–[13]. As large-scale face, speech, and video datasets became available, talking head generation moved from controlled lip animation towards more general portrait animation, including identity-preserving reenactment, audio-driven facial dynamics, emotion-aware synthesis, 3D-aware avatars, and real-time interactive generation.

The field has also evolved technically. Earlier neural methods often relied on landmarks, 3D morphable models, optical flow, or keypoint-based motion transfer to separate identity from motion. GAN-based approaches improved photorealism and image-level fidelity, while attention mechanisms helped models focus on semantically important regions such as the mouth, eyes, and facial contours [9], [14]–[16]. Transformer-based and codebook-based methods later improved long-range temporal modeling and controllability. More recently, diffusion-based approaches have become increasingly prominent because of their ability to model complex audio-visual distributions and generate expressive, high-quality portrait videos under weak conditioning [17]–[19]. In parallel, neural radiance fields, 3D-aware representations, and Gaussian splatting-based avatars have shifted part of the field towards explicit or implicit 3D representations, enabling improved view consistency, controllability, and real-time rendering [20], [21]. These developments have expanded the scope of talking head generation from facial animation alone to the broader problem of controllable, photorealistic, and interactive human avatar synthesis.

Despite this progress, the field remains difficult to assess. Talking head generation methods differ substantially in their inputs, assumptions, training data, target applications, and evaluation protocols. Some methods animate a single image using a driving video, some generate facial motion directly from audio, some edit an existing video, while others construct a controllable 3D avatar. These systems are often evaluated on different datasets, under different levels of supervision, and with different preprocessing choices. As a result, reported quantitative results are not always directly comparable. Moreover, commonly used metrics such as PSNR and SSIM primarily measure pixel-level similarity and often fail to capture the perceptual realism of talking heads, including lip synchronization, expression naturalness, identity preservation, temporal coherence, and the absence of distracting artifacts [22]. Metrics such as FID, LPIPS, identity similarity, landmark distance, and lip-sync confidence provide complementary signals, but no single metric reliably captures the overall quality of a generated talking head. Human evaluation remains important, but it is costly, subjective, and difficult to reproduce.

This evaluation gap is particularly important because talking head generation is no longer only a laboratory problem. Modern systems are increasingly close to real-world deployment in telepresence, virtual assistants, entertainment, online education, and social media content creation. Practical deployment requires more than high visual quality: models must be efficient, controllable, robust to diverse identities and languages, and safe to use. Real-time inference, GPU memory usage, latency, and stability over long videos are therefore central concerns. At the same time, the same technology can be misused for impersonation, misinformation, and non-consensual synthetic media. Any contemporary review of talking head generation must therefore consider not only model architectures and benchmark scores, but also provenance, watermarking, deepfake detection, consent, dataset bias, demographic fairness, and responsible deployment.

This survey provides a structured and critical review of talking head generation. We organize the literature into four broad families: image-driven, audio-driven, video-driven, and 3D/neural-rendering-based approaches. Image-driven methods animate a source portrait using visual cues such as landmarks, keypoints, pose, or expression from another image or video. Audio-driven methods synthesize lip motion, facial expressions, and head movements from speech signals. Video-driven methods focus on reenactment, retalking, and motion transfer from a reference sequence. 3D and neural-rendering methods, including NeRF-based and Gaussian splatting-based avatars, aim to improve view consistency, controllability, and photorealistic rendering. For each family, we discuss representative methods, technical assumptions, strengths, limitations, and typical failure cases.

Beyond cataloguing prior work, this survey places particular emphasis on evaluation and practical usability. We review commonly used datasets and metrics, discuss why existing evaluation protocols often disagree with perceptual quality, and compare publicly available models in terms of inference time, memory requirements, and human-rated visual quality. This comparison is intended to complement standard benchmark numbers by highlighting the trade-offs that matter in real applications, where a method must not only look realistic but also run efficiently and consistently. We also identify emerging trends, including diffusion-based portrait animation, long-duration generation, emotion and style control, real-time audio-driven avatars, 3D-aware synthesis, and responsible generation.

In summary, this survey makes the following contributions:

• We provide an updated taxonomy of talking head generation methods, covering image-driven, audio-driven, video-driven, and 3D/neural-rendering-based approaches.

• We review the evolution of the field from rule-based and landmark-driven systems to GANs, transformers, diffusion models, NeRF-based methods, and Gaussian avatar representations.

• We analyse datasets and evaluation metrics, highlighting the mismatch between commonly reported quantitative scores and perceived visual quality.

• We compare publicly available models using practical criteria such as inference time, GPU memory usage, and human-rated quality.

• We discuss open challenges around lip synchronization, identity preservation, emotional expressiveness, temporal consistency, demographic robustness, computational efficiency, provenance, and responsible use.

By synthesizing technical progress, evaluation practice, deployment constraints, and societal risks, this survey aims to provide researchers and practitioners with an up-to-date map of the talking head generation landscape and a clear view of the challenges that should shape future work.

2 What Comprises a Good Talking Head Generation?↩︎

A talking head generation system is successful only when it satisfies several constraints simultaneously. A generated video may be visually sharp but poorly synchronized with speech; it may achieve accurate lip motion but fail to preserve identity; or it may perform well on short benchmark clips while breaking under long-duration generation, large head poses, emotional speech, multilingual audio, or real-time deployment. Therefore, the quality of a talking head should not be reduced to a single visual metric. Instead, it should be understood as a multi-objective problem involving identity, synchronization, visual realism, temporal stability, controllability, efficiency, and responsible use.

Prior work [23] formalized several desirable properties for synthesized talking-head videos, including identity preservation, lip synchronization, visual quality, and spontaneous natural motion. These properties remain central. However, the field has moved substantially beyond the setting considered by early benchmarks. Recent methods now include efficient implicit-keypoint systems [24], real-time audio-driven models [25]–[27], diffusion and video-transformer-based portrait animation [28]–[30], one-shot and large-scale 3D avatar models [31], [32], and Gaussian-splatting-based talking head avatars [33]–[37]. As a result, a contemporary definition of a “good” talking head must also include long-range consistency, emotional expressiveness, controllability, view consistency, latency, and safety.

We summarize the main quality dimensions as follows:

• Identity preservation: The generated video should maintain the source subject’s identity across frames, expressions, poses, and lighting conditions. This is commonly evaluated using face-recognition embeddings, for example by computing cosine similarity between the source and generated identities.

• Lip synchronization: The mouth motion should match the spoken content at both phonetic and semantic levels. Good lip-sync requires not only plausible mouth opening and closing, but also correct viseme formation, stable teeth and tongue appearance, and robustness across speakers, accents, languages, and speaking rates.

• Visual fidelity: Individual frames should be realistic, sharp, and free from local artifacts around the mouth, teeth, eyes, hair, face boundary, and background. Metrics such as PSNR, SSIM, FID, LPIPS, and CPBD are often used, but they only partially capture perceived realism [22].

• Temporal consistency: The generated video should remain stable over time. Flickering, identity drift, unstable mouth interiors, inconsistent eye gaze, and frame-to-frame texture changes can make even high-quality individual frames appear unrealistic in video form.

• Natural motion and expressiveness: A convincing talking head should include natural head motion, blinking, gaze changes, facial expressions, and emotion-consistent dynamics. Recent works increasingly model not only lip motion but also full facial dynamics, head pose, emotional intensity, and upper-body or accessory motion [25], [26], [28], [37].

• Controllability: Practical systems should allow control over relevant attributes such as expression, emotion, gaze, pose, speaking style, viewpoint, and sometimes body motion or background. This is especially important for avatars, editing tools, and professional content creation [24], [30], [38].

• View and geometry consistency: 3D-aware and Gaussian-splatting-based methods introduce an additional requirement: the generated head should remain consistent under camera movement and novel views. This is important for virtual reality, telepresence, and interactive avatars [31], [33], [35], [36].

• Generalization and robustness: A strong model should generalize across identities, ages, ethnicities, genders, lighting conditions, head poses, occlusions, accents, emotions, and languages. Robustness is particularly important because many commonly used datasets contain biases in speaker demographics, recording conditions, and video quality.

• Efficiency and deployability: Real-world systems require acceptable inference time, GPU memory usage, latency, and model size. For video conferencing, virtual assistants, gaming, and telepresence, a model that is visually impressive but slow or expensive to run may be unsuitable for deployment [24]–[27], [34].

• Safety and responsible use: Talking head generation can be misused for impersonation, misinformation, and non-consensual synthetic media. Responsible deployment therefore requires attention to consent, provenance, watermarking, dataset governance, misuse prevention, and compatibility with deepfake detection systems.

These criteria are often in tension. Improving expressiveness may increase the risk of identity drift; increasing resolution may reduce real-time performance; optimizing for lip synchronization may produce unnatural head motion; and enforcing temporal smoothness may suppress spontaneous expression. The relative importance of each criterion therefore depends on the application. A video conferencing system may prioritize latency, identity preservation, and stability, while a film-production tool may prioritize high-resolution controllability and editability. An assistive communication system may require intelligible lip motion and emotional clarity, while a virtual reality avatar may require real-time performance and view-consistent 3D rendering.

2.1 Fundamentals in Talking Head Generation↩︎

The central goal of talking head generation is to synthesize a video of a target identity speaking or moving according to a driving signal. The target identity may be provided as a still image, a short video, a learned avatar, a 3D representation, or a Gaussian-splatting-based head model. The driving signal may be audio, text, another video, facial landmarks, keypoints, 3D morphable model parameters, expression codes, pose trajectories, diffusion latents, or combinations of these. In a general form, the task can be written as: \[V_g = f(I_s, c; \theta),\] where \(I_s\) denotes the source identity or appearance reference, \(c\) denotes the conditioning signal, \(V_g\) is the generated talking-head video, and \(f\) is a generative model with parameters \(\theta\). In video-driven methods, \(c\) may be a driving video \(V_d\); in audio-driven methods, it may be an audio signal \(A\); in text-driven or editing methods, it may include text or transcript information; and in 3D-aware methods, it may include camera, pose, geometry, or expression parameters.

Although architectures differ, many talking head generation systems can be understood through four broad stages: representation, motion or condition modeling, synthesis, and refinement.

2.1.0.1 Representation.

The model first extracts representations of identity and conditioning information: \[z_s = E_s(I_s; \theta_s), \quad z_c = E_c(c; \theta_c),\] where \(E_s\) encodes source identity or appearance, and \(E_c\) encodes the driving signal. Depending on the method, \(z_c\) may represent landmarks, keypoints, optical flow, audio embeddings, phoneme features, expression coefficients, head pose, 3D geometry, diffusion conditioning tokens, or neural-rendering features. Earlier approaches often relied on landmarks, 3DMM coefficients, or keypoints [39], [40], while recent approaches increasingly use richer latent representations learned by diffusion models, transformers, or 3D Gaussian avatar models [25], [27], [29], [33], [36].

2.1.0.2 Motion and condition modeling.

The model then maps the conditioning representation to a motion, expression, or control representation: \[m = M(z_s, z_c; \theta_m),\] where \(m\) may denote dense motion fields, facial landmarks, 3DMM coefficients, head pose, expression codes, attention maps, diffusion latents, deformation fields, or Gaussian offsets. This stage is central because it determines how speech, pose, and expression are transferred while preserving identity. For example, implicit-keypoint methods learn compact motion representations for efficient animation [24], diffusion-transformer methods model audio-to-motion or audio-to-video distributions [27], [28], and Gaussian avatar methods predict audio-driven deformations in a 3D representation [33]–[35].

2.1.0.3 Synthesis.

A generator, decoder, renderer, or denoising model produces the output frames: \[V_g = G(z_s, m; \theta_g).\] In 2D methods, \(G\) is often an image or video generator. In GAN-based systems, \(G\) is trained adversarially to produce photorealistic frames [41]. In diffusion-based systems, synthesis may proceed through iterative denoising in pixel space or latent space [27]–[29]. In 3D-aware methods, \(G\) may instead be a neural renderer using a mesh, triplane, NeRF, or Gaussian representation [31]–[33], [36].

2.1.0.4 Refinement.

Many systems include refinement stages to improve mouth details, teeth stability, eye motion, high-frequency texture, background consistency, or temporal smoothness. These refinements may involve local mouth generators, face enhancement, super-resolution, temporal discriminators, optical-flow constraints, video-to-video editing, or region-aware inpainting [38]. Such components are particularly important for high-resolution output and long-duration generation.

2.1.0.5 Training objectives.

Talking head generation models are usually trained with a combination of losses. A reconstruction loss encourages the generated video to match the target video: \[L_{\text{rec}} = \lVert V - V_g \rVert_1.\]

A perceptual loss compares high-level visual features extracted by a pre-trained network: \[L_{\text{perc}} = \lVert \phi(V) - \phi(V_g) \rVert_2,\] where \(\phi(\cdot)\) denotes a feature extractor such as VGG [42].

An adversarial loss encourages photorealistic outputs: \[L_{\text{adv}} = \mathbb{E}_{V \sim p_{\text{data}}}[\log D_{\text{adv}}(V)] + \mathbb{E}_{V_g \sim p_{\text{gen}}}[\log(1 - D_{\text{adv}}(V_g))],\] where \(D_{\text{adv}}\) is a discriminator.

Identity preservation can be encouraged using a face-recognition embedding: \[L_{\text{id}} = 1 - \cos(\psi(I_s), \psi(V_g)),\] where \(\psi(\cdot)\) is an identity encoder.

Lip synchronization can be encouraged by comparing audio and visual speech embeddings: \[L_{\text{sync}} = d(\eta_a(A), \eta_v(V_g)),\] where \(\eta_a(\cdot)\) and \(\eta_v(\cdot)\) encode audio and visual speech features, and \(d(\cdot)\) is a distance function.

Temporal stability can be encouraged through frame-difference, optical-flow, or feature-consistency losses: \[L_{\text{temp}} = \sum_t \lVert \Delta V_t - \Delta V_{g,t} \rVert_1.\]

The full training objective is typically a weighted combination of complementary terms: \[L_{\text{total}} = \lambda_{\text{rec}}L_{\text{rec}} + \lambda_{\text{perc}}L_{\text{perc}} + \lambda_{\text{adv}}L_{\text{adv}} + \lambda_{\text{id}}L_{\text{id}} + \lambda_{\text{sync}}L_{\text{sync}} + \lambda_{\text{temp}}L_{\text{temp}}.\]

Not all methods use all these objectives. Keypoint-based reenactment methods often emphasize reconstruction, perceptual, equivariance, and motion losses [24], [39], [40]. Audio-driven methods often include synchronization and expression-related losses [8], [12], [25], [28]. Diffusion-based methods optimize denoising or score-matching objectives, conditioned on audio, pose, identity, or reference frames [27], [29], [38]. Neural-rendering and Gaussian-splatting methods may include geometry, deformation, photometric, landmark, perceptual, wrinkle, and view-consistency losses [33], [35]–[37].

Figure 1 illustrates a representative pipeline, but modern talking head generation methods differ substantially in how they encode identity, represent motion, impose temporal consistency, and synthesize output frames. This diversity motivates the taxonomy used in the remainder of this survey.

3 Taxonomy of Talking Head Generation Methods↩︎

Talking head generation methods can be categorized in several ways: by the driving signal, by the internal representation, by the generative backbone, or by the intended application. Earlier surveys often grouped methods mainly according to the input modality, such as image-driven, audio-driven, video-driven, or 3D-based generation. This remains useful, but the field has evolved substantially. Recent methods increasingly combine multiple conditioning signals, use diffusion or video-transformer backbones, rely on 3D-aware or Gaussian-splatting representations, and target real-time or interactive deployment. Therefore, we adopt a broader taxonomy that reflects both the driving modality and the representation/generation paradigm.

Figure 2 summarizes the taxonomy used in this survey. The categories are not mutually exclusive. For example, VASA-1 is audio-driven but also designed for real-time generation [25]; EditYourself is both diffusion-based and editing-oriented [38]; and GaussianSpeech is both audio-driven and 3D Gaussian-splatting-based [33]. The taxonomy should therefore be read as a way of understanding the dominant design choice of each family, rather than as a strict partition.

3.1 2D Visual- and Video-driven Animation↩︎

Visual- and video-driven methods animate a source identity using visual motion cues such as landmarks, keypoints, optical flow, dense motion fields, pose parameters, or a driving video. This family includes early facial reenactment systems, few-shot neural head models, expression transfer methods, motion retargeting approaches, and recent efficient portrait animation systems.

Early reenactment methods formulated talking head generation as the transfer of facial expressions from a source performer to a target identity. Face2Face [43] demonstrated real-time facial reenactment from monocular video using explicit face tracking and rendering. ReenactGAN [44] moved this idea towards neural synthesis by mapping facial boundaries into a latent space before generating the target face. Few-shot approaches such as NeuralHead [45] and MarioNETte [46] then showed that realistic talking heads could be produced from only a small number of reference images.

A second line of visual-driven work focuses on expression transfer and attribute manipulation. Methods such as X2Face [47], PIRenderer [48], FD-GAN-LS [49], StyleMask [50], and AVFR [51] aim to separate identity from pose, expression, lighting, or other facial attributes. These methods are useful when the goal is not only to reproduce speech motion, but also to control specific aspects of facial appearance or expression.

Motion retargeting methods provide a more general formulation: animate a source image according to the motion of a driving sequence. FOMM [39] introduced a self-supervised keypoint-based motion representation that became influential beyond talking heads. Subsequent methods such as DAM [52], TPSM [7], CrossID-GAN [53], and DaGAN/DaGAN++ [9], [10] improved motion modeling, occlusion handling, depth awareness, and cross-identity transfer. More recent systems such as LivePortrait [24] revisit the implicit-keypoint paradigm with stronger training data, stitching, retargeting control, and a focus on efficiency and practical deployment.

High-resolution visual animation is another important direction. StyleHEAT [54], MegaPortrait [55], and LipFormer [56] address the limitations of low-resolution face datasets and low-detail synthesis by using StyleGAN priors, high-resolution training data, codebooks, or specialized face warping modules. These methods are particularly relevant for content creation and applications where frame-level fidelity matters.

Overall, 2D visual- and video-driven methods are often efficient and controllable, and many of them work well for reenactment and motion transfer. However, they can struggle with large pose changes, long-duration temporal consistency, accurate mouth interior synthesis, and generalization beyond the motion patterns seen during training.

3.2 Audio-driven Talking Head Generation↩︎

Audio-driven methods synthesize talking head videos from speech signals. Their central challenge is to infer plausible lip motion, facial expression, head movement, and speaking style from audio alone. This is difficult because the mapping from speech to face motion is one-to-many: the same utterance can be spoken with different expressions, poses, emotional states, and gestures.

Early audio-driven work focused heavily on lip synchronization. Synthesizing-Obama [57] demonstrated realistic audio-driven lip motion for a specific identity. Wav2Lip [58] became a widely used lip-syncing baseline by using an expert sync discriminator to align mouth motion with speech. TalkLip [59] further improved intelligibility by using lip-reading experts and contrastive learning. These methods are strong when the primary goal is accurate mouth movement, but lip-sync alone is not sufficient for a natural talking head.

A broader set of methods maps audio to facial and head dynamics. ATVGNet [60], MakeItTalk [61], NeuralVoicePuppetry [62], LSP [63], Audio2Head [12], and Speech2TalkingFace [64] generate facial motion, head pose, or upper-body cues from speech. SadTalker [8] improved practical audio-driven generation by predicting 3D motion coefficients and separately modeling expression and pose. These methods reflect the shift from lip-only synthesis to full talking-head dynamics.

Emotional and expressive talking heads form another important subfamily. EVP [65], GC-AVT [66], EAMM [67], and EmoGen [68] explicitly model affective or expressive facial dynamics. More recent systems such as VASA-1 [25], Dimitra [28], Teller [26], READ [27], and GaussianEmoTalker [37] continue this trend by emphasizing realism, emotional liveliness, streaming inference, or real-time deployment.

Audio-driven methods are central to virtual assistants, dubbing, telepresence, and assistive communication. Their main limitations are ambiguity in audio-to-motion mapping, difficulty producing natural non-verbal motion, sensitivity to language/accent variation, and the tendency to generate either overly static faces or exaggerated motion.

3.3 Diffusion and Foundation Video Models↩︎

A major recent shift is the move from task-specific GAN or keypoint pipelines to diffusion and video-transformer-based generation. These methods use large generative backbones to synthesize portrait videos conditioned on audio, identity images, pose, text, or multimodal control signals. Compared with earlier pipelines, they often produce more expressive and visually rich outputs, but they can also be computationally expensive and harder to control precisely.

EMO [17], Hallo [18], Hallo2 [19], and Hallo3 [29] are representative of this direction. These methods use diffusion or video diffusion transformer formulations for audio-driven portrait animation, with increasing attention to long-duration generation, high resolution, dynamic motion, and realistic backgrounds. Dimitra [28] similarly frames expressive talking head generation through an audio-driven diffusion model.

Large-scale human animation systems extend the problem beyond close-up faces. OmniHuman-1 [30] treats talking human generation as a one-stage conditioned human animation problem, supporting face close-ups, portraits, half-body and full-body animation, and multiple driving modalities. Such methods blur the boundary between talking head generation and general human video generation.

Diffusion-based methods are attractive because they can model complex distributions and generate high-fidelity results under weak or flexible conditioning. However, they also raise new challenges: slower inference, weaker temporal control in long videos, difficulty with exact identity preservation, and more complex evaluation. As these models become larger, practical deployment also depends on distillation, streaming generation, memory reduction, and controllable editing.

3.4 3D, NeRF, and Gaussian Avatar Methods↩︎

3D-aware methods aim to improve controllability, geometry consistency, and novel-view rendering. This family includes 3D morphable model methods, mesh-based avatars, NeRF-based representations, implicit avatars, and 3D Gaussian-splatting systems. Unlike purely 2D methods, these approaches try to model the head as a view-consistent object, making them especially relevant for telepresence, virtual reality, and interactive avatars.

Early 3D-aware talking head methods used 3D morphable models or mesh representations to separate identity, pose, and expression. MeshG [69], HeadGAN [11], HifiHead [70], CoRF [71], PECHead [72], IMavatar [73], ROME [74], and Real3D-Portrait [31] all use geometry or 3D-aware representations to improve motion control, view consistency, or identity preservation.

NeRF-based methods model the face or head as a neural radiance field. NerFACE [75], HeadNeRF [76], and HiDe-NeRF [77] represent important steps towards free-view or high-fidelity talking head synthesis. These methods can produce view-consistent results, but they may require identity-specific optimization or expensive rendering, limiting real-time applicability.

More recently, 3D Gaussian splatting has become a prominent representation for real-time avatars. GaussianAvatars [20], SplattingAvatar [21], LAM [32], GaussianSpeech [33], PGSTalker [34], UniGAHA [35], VASA-3D [36], and GaussianEmoTalker [37] show how Gaussian representations can support efficient rendering, personalized avatars, audio-driven deformation, and expressive talking heads. This direction is likely to become increasingly important because it combines the geometric consistency of 3D representations with the rendering speed needed for interactive applications.

The main trade-off is that 3D and Gaussian methods often require stronger assumptions, reconstruction steps, calibration, identity-specific data, or more complex training. Their advantage is strongest when novel views, real-time rendering, and interactive control matter.

3.5 Text, Semantic, and Editing-based Control↩︎

A final family focuses on semantic control, text-driven animation, and editing. Rather than only generating a talking head from audio or video, these methods allow users to control expression, style, pose, emotion, or speech content more directly.

Write-a-speaker [78] generates talking-head videos from text by modeling speech rhythm, pauses, contextual sentiment, facial expressions, and head motion. TalkCLIP [79] uses natural-language descriptions to control expressions through CLIP-aligned style representations. EditYourself [38] extends this direction by combining audio-driven generation with diffusion-transformer-based manipulation of talking head videos.

This category is important because talking head generation is increasingly becoming an editing and authoring problem, not only a synthesis problem. In practical applications, users may want to change what a person says, alter expression intensity, adjust gaze, retime speech, translate speech into another language, or modify a generated video while preserving identity and realism. These capabilities also increase ethical risks, making provenance, consent, and misuse prevention especially important.

3.6 Discussion of the Taxonomy↩︎

The taxonomy above reflects a shift in the field. Earlier work could be reasonably separated into image-driven, audio-driven, video-driven, and 3D-based methods. In the current landscape, those boundaries are less clear. Modern systems often combine multiple inputs, such as audio, image, pose, text, and video; multiple representations, such as 2D keypoints, 3DMM coefficients, diffusion latents, and Gaussian splats; and multiple objectives, such as lip-sync, identity preservation, emotional realism, and real-time rendering.

This has two implications. First, future surveys and benchmarks should avoid comparing methods only by input type. A fair comparison also needs to consider resolution, identity setting, duration, controllability, computational cost, and whether the method is optimized for offline generation or real-time interaction. Second, evaluation needs to move beyond frame-level visual quality. As methods become more expressive and interactive, the central questions are increasingly about temporal consistency, controllability, robustness, deployment cost, safety, and whether the generated video is useful for the intended application.

4 Datasets and Evaluation Metrics↩︎

4.1 Datasets↩︎

Datasets play a central role in talking head generation because they define the range of identities, poses, expressions, languages, recording conditions, and motion patterns that a model can learn. Early audio-visual datasets were often designed for lip reading, speaker recognition, or controlled emotional speech, and were later repurposed for talking head synthesis. More recent datasets are increasingly tailored to high-resolution facial video generation, text-to-video facial synthesis, multiview avatar reconstruction, diffusion-based pretraining, and fairness-aware evaluation. Therefore, dataset choice should be aligned with the intended task: lip synchronization, audio-driven portrait animation, video-driven reenactment, expression control, neural rendering, Gaussian avatar construction, or full/upper-body digital human generation.

The most commonly used datasets in earlier talking head generation work include GRID [80], CREMA-D [81], MSP-Improv [82], LRW [83], ObamaSet [57], VoxCeleb1 [84], VoxCeleb2 [85], LRW-1000 [86], FaceForensics++ [87], MEAD [88], and HDTF [89]. These remain important because many established methods report results on them. However, the field has moved towards larger, higher-resolution, and more richly annotated datasets such as TalkingHead-1KH [90], VFHQ [91], CelebV-HQ [92], Multiface [93], CelebV-Text [94], TalkVid [95], and SpeakerVid-5M [96]. Table [tab:datasets] summarizes representative datasets and their typical use cases.

Several trends are visible from Table [tab:datasets]. First, the field has moved from controlled, lab-based speech datasets towards large-scale in-the-wild video collections. Second, newer datasets increasingly provide richer supervision, including facial attributes, text descriptions, emotional labels, multiview geometry, demographic metadata, or quality filtering. Third, dataset scale alone is not sufficient: high-resolution detail, temporal stability, language coverage, consent, demographic diversity, and annotation quality are increasingly important. Finally, as talking head generation moves towards real-world deployment, benchmarks should evaluate not only average visual quality but also subgroup robustness, multilingual performance, long-duration stability, and susceptibility to misuse.

5 Evaluating Quality↩︎

Evaluation is one of the most difficult aspects of talking head generation. A generated video can score well under frame-level similarity metrics while still appearing unnatural to human viewers. Similarly, a method can achieve strong lip synchronization while failing because of identity drift, unstable teeth, flickering facial texture, poor eye motion, or unrealistic head movement. Talking head evaluation is therefore not a single-metric problem, but a multi-dimensional assessment of visual fidelity, audio-visual alignment, identity preservation, temporal consistency, expressiveness, efficiency, and human perceptual quality.

This issue has become more important as the field has moved from short landmark-driven or GAN-based videos to diffusion models, real-time audio-driven systems, and 3D/Gaussian avatar methods. Newer models often optimize for realism, controllability, streaming generation, and long-duration stability, but they are still evaluated using different datasets, resolutions, clip lengths, and hardware. As a result, benchmark numbers should always be interpreted together with the evaluation setting. In this section, we summarize the commonly used metrics and then compare representative methods using reconstruction metrics, reported benchmark values, and practical runtime/perceptual comparisons.

• Peak Signal-to-Noise Ratio (PSNR): PSNR measures the pixel-level similarity between generated frames and ground-truth frames. It is defined as: \[PSNR = 10 \log_{10}\left(\frac{MAX^2}{MSE}\right) \textcolor{blue}{\cite{huynh2008scope}},\] where \(MAX\) is the maximum possible pixel value and \(MSE\) is the mean squared error. Although PSNR is easy to compute, it often correlates poorly with perceptual realism in talking head videos.

• Structural Similarity Index (SSIM): SSIM measures structural similarity by considering luminance, contrast, and local image structure [97]. It is more perceptually meaningful than PSNR, but it is still frame-based and does not capture temporal artifacts, lip synchronization, or expression naturalness.

• Fréchet Inception Distance (FID): FID measures the distance between feature distributions of real and generated images: \[FID = \lVert \mu_{\text{real}}-\mu_{\text{gen}}\rVert_2^2 + \operatorname{Tr}\left(\Sigma_{\text{real}}+\Sigma_{\text{gen}} - 2(\Sigma_{\text{real}}\Sigma_{\text{gen}})^{1/2}\right) \textcolor{blue}{\cite{heusel2017gans}}.\] Lower FID usually indicates more realistic image-level generation. However, FID ignores temporal coherence and may not reflect lip-sync quality.

• Fréchet Video Distance (FVD): FVD extends the idea of FID to video by comparing feature distributions extracted from generated and real video clips [98]. It is more suitable for talking head generation than image-level FID because it accounts for temporal dynamics. However, it still does not explicitly measure whether the lips match the audio.

• Inception Score (IS): IS evaluates the quality and diversity of generated images using the predictions of a pre-trained classifier [99]. It is less commonly emphasized in recent talking head generation because it is not designed for faces, identity preservation, or speech synchronization.

• Cumulative Probability Blur Detection (CPBD): CPBD assesses image sharpness by estimating the probability of blur detection [100]. It is useful for detecting over-smoothed outputs, but it does not capture realism, identity, or temporal quality.

• LPIPS: LPIPS measures perceptual similarity using deep feature distances. It is often more aligned with human perception than PSNR or SSIM, but it is still usually computed frame-wise and therefore cannot fully capture video-level artifacts.

• Cosine Similarity (CSIM): CSIM is commonly used for identity preservation. It measures cosine similarity between face-recognition embeddings of the source identity and generated frames [101]. A high CSIM suggests that the generated video preserves the source identity.

• Lip-sync metrics: Lip synchronization is often evaluated using SyncNet-style audio-visual correspondence metrics [102]. LSE-D and LSE-C from Wav2Lip [58] measure lip-sync distance and confidence respectively. More recent papers often report Sync-C and Sync-D, where higher Sync-C and lower Sync-D indicate better audio-visual alignment.

• Head motion and expression metrics: Head motion diversity can be estimated using pose or head-motion embeddings, for example using Hopenet [103]. Beat Align Score measures temporal alignment between audio beats and generated motion [104]. Recent expressive talking head papers also use expression-aware metrics such as E-FID to measure expression mismatch.

• Mean Opinion Score (MOS) and user studies: Human evaluation remains important because no automatic metric captures all aspects of talking head quality. MOS asks human raters to judge realism, lip-sync, smoothness, and overall visual quality on a scale, typically from 1 to 5 [105]. However, user studies can be expensive, subjective, and difficult to reproduce.

• Qualitative failure mode analysis: Qualitative analysis is still necessary. Common failure modes include distorted teeth, blurry mouths, identity drift, unnatural eye gaze, frozen expressions, flickering textures, unstable backgrounds, exaggerated pose, poor emotional alignment, and failure under occlusion or large head rotation.

• Generalization across datasets: A method should be evaluated beyond a single curated benchmark. Testing across datasets with different lighting, demographics, languages, resolutions, and recording conditions helps reveal whether a method is robust or overfitted to a narrow distribution [106].

• Efficiency and deployment metrics: For real-world use, runtime, latency, GPU memory, model size, and streaming capability are essential. This is especially important for video conferencing, virtual assistants, gaming, and telepresence. A visually strong model that requires several minutes per short video may be unsuitable for interactive deployment.

5.1 Evaluation Using Existing Metrics↩︎

We first compare representative video-driven, portrait-animation, and recent audio-driven talking head methods using commonly reported reconstruction and perceptual metrics. Table 1 shows that no single model dominates across all metrics. For example, DaGAN++ achieves the best PSNR, SSIM, and AED, while FOMM and Face vid2vid perform better on other metrics. READ and Teller are included to reflect the newer generation of efficient and real-time audio-driven systems. This reinforces a central issue in talking head evaluation: different metrics reward different properties, and none of them alone captures perceived talking-head quality.

To complement the reconstruction-based metrics in Table 1, Table 2 reports additional metrics for recent audio-driven systems. FID and FVD capture image- and video-level distribution quality, while Sync-C and Sync-D measure audio-visual synchronization. These metrics are increasingly common in recent talking head evaluation and are particularly useful for audio-driven systems where speech alignment and temporal realism are central.

5.2 Comparing Visual Quality of Methods↩︎

To visually investigate how these methods perform, we use several source images and compare publicly available implementations. The goal of this analysis is not to produce a perfectly controlled benchmark, but to illustrate the gap between reported quantitative metrics and the visual quality perceived by human viewers.

First, we use images of Barack Obama, LeBron James, Chris Hemsworth, and Son Heung-Min, all public figures from different countries. We test TPSM [7], SadTalker [8], LipGAN/Wav2Lip [58], EmoGen [68], and other available methods. The outputs for Obama can be seen in Figure 3, for LeBron James in Figure 4, for Chris Hemsworth in Figure 5, and for Son Heung-Min in Figure 6. For future versions of this benchmark, the same evaluation should also be repeated using public benchmark identities or consented data to avoid copyright and synthetic-media concerns.

Across these examples, we observe that metric performance does not always align with visual preference. Some methods that perform strongly under reconstruction metrics still produce visible artifacts, while methods with weaker reported benchmark numbers can appear more natural in selected qualitative examples. Older methods such as Audio2Head tend to show weaker visual quality, whereas FOMM remains surprisingly competitive despite its age. TPSM, SadTalker, and Video Re-Talking often produce strong visual outputs because they preserve more of the source face and focus generation on regions such as landmarks, mouth motion, or facial deformation. However, these methods differ in driving signal: TPSM benefits from a reference video, while SadTalker is audio-driven, making direct comparison difficult.

Table 3 compares methods in terms of practical usability. In addition to inference time and GPU memory, we include human quality ratings. The 2025 entries show the improvement of recent efficient audio-driven systems, which achieve strong perceptual quality while substantially reducing inference time compared with slower earlier methods.

The main conclusion is that talking head evaluation remains unsettled. Frame-level metrics such as PSNR and SSIM are easy to report but often miss perceptual realism. FID and FVD better capture image and video distributions, but they do not directly measure identity preservation or speech alignment. Sync metrics capture lip synchronization, but they can ignore expression naturalness, head motion, and temporal stability. Human evaluation remains necessary, but it is expensive and difficult to reproduce. Therefore, future evaluations should report a balanced set of metrics: visual quality, identity preservation, lip-sync, temporal consistency, expressiveness, runtime, memory usage, and human preference. Crucially, comparisons should also state the driving modality, dataset, resolution, clip duration, and hardware, otherwise the numbers can be misleading.

6 What Next?↩︎

Talking head generation is moving from a narrow synthesis problem towards a broader problem of controllable, interactive, and responsible human video generation. Early systems mainly aimed to animate a face from audio or reenact one person using another person’s motion. Recent work has expanded this scope in several directions: diffusion and video-transformer models generate richer and more expressive motion; real-time systems make interactive deployment feasible; 3D and Gaussian-splatting approaches improve view consistency and avatar control; and text- or semantics-driven methods turn talking head generation into an editing and authoring tool [17]–[19], [25]–[29], [33]–[38].

This progress creates substantial opportunities. Talking heads can support virtual assistants, telepresence, dubbing, education, accessibility, gaming, social media, and professional content creation. However, the same technical progress also increases the risk of impersonation, non-consensual synthetic media, identity misuse, and misinformation. The next stage of the field therefore requires not only better generation quality, but also better evaluation, consent mechanisms, provenance, watermarking, fairness analysis, and deployment safeguards. A future-facing view of talking head generation must treat realism and responsibility as coupled goals rather than separate concerns.

6.1 Application Areas↩︎

Talking head models have demonstrated potential across a wide range of applications:

• Digital avatars and virtual assistants. Talking heads can be used to create realistic virtual agents with synchronized speech, facial expressions, gaze, and head motion. This is increasingly relevant for conversational AI, customer-service agents, gaming characters, educational tutors, and embodied assistants. Recent real-time and streaming methods make this application more practical because they reduce latency and enable interactive responses [25]–[27].

• Video conferencing and telepresence. Talking head generation can support more natural remote communication by correcting gaze, improving facial expressiveness, reducing bandwidth requirements, or generating view-consistent avatars. 3D-aware and Gaussian-splatting-based methods are especially relevant here because they can provide stronger geometric consistency and real-time rendering [33]–[36].

• Dubbing, translation, and localization. Audio-driven talking heads can adapt lip motion and facial expression to translated speech, making films, online lectures, news, and educational material more accessible across languages. This requires not only lip synchronization but also preservation of speaker identity, emotional tone, and culturally appropriate expression.

• Synthetic media and content creation. Talking head models can reduce the cost of producing videos for films, advertising, podcasts, audiobooks, online courses, and social media. Diffusion and video-transformer methods are particularly important for this direction because they can generate higher-quality and more expressive outputs under flexible conditioning [17]–[19], [28]–[30].

• Semantic editing and post-production. Recent methods increasingly treat talking head generation as an editing problem rather than only a generation problem. Users may want to alter speech, expression, gaze, emotion, pose, or timing while preserving identity and realism. Text- and semantics-driven approaches such as TalkCLIP, Write-a-speaker, and EditYourself point towards more controllable authoring tools [38], [78], [79].

• Accessibility and assistive communication. Talking heads can support assistive communication by generating expressive avatars for people who have difficulty speaking, enabling more natural text-to-speech interfaces, or creating educational material with synchronized visual speech. They may also support visual speech learning and language education, provided that the generated lip motion is accurate and reliable.

• Healthcare, therapy, and social interaction. Expressive avatars may support remote care, mental-health coaching, rehabilitation, social companionship, and training simulations. These applications require additional caution because failures in expression, identity, trust, or consent can have serious consequences.

• Interactive 3D and virtual reality avatars. Gaussian-splatting and neural-rendering methods are pushing the field towards real-time, view-consistent avatars for virtual reality, augmented reality, and immersive telepresence. In these settings, talking heads must be stable under viewpoint changes, responsive to user interaction, and robust over long sessions [33], [35]–[37].

• Misuse and digital deception. The same capabilities that make talking heads useful also make them dangerous. High-quality talking head generation can be used for impersonation, scams, misinformation, harassment, and non-consensual synthetic media. This is not a peripheral issue; it is one of the central deployment challenges for the field.

In summary, talking head generation has moved from a visual synthesis task to a platform technology for human-computer interaction, media production, and virtual presence. The strongest future applications will likely combine high-quality generation with controllability, low latency, consent, and clear provenance.

6.2 Ethical and Societal Considerations↩︎

While talking head models unlock new creative and communicative possibilities, they also raise serious ethical and societal concerns:

• Deepfakes, misinformation, and impersonation. Talking head generation can create photorealistic videos of public figures or private individuals saying things they never said. As models become more realistic, faster, and easier to use, the cost of producing deceptive media decreases. This creates risks for elections, journalism, financial scams, personal reputation, and public trust.

• Consent and control over likeness. Talking head models rely on facial appearance, voice, expression, and motion, all of which are closely tied to personal identity. Generating a person without consent, or reusing their likeness outside the original context, raises serious ethical concerns. Consent should be explicit, revocable where possible, and specific to the intended use.

• Privacy and biometric data. Training and deploying talking head models often requires large collections of face videos, speech recordings, and identity-related data. These are biometric signals, not ordinary images. Datasets and commercial systems should therefore document data sources, consent procedures, retention policies, and restrictions on downstream use.

• Watermarking, provenance, and content credentials. As generated media becomes harder to detect visually, provenance becomes increasingly important. Watermarking and content-credential standards can help record when, how, and by whom media was generated or edited. These mechanisms should be considered part of the generation pipeline rather than optional post-processing.

• Detection and the arms race problem. Deepfake detection is necessary but not sufficient. Detection systems often lag behind generation models and can fail under compression, editing, adversarial perturbations, or distribution shift. A stronger safety strategy should combine detection, provenance, access control, dataset governance, and user-facing disclosure.

• Bias, fairness, and demographic robustness. Talking head models may perform worse for underrepresented groups if training data is imbalanced across skin tone, age, gender, language, accent, disability, or cultural expression. Future evaluation should report subgroup performance and failure cases rather than only average metrics.

• Authenticity, disclosure, and user trust. Users should know when they are interacting with a generated avatar or watching synthetic media. This is especially important in journalism, education, healthcare, politics, and customer service. Disclosure should be visible and persistent, not hidden in metadata that platforms may remove.

• Rights of public figures, private individuals, and the deceased. Talking head generation enables synthetic videos of celebrities, politicians, private citizens, and deceased individuals. These cases raise difficult questions about dignity, defamation, copyright, publicity rights, family consent, and historical representation.

• Labour and creative-industry impacts. Synthetic presenters, actors, and dubbing tools may reduce production costs, but they may also affect performers, voice actors, translators, and video editors. Responsible deployment should consider compensation, attribution, licensing, and labour rights.

Overall, talking head generation sits at the intersection of creativity, identity, and trust. Technical progress should therefore be accompanied by governance mechanisms that make generated media traceable, consensual, fair, and accountable.

6.3 Challenges and Future Directions↩︎

Despite rapid progress, several open challenges remain:

• Evaluation beyond frame-level metrics. PSNR, SSIM, LPIPS, FID, and FVD each capture only part of the problem. Future benchmarks should jointly evaluate identity preservation, lip synchronization, expression realism, head motion, temporal stability, long-duration consistency, runtime, memory, and human preference.

• Long-duration stability. Many methods perform well on short clips but degrade over longer videos. Common failures include identity drift, expression collapse, background instability, repeated motion patterns, and gradual loss of lip-sync. Long-duration generation should become a standard evaluation setting.

• Real-time and streaming generation. Recent systems such as VASA-1, Teller, and READ show that real-time audio-driven generation is becoming feasible [25]–[27]. Future work should reduce latency further, support consumer hardware, and maintain quality under streaming audio, interruptions, and conversational turn-taking.

• Controllable expressiveness. A good talking head should not only synchronize lips with audio, but also control expression, gaze, emotion, gesture, head pose, and speaking style. The challenge is to provide this control without making outputs look artificial or causing identity drift.

• 3D consistency and interactive avatars. NeRF and Gaussian-splatting methods improve view consistency and rendering speed, but challenges remain in reconstruction accuracy, mouth interior modelling, hair, accessories, occlusion, and generalization from a single image or short video [33]–[37].

• Diffusion efficiency and controllability. Diffusion and video-transformer models can generate high-quality and expressive outputs, but they are often computationally expensive and can be difficult to control precisely. Future research should focus on faster sampling, distillation, streaming diffusion, better temporal control, and stronger identity preservation [17]–[19], [27]–[29].

• Multilingual and cross-cultural generation. Lip motion, facial expression, gesture, and prosody vary across languages and cultures. Future models should be evaluated on multilingual speech, accents, code-switching, and culturally diverse expression patterns rather than only English-centric datasets.

• Personalization with limited data. Many applications require a personalized avatar from a single image, short clip, or small set of examples. Future systems should improve few-shot and zero-shot generation while avoiding overfitting, identity leakage, or unauthorized reuse of personal likeness.

• Fairness and dataset governance. The field needs better documentation of dataset composition, consent, demographics, capture conditions, and allowed uses. New datasets such as TalkVid and SpeakerVid-5M reflect the move towards larger and more diverse training corpora, but scale alone does not guarantee fairness or responsible use [95], [96].

• Safety-by-design generation. Future talking head systems should integrate safeguards such as consent verification, watermarking, provenance metadata, misuse monitoring, and restrictions on high-risk identities or contexts. Safety should be built into the model, interface, and deployment pipeline.

• Human-centred evaluation. Talking head videos are ultimately consumed by people. Future work should include user studies that measure trust, comfort, perceived authenticity, emotional appropriateness, accessibility, and task effectiveness, rather than only visual similarity.

In the next phase of talking head generation, the most important advances may not come from visual realism alone. The field is likely to be shaped by the ability to generate videos that are controllable, temporally stable, efficient, multilingual, fair, and clearly disclosed as synthetic when appropriate. The long-term goal should be trustworthy talking head generation: systems that expand communication and creativity while protecting identity, consent, and public trust.

7 Conclusion↩︎

Talking head generation has evolved from early rule-based and landmark-driven facial animation into a broad research area spanning audio-driven synthesis, video reenactment, diffusion-based portrait animation, 3D-aware neural rendering, Gaussian avatar models, and semantic editing. This survey reviewed the major methodological families, datasets, evaluation protocols, practical deployment considerations, and ethical challenges that define the current landscape. While recent methods have made substantial progress in realism, lip synchronization, controllability, and real-time performance, the field still lacks unified evaluation standards that jointly capture identity preservation, temporal consistency, expression naturalness, computational efficiency, and human perceptual quality. At the same time, the growing realism and accessibility of these systems makes responsible development essential, particularly around consent, provenance, watermarking, demographic fairness, and misuse prevention. Future progress will therefore depend not only on generating more realistic talking heads, but on building systems that are robust, controllable, efficient, transparent, and trustworthy for real-world use.

References↩︎