1 Introduction↩︎

3D animatable head avatar creation aims to reconstruct 3D animatable heads from 2D inputs, with widespread applications such as teleconferencing, virtual reality (VR), and augmented reality (AR). Despite the success achieved in 3D head reconstruction from multi-view or monocular videos [1]–[4], the requirement for video-specific optimization makes such methods time-consuming. To improve efficiency, some approaches reconstruct 3D heads from a single image in a feed-forward manner [5]–[9] using Neural Radiance Fields (NeRF) [10]. Recently, 3D Gaussian Splatting (3DGS) [11] is adopted [12]–[14] for its impressive rendering quality and speed.

Despite the advancement made in generalizable one-shot 3D animatable head avatar reconstruction, existing approaches generally fail when the rendering views significantly differ from the input camera pose. Trained on large-scale monocular face video datasets [15], [16], which predominantly capture frontal face motions, these methods are fundamentally limited to near-frontal or partial views, and cannot handle the severe ambiguity of unseen regions (, side/back head), which harms the realism of the reconstructed 3D heads.

To realize efficient avatar animation and \(360^\circ\) view synthesis at the same time, we introduce a new framework for the novel setting of one-shot feed-forward 3D full-head animatable avatar creation via inpainted UV-space Gaussian modeling. To allow for animation control, we generate Gaussian primitives in the UV space of a parametric face model (FLAME [17]) so that Gaussians are rigged to the mesh surface and can move along with it during animation. Since the input image only provides appearance information from a single view, projecting its features to the UV space results in incomplete local UV feature maps. To obtain knowledge of full-head geometry and textures, inspired by the recent success of 3D GANs in full-head generation [18], [19], we utilize a pretrained 3D GAN [18] and its feed-forward inversion method [20] to efficiently extract UV-space global full-head features from the input image and provide multi-view supervision during training. As 3D GAN inversion has limited reconstruction fidelity, we inpaint the incomplete but fine-grained local UV features with the structured global UV full-head feature. Specifically, to make full use of the input view, we leverage the symmetric nature of human faces and the UV space with a transformer-based feature fusion architecture. This approach allows tokens from the global UV map to query symmetric positions on the incomplete local UV feature maps for input appearances, guiding the recovery of local details, which results in inpainted UV features for Gaussian avatar modeling. To further improve the 3D visual quality of the reconstructed avatar, we propose a 3D total variation loss that encourages Gaussians to fully cover the whole avatar surface, alleviating hole-like artifacts.

Extensive experiments confirm that our method not only is the first to reconstruct 3D full-head avatars that can be viewed in 360\(^\circ\), but also surpasses previous methods in avatar animation performance. Our contributions are summarized as follows:

• We propose a new framework for the novel setting of one-shot feed-forward 360\(^\circ\) animatable avatar creation, which inpaints incomplete input local appearance details using global full-head priors from a pretrained 3D GAN in the UV space of a parametric face model for full-head Gaussian modeling.

• We perform symmetric inpainting on the local UV feature maps by taking advantage of the symmetric nature of human faces and the UV space, effectively enhancing the input appearance for high-fidelity 3D reconstruction.

• Extensive experiments on large-scale video datasets [15], [21] confirm that our method is the first to produce one-shot feed-forward 360\(^\circ\) animatable avatars, with clear improvements in side and back view synthesis while achieving state-of-the-art performance in avatar animation.

2 Related Work↩︎

2D Talking Head Generation. By leveraging powerful 2D generative models [22], [23], promising advancements have been made in 2D talking head generation by adding motion signals into them. Some works estimate motion in an unsupervised manner, representing it with warping fields described by unsupervised keypoints [24]–[28] or latent features [29], [30]. Others adopt pretrained models to predict facial landmarks [31]–[33]. However, the lack of 3D face modeling hinders their performance, leading to face distortion and identity leakage. Some methods [34]–[37] extract 3D Morphable Model (3DMM) [38] parameters to introduce 3D information, but they cannot support arbitrary viewpoint rendering. In contrast, we directly model human heads in 3D to enable free-viewpoint rendering. Optimization-based 3D Animatable Head Avatar Reconstruction. To achieve free-viewpoint rendering, a line of research reconstructs 3D animatable heads from multi-view or monocular videos of a specific person. By optimizing on videos of the same identity, these methods [1]–[4], [39]–[41] generally achieve high visual quality. However, they require videos with thousands of frames for optimization, which hinders practical use. To reduce the need for lengthy video data, another line of research leverages strong priors from 3D GANs [18], [19], [42], [43] and diffusion models [44]. Some methods [45]–[47] generate multi-view data with different motions from a single image and optimize on the generated data. Others perform optimization-based GAN inversion [48]–[50] from the input image on pretrained 3D GANs to create animatable 3D heads. However, the required optimization process remains time-consuming. In contrast to these methods, we create one-shot 3D animatable heads in a feed-forward manner, avoiding the cumbersome optimization process.

Feed-Forward 3D Animatable Head Avatar Reconstruction. To improve the generalizability, one-shot feed-forward 3D head reconstruction has gained increasing attention. ROME [51] extends 3DMM [17] meshes to model non-facial parts. With the excellent rendering quality of NeRF [10], some methods [5]–[9], [52] incorporate motion features into efficient tri-planes [42] for high-fidelity head avatar creation. However, their rendering speed generally remains slow. In recent years, 3DGS [11] is widely leveraged for its extraordinary rendering quality and efficiency. GAGAvatar [12] generates two sets of Gaussians for appearance and expression, and adds a neural renderer to enhance details. LAM [13] initializes Gaussians on FLAME [17] vertices and generates Gaussian attributes by querying image features through a transformer. However, trained on large monocular video datasets [15], [16], which rarely contain frames of the side and back of the head, these methods generally fail when the rendering viewpoint significantly differs from the input view. Avat3r [53] and FastGHA [54] leverage multi-view inputs and Vision Transformers to infer multi-view correspondence for high-quality reconstruction. On the contrary, we aim for 360\(^{\circ}\) renderings from a single image. Recent FlexAvatar [55] unifies training across monocular and multi-view datasets with learnable data source tokens, enabling the creation of complete 3D heads. In contrast, we directly inpaint the incomplete input local UV-space features with full-head priors from a pretrained 3D GAN [18], thus generating complete UV Gaussian attribute maps for full-head modeling.

3 Method↩︎

3.1 Overview↩︎

Given an input source image of a human face, denoted as \(\mathbf{I}_s\), we aim to reconstruct an animatable 3D full-head Gaussian avatar on the FLAME [17] mesh surface via UV parameterization. Fig. 2 depicts the overview of our framework. First, the UV-space feature extraction module extracts the texture features from \(\mathbf{I}_s\) and projects them into the UV space, producing a global UV feature map \(\mathbf{F}_g\) and multi-scale local UV feature maps \(\{\mathbf{F}_l^i\}_{i=1}^N\). It also predicts a shape offset \(\Delta \mathbf{p}_{uv}\) to refine the estimated FLAME mesh shape based on \(\mathbf{I}_s\). Next, the symmetric UV-space inpainting module takes advantage of the symmetric nature of human faces and the UV space, inpainting the incomplete but fine-grained local UV feature maps with global features to generate a set of UV Gaussian attribute maps, from which 3D Gaussian primitives are sampled to reconstruct the Gaussian avatar. Bound to the surface of the source FLAME mesh in canonical space, the Gaussian avatar can be animated with a novel FLAME expression derived from a driving image \(\mathbf{I}_d\) and rendered given a camera pose to obtain the output image \(\mathbf{I}_o\). Details on UV-space Gaussian modeling, the design of the two modules, the generation of UV Gaussian attribute maps, the regularization and training strategy are described in the following subsections.

3.2 UV-Space Gaussian Modeling↩︎

To utilize the animatable FLAME model for 3D full-head reconstruction and animation, we follow [56], [57] and bind 3D Gaussian primitives to the FLAME mesh surface via UV parameterization, enabling the use of efficient 2D backbones and regularizing Gaussian positions. Specifically, we generate a UV Gaussian attribute map \(\mathbf{A}_{\star}\in \mathbb{R}^{K\times K\times d_{\star}}\) for each Gaussian attribute \(\star \in \{\mathrm{color,rotation,} \mathrm{scale,opacity,position}\}\), where \(K\) is the size of the UV map and \(d_{\star}\) is the Gaussian attribute dimension, forming a set of UV Gaussian attribute maps \(\mathbf{A} \in \mathbb{R}^{K\times K\times 14}\). Since each valid texel in the UV space already corresponds to a 3D position, we use \(\mathbf{p} \in \mathbb{R}^{K\times K\times 3}\) to represent the FLAME-based UV-space 3D position map, and \(\mathbf{A}_{\mathrm{position}}\) to represent the 3D position offset. To balance the size differences between mesh faces in the UV space and in 3D space, we rescale the UV Gaussian scale map. Specifically, we compute the relative scales between the mesh face sizes on the 3D mesh surface and their corresponding UV face sizes as a relative scaling map \(\mathbf{s} \in \mathbb{R}^{K\times K\times 1}\). The UV Gaussian scale map is further updated as \(\mathbf{A}_{\mathrm{scale}} = \mathbf{s} \odot \mathbf{A}_{\mathrm{scale}}\), where \(\odot\) is the Hadamard product.

To produce 3D Gaussian primitives \(\mathcal{G}\), we perform grid sampling on \(\mathbf{A}\) following [56], [57]. The process is formulated as: \[\mathcal{G}= \mathrm{grid\_sample}(\mathbf{A},\mathcal{X}),\] where \(\mathcal{X}\) is the set of sampling positions in the UV space.

3.3 UV-Space Feature Extraction↩︎

To generate UV Gaussian attribute maps, we first extract the UV-space feature maps from the source image \(\mathbf{I}_s\). The extracted UV feature maps are utilized for later integration.

Full-Head Feature Generation. Since \(\mathbf{I}_s\) only provides the head appearance from the source camera view, it is insufficient to reconstruct the full-head appearance using only information from \(\mathbf{I}_s\). To obtain knowledge from the invisible areas in \(\mathbf{I}_s\), we propose to utilize the full-head prior of a 3D full-head GAN named PanoHead [18] and its feed-forward inversion method [20] to map \(\mathbf{I}_s\) to a full-head tri-plane \(\mathbf{T}\), which provides coarse 3D full-head features for \(\mathbf{I}_s\).

UV Shape Refinement. Utilizing the source FLAME mesh predicted by the 3DMM estimator, we can directly sample the tri-plane feature for each 3D position corresponding to a UV-space texel, producing a UV tri-plane feature map \(\mathbf{F_{T}^p}\). However, FLAME meshes often fail to model the shape of the hair, making the 3D position-based feature sampling inaccurate. We propose to use a 2D UNet \(\mathcal{F}_{refine}\) to refine the mesh shape. Given \(\mathbf{F_{T}^p}\) and the UV-space 3D position map \(\mathbf{p}\), we predict the UV shape offset \(\Delta \mathbf{p}_{uv}\) as: \[\Delta \mathbf{p}_{uv}=\mathcal{F}_{refine}([\mathbf{F_{T}^p},\mathbf{p}]),\] where \([\cdot,\cdot]\) is the concatenation operation. Then, by adding \(\Delta \mathbf{p}_{uv}\) to the original 3D position map \(\mathbf{p}\), we obtain a refined 3D position map \(\mathbf{p}_r=\mathbf{p}+\Delta \mathbf{p}_{uv}\).

Global UV Feature Extraction. With the refined 3D position map \(\mathbf{p}_r\), we directly sample the full-head tri-plane \(\mathbf{T}\) to produce the global UV feature map \(\mathbf{F}_g\). Produced by the pretrained 3D GAN with rich full-head prior knowledge, the tri-plane feature \(\mathbf{T}\) contains convincing global head geometry and full-head textures for \(\mathbf{I}_s\). Therefore, \(\mathbf{F}_g\) contains a complete full-head representation of \(\mathbf{I}_s\). However, constrained by the latent code dimension of the GAN inversion method and the resolution of the tri-plane, \(\mathbf{T}\) cannot faithfully preserve the appearance details visible in \(\mathbf{I}_s\), and consequently, neither can \(\mathbf{F}_g\). Additional local features from \(\mathbf{I}_s\) are needed for high-fidelity avatar reconstruction.

Local UV Feature Extraction. To extract appearance details from \(\mathbf{I}_s\), we use a CNN-based encoder to encode \(\mathbf{I}_s\) into multi-scale 2D image features \(\{\mathbf{F}_s^i\}_{i=1}^N\). To map the image features to the UV space, we utilize the refined 3D position map \(\mathbf{p}_r\). By projecting the 3D positions in \(\mathbf{p}_r\) onto the 2D feature space according to the source camera pose, we can sample the corresponding 2D image feature for each UV texel, resulting in a set of multi-scale UV source features \(\{\mathbf{F}_{s,uv}^i\}_{i=1}^N\). However, due to occlusion, only part of the 3D positions in \(\mathbf{p}_r\) are visible in the source image. To avoid obtaining mismatched features in the UV space, we filter out invisible 3D positions in \(\mathbf{p}_r\) following [58], [59], which results in a UV visibility mask \(\mathbf{M}_v\). By applying this mask to \(\{\mathbf{F}_{s,uv}^i\}_{i=1}^N\), we exclude textures sampled from the invisible positions in the source image. The resulting multi-scale local UV feature maps \(\{\mathbf{F}_{l}^i\}_{i=1}^N\) preserve the local appearance details from \(\mathbf{I}_s\), but they are incomplete for full-head modeling.

3.4 Symmetric UV-Space Inpainting↩︎

In this module, we aim to inpaint the fine-grained local UV feature maps \(\{\mathbf{F}_{l}^i\}_{i=1}^N\) with the coarse but complete global UV feature map \(\mathbf{F}_{g}\) for UV Gaussian attribute map generation.

To better inpaint the multi-scale local UV feature maps \(\{\mathbf{F}_{l}^i\}_{i=1}^N\) with the global UV feature map \(\mathbf{F}_{g}\), we also encode \(\mathbf{F}_{g}\) with a CNN-based UV map encoder to produce multi-scale global UV feature maps \(\{\mathbf{F}_{g}^i\}_{i=1}^N\), and inpaint \(\mathbf{F}_{l}^i\) with \(\mathbf{F}_{g}^i\) at scale \(i\) across \(N\) scales. Fig. 3 illustrates symmetric UV-space inpainting at scale \(i\).

Coarse Symmetric UV Inpainting. As a global full-head feature, \(\mathbf{F}_{g}^i\) provides plausible texture for each valid texel in the UV space, even for the invisible area in the source view. However, it lacks the appearance details visible in the source image, which are contained in \(\mathbf{F}_{l}^i\). Therefore, to inpaint \(\mathbf{F}_{l}^i\), we can transfer appearance details from \(\mathbf{F}_{l}^i\) to \(\mathbf{F}_{g}^i\), which will result in a completed UV-space feature. Intuitively, \(\mathbf{F}_{l}^i\) and \(\mathbf{F}_{g}^i\) are inherently aligned, as each valid UV texel in \(\mathbf{F}_{l}^i\) and \(\mathbf{F}_{g}^i\) corresponds to the same 3D position on human heads. However, we generate the global and local UV feature maps using 3D position-based sampling and 3D projection, and the 3D positions and projections may still contain errors. To alleviate the potential misalignment issue caused by the errors, inspired by [60], we use a transformer architecture to fuse the features, where \(\mathbf{F}_{g}^i\) serves as the query while \(\mathbf{F}_{l}^i\) serves as the key and value at the cross-attention layer. As illustrated in Fig. 3 (a), for each patch token in \(\mathbf{F}_{g}^i\), we query tokens located in a local window of size \(w\times w\) centered at the corresponding patch token in \(\mathbf{F}_{l}^i\), allowing for small deviations in 3D positions and projections. However, if the source image contains a side view, tokens on the opposite side of the face in the UV space can hardly retrieve useful information from \(\mathbf{F}_{l}^i\), as its corresponding local window only contains the masked-out area. Essentially, human faces are generally symmetric, and the face UV space is also symmetric. To better utilize the symmetry and retrieve more useful information from \(\mathbf{F}_{l}^i\), we also use patch tokens in the symmetric local window of \(\mathbf{F}_{l}^i\) as key-value pairs for cross-attention.

Local Symmetric UV Detail Enhancement. As patchification inevitably leads to information loss, especially for large patches used in large-resolution feature maps at large scale \(i\), appearance details contained in \(\{\mathbf{F}_{l}^i\}_{i=2}^N\) may not be fully transferred to the transformer outputs \(\{\mathbf{F}_{c}^i\}_{i=2}^N\). To further enhance the local appearance details from \(\{\mathbf{F}_{l}^i\}_{i=2}^N\) within the aligned areas, we predict an occlusion mask \(\mathbf{M}_o\) based on \(\mathbf{F}_{c}^i\) and \(\mathbf{F}_{l}^i\) to mask out the region of erroneous 3D projections in \(\mathbf{F}_{l}^i\), producing a more accurate local feature \(\mathbf{F}_{l,m}^i\): \[\mathbf{F}_{l,m}^i=\mathbf{M}_o \odot \mathbf{F}_{l}^i,\] where \(\odot\) is the Hadamard product. To utilize the symmetric nature of faces and the UV space, we horizontally flip the masked \(\mathbf{F}_{l,m}^i\) and add it to the masked-out area in \(\mathbf{F}_{l,m}^i\), taking full advantage of the appearance information from \(\mathbf{F}_{l}^i\). The output \(\mathbf{F}_{m}^i\) is further fused with \(\mathbf{F}_{c}^i\) with a convolution layer to produce the final \(\mathbf{F}_{f}^i\). The process is formulated as: \[\mathbf{F}_{m}^i = \mathbf{F}_{l,m}^i+\mathcal{F}_{\leftrightarrow}(\mathbf{F}_{l,m}^i) \odot (1-\mathbf{M}_v \odot \mathbf{M}_o),\] \[\mathbf{F}_{f}^i = \mathrm{Conv}([\mathbf{F}_{c}^i,\mathbf{F}_{m}^i]),\] where \(\mathcal{F}_{\leftrightarrow}\) is the horizontal flip operation, \(\odot\) is the Hadamard product, and \([\cdot,\cdot]\) is the concatenation operation.

3.5 UV Gaussian Attribute Map Generation↩︎

We generate the set of UV Gaussian attribute maps \(\mathbf{A}\) with the UV map decoder \(\mathcal{D}_{uv}\) and the inpainted UV features \(\mathbf{F}_{c}^1\) and \(\{\mathbf{F}_{f}^i\}_{i=2}^N\). We use the transformer output \(\mathbf{F}_{c}^1\) instead of \(\mathbf{F}_{f}^1\), as \(\mathbf{F}_{l}^1\) has the smallest resolution and contains few appearance details. \(\mathbf{F}_{c}^1\) serves as the initial input to \(\mathcal{D}_{uv}\), where it is gradually upsampled and processed with residual blocks. \(\{\mathbf{F}_{f}^i\}_{i=2}^N\) is added to the intermediate feature maps with the same size in \(\mathcal{D}_{uv}\). The final output of \(\mathcal{D}_{uv}\) is \(\mathbf{A}\). We also generate another attribute map \(\mathbf{A}^1\) only using \(\mathbf{F}_{c}^1\) as the input, which is also used to sample Gaussian attributes \(\mathcal{G}^1\) and render an output image \(\mathbf{I}_o^1\) during training, preventing \(\mathcal{D}_{uv}\) from relying too much on high-resolution inputs.

3.6 Regularization and Training Strategy↩︎

3D Total Variation Loss. As shown in Fig. 4, we observe that Gaussian primitives modeled on the FLAME mesh surface normally cannot cover the whole avatar surface, resulting in holes that make the Gaussians representing the opposite side of the head visible, which harms 3D reconstruction quality. To alleviate such artifacts, inspired by the UV total variation (TV) loss in [56] that uses TV loss on the UV renderings \(\mathbf{I}_{uv}\) rendered from the same Gaussians \(\mathcal{G}\) but with \(\mathcal{G}_{\mathrm{color}}\) set to the UV coordinates, we propose to use the TV loss on the 3D renderings \(\mathbf{I}_{3d}\), which is more suitable for the UV-space definitions where neighboring pixels in the rendered image can be far from each other in the UV space (, eyeballs and eyelids). Specifically, based on the sampled Gaussians \(\mathcal{G}\), we change the value of \(\mathcal{G}_{\mathrm{color}}\) to \(\mathcal{G}_{\mathrm{position}}\) while leaving other Gaussian attributes unchanged, and perform rendering to obtain the 3D rendering \(\mathbf{I}_{3d}\). The 3D total variation loss is defined as: \[\mathcal{L}_{3d}=\mathrm{TV}(\frac{\mathbf{I}_{3d}-(1-\mathbf{I}_{\alpha})}{\mathbf{I}_{\alpha}}),\] where \(\mathbf{I}_\alpha\) is the rendered alpha map. This regularization encourages neighboring pixels in the rendered image to be rendered from neighboring Gaussians in the 3D space.

Training Strategy. Except the offline 3DMM estimator and the frozen pretrained 3D full-head GAN and its inversion, we train the whole framework end-to-end. During training, we randomly pick two frames from a video of the same identity, in which one frame serves as the source image and the other serves as the driving image. With the source image \(\mathbf{I}_s\) and the 3DMM motion parameter from the driving image \(\mathbf{I}_d\), we aim to reconstruct a target image, which is \(\mathbf{I}_d\). As large-scale face video datasets generally contain frontal faces, we rely on the pretrained 3D full-head GAN and its inversion to generate pseudo multi-view images of \(\mathbf{I}_d\) for supervision. However, we find that the inversion method tends to generate inconsistent invisible parts of the head given different frames of a video, providing inconsistent multi-view supervision for different \(\mathbf{I}_d\) given the same \(\mathbf{I}_s\), which may hamper the 3D reconstruction quality. Therefore, we follow [8] to randomly switch between the animation mode and the 3D reconstruction mode. Specifically, in animation mode, \(\mathbf{I}_s\) serves as the source image and \(\mathbf{I}_d\) as the target image. In 3D reconstruction mode, we assign \(\mathbf{I}_d\) as the source image, \(\mathbf{I}_d\) together with its inverted novel views as the target images to avoid the inconsistency in the pseudo supervision.

For the training objective, we render RGB images \(\mathbf{I}_o\) and \(\mathbf{I}_o^1\), alpha maps \(\mathbf{I}_{\alpha}\) and \(\mathbf{I}_{\alpha}^1\) from \(\mathcal{G}\) and \(\mathcal{G}^1\), respectively. We supervise the RGB images using the target image(s) with a reconstruction loss \(\mathcal{L}_{re}=\mathcal{L}_{1}+\mathcal{L}_{lpips}\), and supervise the alpha maps with \(\mathcal{L}_{1}\) loss, denoted as \(\mathcal{L}_\alpha\). When \(\mathbf{I}_o\) contains frontal faces, we also use an ID similarity loss [61] \(\mathcal{L}_{id}\).

Apart from the 3D total variation loss, we also apply a series of regularizations to constrain the Gaussian primitives for better rendering quality. Following [57], we apply the eyeball TV loss \(\mathcal{L}_{eye}\) and the position offset regularization \(\mathcal{L}_{pos}=||\mathbf{A}_{\mathrm{position}}||_2\). To limit the extent of FLAME mesh offset, we regularize the predicted UV shape offset \(\Delta \mathbf{p}_{uv}\): \[\mathcal{L}_{shape}=\max(||\Delta \mathbf{p}_{uv}||_2-\epsilon ,0),\] where \(\epsilon\) is a threshold. To avoid producing Gaussian outliers, we further add a TV regularization to \(\Delta \mathbf{p}_{uv}\): \[\mathcal{L}_{shape}^{tv}=\mathrm{TV}(\Delta \mathbf{p}_{uv}).\] Thus, our regularization is defined as: \[\mathcal{L}_{reg}=\lambda_{3d} \mathcal{L}_{3d} +\lambda_{eye} \mathcal{L}_{eye}+\lambda_{pos}\mathcal{L}_{pos}+\lambda_{shape}\mathcal{L}_{shape}+\lambda_{shape}^{tv} \mathcal{L}_{shape}^{tv},\] where \(\lambda_*\) is the regularization weight.

To summarize, our training objective is: \[\mathcal{L}=\mathcal{L}_{re}+\mathcal{L}_{\alpha}+\lambda_{id}\mathcal{L}_{id}+\lambda^1(\mathcal{L}_{lpips}^1+\mathcal{L}_{\alpha}^1)+\mathcal{L}_{reg},\] where \(\lambda_{id}\) and \(\lambda^1\) are the loss weights, and \(\mathcal{L}_{lpips}^1\) and \(\mathcal{L}_{\alpha}^1\) denote the losses for \(\mathbf{I}_o^1\) and \(\mathbf{I}_{\alpha}^1\), respectively.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets. We train our model on the VFHQ dataset [15]. For data preprocessing, we follow [18], [20] to estimate the camera pose, crop and remove the background of each video frame. All frames are resized to \(512\times 512\). To obtain the FLAME parameters, we utilize the head tracker from [1]. For performance evaluation, we use the VFHQ test split. To evaluate the generalizability, we randomly sample 20 videos from HDTF dataset [21] following [12], [13]. We also sample 21 videos from a multi-view MEAD [62] video dataset to assess the effectiveness of our design under large camera pose changes.

Metrics. For self-reenactment, given ground-truth frames, we employ PSNR, SSIM, LPIPS [63] to measure the reconstruction quality. For identity preservation, we use cosine similarity (CSIM) of identity features extracted by [61]. For driving accuracy, we use average keypoint distance (AKD) based on [64], average expression distance (AED) based on [65] and average pose distance (APD) based on [66]. For the setting of cross-identity reenactment, with no ground-truth available, we use CSIM, AED and APD for evaluation, following prior works [12], [13].

4.2 Comparison with State-of-the-art Methods↩︎

We compare with 2D-based StyleHEAT [37], and 3D-based GOHA [5], Real3DPortrait [7], Portrait4D [8], Portrait4D-v2 [9], GAGAvatar [12] and LAM [13].

Quantitative Comparison. We report the results of the quantitative comparison with state-of-the-art methods on the VFHQ [15] and HDTF [21] datasets in Tab. 1 and Tab. 2. For self-reenactment, we utilize the initial frame from each video as the source image and the remainder as the driving images. To perform cross-identity reenactment, we randomly sample a frame from the video of a different identity as the source image. Our method generally achieves the best performance for self-reenactment and remains competitive for cross-identity reenactment. With respect to reconstruction quality (, PSNR, SSIM), our method significantly outperforms previous methods on both datasets, highlighting the effectiveness of symmetric UV-space inpainting, where appearance details from the input source image are fully preserved for Gaussian attribute map generation. Our method also achieves the best AKD and APD for self-reenactment, which confirms the accuracy of expression and pose during animation. For cross-identity reenactment, our method outperforms others in identity preservation on the VFHQ dataset, where large differences in camera pose between source and driving image pairs occur most frequently. This indicates the superiority of modeling 3D full heads. In terms of motion accuracy for cross-identity reeanctment, we achieve slightly worse AED and APD. Based on the motion from the tracked FLAME parameters, the expression and pose accuracy of our method is limited by the tracking accuracy. Nevertheless, for AED and APD scores, our method outperforms LAM [13], which shares the same FLAME head tracker with us.

Qualitative Comparison. Fig. 6 presents a qualitative comparison of our cross-identity reenactment results against state-of-the-art methods. Our method better maintains source face identity and head shape than previous methods under large camera pose changes (rows 1 and 3), while accurately imitating driving expressions (rows 2 and 4). This confirms the importance of 3D full-head reconstruction during animation. The 360\(^\circ\) rendering results of the avatars reconstructed by our method in Fig. 5 validate the effectiveness of our UV-space full-head modeling, allowing for more realistic 3D avatars. More qualitative comparison results are shown in the supplementary material.

Animation Speed. Tab. 3 measures the time of reenactment during inference using an NVIDIA RTX 3090. Based on the efficient 3DGS representation, our method achieves an FPS of 246, outperforming previous methods. This confirms the efficiency of our method and highlights its potential real-time applications.

4.3 Ablation Study↩︎

We perform ablation studies to analyze the effectiveness of our framework design. The quantitative comparison is presented in Tab. 4 and the qualitative results are depicted in Fig. 7, Fig. 8 and Fig. 4.

UV Shape Refinement. According to Tab. 4, removing UV shape refinement (w/o \(\Delta \mathbf{p}_{uv}\)) degrades the overall performance, as we cannot obtain a more accurate 3D position to sample the appropriate features from the full-head tri-plane and the 2D image features. Fig. 7 shows blurry teeth and hair as well as clear boundary artifacts in the rendered frames without UV shape refinement.

Local UV Feature Maps. Tab. 4 and Fig. 7 show that we can hardly preserve the identity from the input source image without using local UV feature maps (w/o \(\{\mathbf{F}_{l}^i\}_{i=1}^N\)), with a significant drop in CSIM. Although the full-head tri-plane is inverted from the source image, it cannot preserve appearance details, as the dimension of the inverted latent code and the tri-plane resolution are limited. We cannot reconstruct faithful 3D head avatars without using local UV feature maps, which confirms the importance of inpainting based on them.

Symmetric UV-Space Inpainting. We utilize a transformer-based UV feature inpainting strategy with a symmetry prior. We analyze its effectiveness by removing the transformer (w/o \(\mathcal{T}\)) and removing symmetric operations (w/o Sym.) respectively.

When the transformer is removed, the symmetric local UV features are directly fused with the global UV features without filtering out potential errors in 3D projection, which decreases the overall performance as shown in Tab. 4. We can observe that the teeth are blurry and artifacts appear at the eye in Fig. 7, which is caused by inaccurate 3D projections.

After we remove the symmetric operations, the quantitative results in Tab. 4 are generally comparable to the full design on HDTF, but drop consistently when evaluated on the multi-view MEAD dataset. Since the symmetric operations take the most effect when only a side face is visible in the source image, they cannot contribute significantly to the final results given the videos with frontal faces in HDTF. Nevertheless, they can generally improve full-head avatar animation quality on the multi-view video dataset where source and driving camera poses may vary significantly. As shown in Fig. 7 row 2, the symmetric operations reduce artifacts at the source view boundaries, improving the reconstruction quality of the occluded parts. Additionally, Tab. 4 indicates that the symmetric operations consistently improve facial motion accuracy (, AKD, AED, APD). The reason is that by enforcing symmetry on local UV feature maps, the accuracy of projections from 2D image features to the UV space can be improved, making the generated Gaussian attribute maps more aligned with the underlying FLAME meshes. The improved projection accuracy is also confirmed by the clearer teeth when the symmetric operations are added in Fig. 7 row 1.

Fig. 8 visualizes the UV-space feature maps before and after symmetric UV-space inpainting. Compared to the local UV map \(\mathbf{F}_{l}^i\), the transformer output \(\mathbf{F}_{c}^i\) provides a coarse but complete UV-space full-head representation. Further enhanced by \(\mathbf{F}_{l}^i\), the final inpainting output \(\mathbf{F}_{f}^i\) reduces grid-like artifacts caused by patchification in \(\mathbf{F}_{c}^i\), preserving appearance details from the source view. The inpainted UV-space features are utilized for UV Gaussian attribute map generation, enabling 360\(^\circ\) animatable Gaussian avatar reconstruction.

3D Total Variation Loss. As shown in Fig. 4, without using the 3D total variation loss (w/o \(\mathcal{L}_{3d}\)), the generated Gaussians cannot cover the avatar surface, leaving holes on the reconstructed avatars, which greatly harms the 3D quality.

Using the UV total variation loss (\(\mathcal{L}_{uv}\)) from [56] instead of \(\mathcal{L}_{3d}\) can also encourage the Gaussians to cover the avatar surface, as \(\mathcal{L}_{uv}\) encourages neighboring pixels in the rendered image to be from neighboring texels in the UV space. However, we use a different UV parametrization where eyeballs and inner mouth are far from the eyelid and lip areas in the UV space for full head modeling. Therefore, as shown in Fig. 4, using \(\mathcal{L}_{uv}\) leads to artifacts in the eye and mouth areas, causing Gaussians bound to other face areas to cover the eyeballs and mouth, which harms the avatar animation quality. The degradation is also confirmed by the quantitative results in Tab. 4. In contrast, using our \(\mathcal{L}_{3d}\) can realize a 3D consistent full-head avatar without damaging the animation quality.

5 Conclusion↩︎

In this paper, we have tackled a novel setting of building one-shot feed-forward 3D full-head animatable avatars, which allows for more immersive 3D talking heads. We propose a novel framework that generates complete Gaussian attribute maps in the UV space of a parametric head model via inpainting. Specifically, we perform symmetric UV-space inpainting on the incomplete but fine-grained input local image features with the rich 3D full-head prior knowledge from a pretrained 3D GAN, effectively enhancing the fidelity of 3D reconstruction. Extensive experiments confirm that our method is able to generate one-shot feed-forward 3D animatable head avatars that can be viewed in 360\(^\circ\).

Acknowledgements↩︎

The research is supported in part by Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321, ITF PRP/046/24FX, Science & Technology Cooperation Program of Shandong under grant No. SDST26EG01, SAIL Research Project, HKUST-Zeekr Coolaborative Research Fund.

6 Preliminary↩︎

3D Gaussian Splatting (3DGS) [11] has gained recognition as a promising 3D representation, representing scenes with a collection of 3D Gaussians. Each Gaussian primitive has a series of attributes: the position \(\mathbf{p} \in \mathbb{R}^3\), the scaling vector \(\mathbf{s} \in \mathbb{R}^3\), the quaternion \(\mathbf{q} \in \mathbb{R}^4\) representing rotation, the opacity \(o \in \mathbb{R}\), and the color \(\mathbf{c} \in \mathbb{R}^3\). The Gaussian is then defined as \(\mathbf{G}(\mathbf{x}) = e^{-\frac{1}{2}(\mathbf{x}-\mathbf{p})^T \mathbf{\Sigma}^{-1} (\mathbf{x}-\mathbf{p})}\), where \(\mathbf{\Sigma} \in \mathbb{R}^{3\times 3}\) is the covariance matrix derived from \(\mathbf{s}\) and \(\mathbf{q}\). Using a fast tile-based rasterizer, 3DGS enables real-time rendering.

FLAME [17] is a parametric face model that represents human face deformation by a combination of blendshapes and linear blend skinning (LBS), with parameters describing shape \(\boldsymbol{\beta}\), pose \(\boldsymbol{\theta}\), and expression \(\boldsymbol{\psi}\). To deal with the deformation that LBS cannot model, additional blendshapes are applied to the template mesh vertices \(\overline{\mathbf{V}}\) as: \[\mathbf{V}_P=\overline{\mathbf{V}} + B_S(\boldsymbol{\beta};\mathcal{S})+ B_P(\boldsymbol{\theta};\mathcal{P})+ B_e(\boldsymbol{\psi};\mathcal{E}),\] where \(B_S(\boldsymbol{\beta};\mathcal{S})\), \(B_P(\boldsymbol{\theta};\mathcal{P})\) and \(B_e(\boldsymbol{\psi};\mathcal{E})\) represent shape, pose and expression blendshapes, respectively. Then, a standard skinning function \(W\) is utilized to produce the position of the deformed vertices \(\mathbf{V}\): \[\mathbf{V}=W(\mathbf{V}_P, \mathbf{J}(\boldsymbol{\beta}), \boldsymbol{\theta}, \mathcal{W}),\] where \(\mathbf{J}\) is the joint locations, and \(\mathcal{W}\) is the blendweights.

7 Implementation Details↩︎

We use the FLAME model [17] with the mouth cavity covered following [3] as the parametric head model. We generate UV Gaussian attribute maps at \(K^2 = 256^2\) resolution. To sample Gaussians from the UV Gaussian attribute maps, we use uniform grid sampling with a size of \(256\times 256\), sampling one Gaussian from each UV map texel, except for the hair region, where we use the uniform grid of size \(1024\times 128\) to make Gaussians cover the back of the head more completely. We also sample the UV coordinates corresponding to the FLAME vertices. The sampling results in about 78K Gaussians. We extract \(N=4\) scales of the source image features for local UV feature extraction. In the symmetric UV-space inpainting module, We use 2 layers of the transformer block with convolution layers instead of linear layers to maintain the spatial structure of the UV map. The local window size \(w\) is set to 7.

During training, we randomly sample 2 inverted novel views in 3D reconstruction mode, with a mode-switch probability of \(0.5\). One pseudo view has 0 elevation and azimuth uniformly sampled from \([0,360^\circ)\). The other is sampled from normal distributions azimuth \(\sim \mathcal{N}(0,(180^\circ)^2)\) and elevation \(\sim \mathcal{N}(0,(37^\circ) ^2)\). We use the Adam optimizer with a learning rate of \(1\times 10^{-4}\). The model is trained with a batch size of 4 for 37500 iterations on one NVIDIA H800 GPU. The loss weights are set as \(\lambda_{3d}=50\), \(\lambda_{eye}=5\), \(\lambda_{pos}=1\), \(\lambda_{shape}=1\), \(\lambda_{shape}^{tv}=10\), \(\lambda_{id}=0.25\), \(\lambda^{1}=0.5\), and the threshold is set to \(\epsilon=0.1\). We add \(\mathcal{L}_{3d}\), \(\mathcal{L}_{eye}\), \(\mathcal{L}_{shape}^{tv}\) and \(\mathcal{L}_{id}\) after training for 7500 iterations, and increase \(\lambda_{3d}\) to \(100\) after training for 25000 iterations.

8 Additional Details on Experiments↩︎

8.1 Additional Comparison Details↩︎

We compare our method with a group of open-source state-of-the-art methods, , the 2D-based StyleHEAT [37], the 3D-based GOHA [5], Real3DPortrait [7], Portrait4D [8], Portrait4D-v2 [9], GAGAvatar [12] and LAM [13]. For each approach, we follow the official data pre-processing and animation procedures, and use the released checkpoint for evaluation. To compute the metrics, we use the processed and generated frames for each method, as they are typically aligned and share the same background color. However, Portrait4D [8] and Portrait4D-v2 [9] generate misaligned images with their processed frames, so we further align these images before computing the metrics.

8.2 Additional Experimental Results↩︎

Video Demos. We provide video demos on the project page¹. These video demos present results for comparison with state-of-the-art approaches in terms of avatar animation and 360\(^\circ\) rendering. They demonstrate the effectiveness of our method, supporting high-quality animation and 360\(^\circ\) rendering views.

Additional 360\(^\circ\) Rendering Results. We compare the quality of 360\(^\circ\) rendering with the 3D-based methods in Fig. 9. The avatar is animated by the driving motion before being rendered to different viewpoints across 360\(^\circ\). All other methods fail at large side or back views that significantly differ from the input source view, unable to model the back of the head. Only GAGAvatar [12] can roughly model the shape of the back head at side views, but the textures are coarse and contain visible artifacts. Meanwhile, due to its use of a 2D neural renderer on the rendered features, GAGAvatar cannot maintain 3D consistency of the face area under large camera pose changes. In contrast, by inpainting UV-space source local features, our method reconstructs full-head avatars with plausible details for the back of the head, improving the realism of animatable 3D head avatars.

Self-reenactment. We provide qualitative comparisons of methods for self-reenactment in Fig. 10. Our method can faithfully preserve appearance details in the source image and imitate the driving motion at the same time.

User Study for Cross-identity Reenactment. We conduct a user study with 20 participants to compare 10 cross-identity reenactment results from our method with recent state-of-the-art approaches in terms of appearance preservation, motion replication, and overall quality. Tab. ¿tbl:reb:table:user95study? shows that users prefer our results, confirming our generation quality.

Analysis of Identity Consistency under Different Poses. We present a stratified quantitative evaluation for identity consistency on yaw ranges relative to the front view (0\(\sim\)​180\(^\circ\) covering the full 360\(^\circ\) range). Specifically, we evaluate on Ava-256 [67] for yaw angles of \(0\) to \(90^\circ\) and on images generated by SphereHead [19] for yaw angles of \(90^\circ+\) in Tab. ¿tbl:reb:table:stratified95finer?. For yaw angles beyond \(90^\circ\), we use LPIPS as CSIM becomes unreliable when faces are not detected. For Ava-256, we randomly select 8 identities with 7 views. For SphereHead, we generate 100 identities and evaluate at 2 random yaw angles in each stratified yaw angle range. The most frontal views serve as input. Our method excels at preserving identity for poses with yaw angles of \(30^\circ+\). We also verify identity consistency at large poses with ground-truths from Ava-256 in Fig. 11. Being limited by the GAN prior and the ambiguity of a single frontal view, the generated novel views lack hair details, but they still preserve convincing back heads and the source identity.

Temporal Consistency for Animation. We measure temporal consistency of the generated videos on VFHQ [15] using JOD [68] following Avat3r [53]. Tab. ¿tbl:reb:table:temporal? shows that our method achieves the best consistency.

Robustness Analysis. We perform a robustness analysis of our method under asymmetric hairstyles, asymmetric face alterations and extreme lighting conditions in Fig. 12. As our method mainly utilizes symmetry priors for the generation of occluded areas, it can preserve asymmetric hairstyles or face alterations that are visible in the input image. Our method is also robust under extreme lighting conditions.

More Ablation Study on 3D GAN Priors. To isolate the effects of global UV feature maps and pseudo multi-view data from 3D GAN priors, ablations on multi-view MEAD [62] are shown in Tab. 5 and Fig. 13. Global UV feature maps offer global face structures, resulting in consistent improvement, especially under occlusion (, lips in Fig. 13). Pseudo multi-view data supplies essential 360\(^\circ\) supervision; without it, significant side/back artifacts occur. Moreover, real multi-view data from NeRSemble [69] enhances animation quality, further improving side views when combined with 3D GAN priors.

Further Improving Visual Quality at Extreme Views. Fig. 9 confirms the strong visual quality of our method at extreme views. We further show that visual results can be improved by adjusting two factors of our method: (i) Using pseudo multi-view images from 3D GAN inversion as 3D supervision constrains head geometry, leading to a slightly pointed back head, as shown in Fig. 14. Fine-tuning on multi-view data generated by SphereHead [19] for 500 iterations alleviates this artifact. (ii) Overlarge Gaussians in the back head region cause blurriness. We mitigate this by rescaling Gaussians in the hair region by 0.5, which improves back head clarity. This leads to better reconstruction on back-view images generated by SphereHead in Tab. 15 and visibly sharper side/back views in Fig. [reb:fig:blurriness95extremeview].

Additional Motion Modeling and Evaluation. Our method uses FLAME for animation, which primarily captures motion-related deformation but fails to model motion-related appearance changes, such as wrinkles. As shown in Tab. ¿tbl:table:motion95module?, incorporating a lightweight motion module for expression-aware UV appearance further improves animation. Fig. 16 shows our method can handle challenging expressions, such as large smiles and frowns.

Comparison with 3D GAN Inversion. We compare with the 3D GAN inversion method [20] used in our framework for one-shot static head reconstruction in Tab. ¿tbl:reb:table:panohead95comparison? by reconstructing each frame of 17 videos from the VFHQ [15] test split. Our method better preserves the input identity and appearance details, which indicates the importance of inpainting input local UV feature maps rather than simply relying on the output of 3D GAN inversion.

Fairness of Using Pseudo Multi-View Data. We use pseudo multi-view data for novel-view supervision during training, following Real3DPortrait [7] and Portrait4D-v2 [9]. Specifically, we use pseudo multi-view data from 3D full-head GAN inversion [18], [20] to realize 360\(^\circ\) view synthesis. To make a fair comparison when such pseudo multi-view data is used, we also present results without using pseudo multi-view data in our method, labeled “Ours (w/o pseudo)" in Tab. ¿tbl:reb:table:pseudo95multi95view95data?. It generally yields better performance, likely because it is not affected by the inverted pseudo multi-view data, which fails to preserve input image details and identity as confirmed in Tab. ¿tbl:reb:table:panohead95comparison?. However, it cannot enable 360\(^\circ\) rendering.

9 Limitations↩︎

Despite the capability of our method to generate animatable 360\(^\circ\) avatars, it still has several limitations that can be addressed in the future. First, our method cannot reconstruct accessories well, such as hats and glasses. The reason is that our method highly depends on the pretrained 3D GAN and its inversion for global full-head feature extraction and novel view supervision, and the inversion method [20] is constrained by the latent space of the pretrained 3D GAN [18], which leads to noticeable artifacts when handling out-of-domain accessories, as shown in Fig. 17 A(b) and A(c). Besides, we obtain local UV feature maps by projecting points on the refined FLAME mesh to the 2D feature space, and the refined FLAME mesh cannot model glasses well, which makes our method treat glasses as textures on face surfaces, as shown in Fig. 17 A(c). Second, our performance on motion transfer is constrained by the estimated FLAME motion parameters. The limitation mainly comes from errors in the estimated parameters and the inability of FLAME to model certain motions, such as the tongue movement, as shown in Fig. 17 A(e) and A(f). Third, our symmetric assumption can fail if the asymmetric conditions in the source image are hidden, such as the ear and lighting in Fig. 17 B.

References↩︎

---

1. https://shaelynz.github.io/fhavatar/↩︎