1 Introduction↩︎

During daily conversation among humans, speakers naturally emit co-speech gestures to complement the verbal channels and express their thoughts [1]–[3]. Such non-verbal behaviors ease speech comprehension [4], [5] and bridge the communicator’s gap for better credibility [6], [7]. Therefore, equipping the social robot with conversation skills constitutes a crucial step to human-machine interaction. To achieve it, researchers delve into the task of co-speech gesture generation [8]–[10], where audio-coherent human gesture sequences are synthesized in the form of structural human representation (e.g., skeletons). However, such representation contains no appearance information of the target speaker, which is crucial for human perception. As demonstrated in audio-driven talking head synthesis [11], [12], generating real-world subjects in the image domain is highly desirable. To this end, we explore the problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio (illustrated in Fig. 1).

Conventional methods require exhaustive human efforts to pre-define the speech-gesture pairs and connection rules for coherent result [13], [14]. With the development of deep learning, neural networks are leveraged to learn the mapping from encoded audio feature to human skeletons in a data-driven manner [8]–[10]. Notably, one category of approaches relies on small-scale MoCap datasets in co-speech setting [15]–[17], which contributes to specific models with limited capacity and robustness. To capture more general speech-gesture correlations, another category of methods builds large training corpus by exploiting off-the-shelf pose estimators [18], [19] to label enormous online videos as pseudo ground truth [8], [20]. However, the inaccurate pose annotations induce error accumulation in the training phase, which makes the generated results unnatural. Besides, most previous works ignore the problem of co-speech gesture video generation. Only few works [8], [9] animate in the image domain as an independent post-processing step, which borrows from the existing pose-to-image generators [21], [22] to train on the target person’s images. How to design a unified framework to generate speaker image sequence driven by speech audio remains unsolved.

To effectively learn the mapping from audio to co-speech gesture video, we pinpoint two important observations from current studies: 1) hand-crafted structural human priors like 2D/3D skeletons would eliminate articulated human body region information. Such a zeroth-order motion representation fails to formulate first-order motion like local affine transformation in image animation [23]. Besides, the error in structural prior labeling impairs cross-modal audio-to-gesture learning [24]. 2) Motivated by previous linguistic studies [25], [26], the co-speech gestures could be decomposed into common motion patterns and rhythmic dynamics, where the former ones refer to large-scale motion templates (e.g., periodically put hands up and down), while the latter ones play a refinement role to complement subtle prosodic movements and synchronize with speech audio (e.g., finger flickers).

We take inspiration from the above observations and propose a novel framework Audio-driveN Gesture vIdeo gEneration (ANGIE) to generate co-speech gesture video. The key insight is to summarize common co-speech gesture patterns from motion representation to quantized codebooks and further refine subtle rhythmic details by motion residuals for fine-grained results. In particular, two modules are designed, namely VQ-Motion Extractor and Co-Speech GPT. In VQ-Motion Extractor, we utilize an unsupervised motion representation to depict the articulated human body and first-order gestures [27]. The codebooks are established to quantize the reusable common co-speech gesture patterns from unsupervised motion representation. To guarantee the validity of gesture patterns, we propose a cholesky decomposition based quantization scheme to relax the motion component constraint. The position-irrelevant motion pattern is extracted as final quantization target to represent the relative motion. In this way, the quantized codebooks naturally contain rich common gesture pattern information. With the quantized motion code sequence, in Co-Speech GPT we use a GPT-like [28] structure to predict discrete motion patterns from speech audio. Finally, a motion refinement network is used to complement subtle rhythmic details for fine-grained results.

To summarize, our main contributions are three-fold: 1) We explore a challenging problem of audio-driven co-speech gesture video generation. To the best of our knowledge, we are the first to generate co-speech gesture in image domain with a unified framework without any structural human body prior. 2) We propose the VQ-Motion Extractor to quantize the motion representation into common gesture patterns and the Co-Speech GPT to refine subtle rhythmic movement details. The codebooks naturally contain reusable motion pattern information. 3) Extensive experiments demonstrate that the proposed framework ANGIE renders realistic and vivid co-speech gesture video generation results.

2 Related Work↩︎

Co-Speech Gesture Generation. Synthesizing co-speech gesture has gained research interest in vision [8], [29], [30], graphics [10], [31], [32] and robotics [20], [33], [34] domains. Recent researches resort to deep neural networks to learn the speech-gesture mapping in a data-driven manner, with major focuses on below perspectives: 1) Dataset. One strand of methods use small-scale MoCap datasets to learn specific models [15]–[17], [35]–[38], while another strand of works exploit off-the-shelf estimator to label enormous videos as structural prior [8]–[10], [20], [24], [29], [39]–[41]. The dataset scale v.s. pose annotation accuracy often acts as a trade-off in this task: A large amount of speech-gesture pairs facilitate the training of more general models with better capacity and robustness, yet error accumulation in annotations induces unnatural results. 2) Framework architecture. CNN-based [8], RNN-based [10] and Transformer-based [42] frameworks show promising results. To further improve the diversity of generated gestures and grasp the fine-grained cross-modal associations, components like adversarial loss [8], VAE sampling [35], [43] and hierarchical encoder-decoder design [24] are proposed. 3) Input modality. Some approaches treat single modality of speech audio [8], [9], [33], [35], [40], [44], [45] or text transcription [20], [29], [34], [42] as input to drive the co-speech gesture, while some others use both modalities as stimuli for generation [10], [24], [39]. To ease the learning of implicit cross-modal mapping from speech to gesture and create more stable results, recent works involve auxiliary input modality such as speaker style [40], pose mode [43] and motion template [9].

In this work, we take a step further in the above three aspects: 1) For the dataset, we collect a new co-speech gesture dataset in image domain, where an unsupervised motion representation is used to model articulated human body and bypass the inaccuracy from structural prior annotations. 2) For the architecture, a vector quantized (VQ) network with novel discretization scheme is proposed to extract valid relative motion patterns. We further devise a motion refinement network to complement subtle rhythmic dynamics. 3) For the input modality, we explicitly decouple the common motion pattern from co-speech gestures, which serves a similar role as motion template [9] to provide auxiliary input. However, our discrete codebook design is more suitable for finite gesture patterns than continuous representation, which is also proven in recent cross-modal generation tasks [46]–[48]. Furthermore, we propose a novel vector quantization network with cholesky decomposition scheme to extract the valid motion patterns. We improve the quantization scheme to encode the relative motion representation that is position (absolute location) irrelevant. A motion refinement network is further devised to complement subtle rhythmic dynamics. Notably, our approach gives an idea on how to deal with the constraints in vector quantization and how to complement sequential results with missing details. Such design could prospectively provide insights for relevant domains like constrained vector quantization problem, cross-modal learning [49] and video generation tasks [48].

Video/Audio-Driven Video Generation. Traditional video-driven approaches for image animation can be categorized into supervised and unsupervised, where the supervised methods typically involve structural human body prior such as landmarks [50], [51] and 3D parametric models [52], [53], while the unsupervised approaches design self-supervised tasks to animate unlabeled images [23], [27], [54], [55]. To facilitate broader applications, researchers explore audio-driven video generation, where one of the most relevant tasks is talking face generation [56], [57]. Different from the strong correlations between audio and mouth shape, the mapping from audio to complicated co-speech motion is multi-modal and harder to learn. Most co-speech gesture studies synthesize human skeletons as final results (e.g., 2D keypoints), while only few works [8], [9] generate co-speech images in a post-processing manner.

3 Our Approach↩︎

We present ANGIE that generates audio-driven co-speech gesture in image domain, where the speakers’ image sequence is driven by speech audio as shown in Fig. 1. The whole pipeline is illustrated in Fig. 2. To make the content self-contained and narration clearer, we first introduce the preliminaries and problem setting in Sec. 3.1. Then, we present the VQ-Motion Extractor which extracts common co-speech gesture patterns as quantized codebooks in Sec. 3.2. Finally, we elaborate the Co-Speech GPT to complement subtle rhythmic dynamics for fine-grained results in Sec. 3.3.

3.1 Preliminaries and Problem Setting↩︎

Unsupervised Motion Representation. To achieve high-fidelity image animation, we take inspiration from MRAA [27] that uses an unsupervised motion representation to drive articulated objects. MRAA first estimates a coarse motion representation from the source and driving frames, then predicts dense pixel-wise flow for image generation. Specifically, an encoder-decoder keypoint predictor produces \(K\) different heatmaps \(\mathbf{H}^1, \mathbf{H}^2, \dots, \mathbf{H}^{K}\), where \(K\) is region number and \(\mathbf{H}^{k}\) denotes the \(k\)-th image region. Afterwards, each heatmap is normalized by softmax operation, i.e, \(\sum_{z\in \mathcal{Z}}\mathbf{H}^{k}(z) = 1\), where \(z\) is the image pixel location and \(\mathcal{Z}\) is the set of all pixels. The key insight behind MRAA is to represent each region’s motion by affine transformation with a shift-translation component. The shift-translation component \(\mu^{k} \in \mathbb{R}^{2}\) and the distribution \(C^{k}\) of the \(k\)-th part can be calculated as: \[\begin{align} \mu^{k} = \sum_{z\in \mathcal{Z}}\mathbf{H}^{k}(z)z, \quad C^{k} = \sum_{z\in \mathcal{Z}}\mathbf{H}^{k}(z)(z-\mu^{k})(z-\mu^{k})^{\mathsf{T}}, \label{eq:sc} \end{align}\tag{1}\] where \(\mathsf{T}\) is matrix transpose, \(C^{k} \in \mathbb{R}^{2 \times 2}\) measures the covariance of heatmap value. It naturally depicts the size and shape of an articulated region. To represent the affine transformation \(A^{k} \in \mathbb{R}^{2 \times 2}\) of the \(k\)-th region, we apply singular value decomposition (SVD) to \(C^{k}\) and derive \(A^{k}\) as: \[\begin{align} C^{k} = U^{k}\Sigma^{k}(V^{k})^{\mathsf{T}}, \quad A^{k} = U^{k}{\Sigma^{k}}^{\frac{1}{2}}, \label{eq:svd} \end{align}\tag{2}\] where unitary matrices \(U^{k}\), \(V^{k}\) and diagonal matrix \(\Sigma^{k}\) are the SVD result of covariance matrix \(C^{k}\). The representation \(\mathcal{M}\) extracted by motion estimator is the concatenation of \(\left[\mu; C; A\right] \in \mathbb{R}^{K \times (2 + 4 + 4)}\) for \(K\) distinct regions, an image generation module with dense pixel-wise flow predictor synthesizes the final generation results. In this work, the motion representation \(\mathcal{M}\) and image generation module \(G\) generally follow MRAA. We suggest the readers referring to [27] for more details.

Problem Setting for Co-Speech Gesture Image Generation. We collect training data of massively-available speaking videos with clear co-speech gestures for natural self-reconstruction supervision. Specifically, given an \((N+1)\)-frame video clip \(\mathbf{V} = \{I_{(0)}, \dots, I_{(N)}\}\), the goal of our framework at the training stage is to predict the motion representation \(\widehat{\mathcal{M}}_{(1:N)}\) based on the first image frame \(I_{(0)}\) and video’s accompanying audio sequence \(\mathbf{a} = \{a_{(1)}, \dots, a_{(N)}\}\). Further, the image generation module \(G\) reconstructs the video frames \(\widehat{I}_{(1:N)}\). At the inference stage, an arbitrary reference image with speech audio clip is provided to generate subsequent image frames. According to the observation in Sec. 1, we decompose the co-speech gestures into common motion patterns and subtle rhythmic dynamics. The overall training setting can be formulated as: \[\label{eq:formulation} \widehat{I}_{(1:N)} = G (I_{(0)}, \widehat{\mathcal{M}}_{(1:N)}(\mathbf{a})) ,\;\quad \;\; \widehat{\mathcal{M}}_{(1:N)} = \widehat{\mathcal{M}}^{\textit{pattern}}_{(1:N)} + \widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:N)},\tag{3}\] where \(\mathcal{M}^{\textit{pattern}}\) denotes the gesture pattern (Sec. 3.2) and \(\mathcal{M}^{\textit{rhythmic}}\) is rhythmic movement (Sec. 3.3).

3.2 Vector Quantized Motion Pattern Extractor↩︎

To decompose the co-speech gestures, we propose to firstly extract the common motion patterns. However, three problems remain: 1) The gesture sequences are different from each other. While some motion sequences share the same action pattern, the dynamic details may vary a lot. How to extract the major motion pattern despite the influence from minor prosodic movements? 2) The covariance matrix \(C\) is symmetric positive definite (Eq. 1 ), which further constrains the range of affine matrix \(A\). How to preserve such characteristic for valid gesture patterns? 3) Since the unsupervised motion representation is extracted in the image pixel space, it is affected by the absolute location of each articulated region. How to represent the position-irrelevant motion pattern information?

Vector Quantized Motion Pattern Learning. Our solution to the first problem is to quantize the common motion pattern into a codebook. Since the gesture pattern is finite, it could be summarized to discrete codebook entries. Besides, each codebook entry refers to a certain type of gesture pattern, which matches our goal to extract the common and reusable co-speech gesture patterns.

A naive way is to quantize the motion representation \(\mathcal{M}\) separately as shift-translation \(\mu\), covariance matrix \(C\) and affine transformation \(A\). Specifically, for a \(T\)-frame co-speech gesture sequence \(I_{(1:T)}\), we transform it into \(\left[\mu; C; A \right]_{(1:T)} \in \mathbb{R}^{T \times K \times (2 + 4 + 4)}\), where \(\left[\mu; C; A \right] = \mathcal{M}\) denotes motion representation of \(K\) regions as in Sec. 3.1. We first build three codebooks \(\mathcal{D}_{\mu} = \{\mathbf{d}_{ \mu, m}\}_{m=1}^{M}\), \(\mathcal{D}_{C} = \{\mathbf{d}_{ C, m}\}_{m=1}^{M}\) and \(\mathcal{D}_{A} = \{\mathbf{d}_{ A, m}\}_{m=1}^{M}\) for each motion component respectively, where \(M\) is codebook size. \(\mathbf{d}_{\mu, m}\), \(\mathbf{d}_{C, m}\) and \(\mathbf{d}_{A, m} \in \mathbb{R}^{\ell}\) are the \(m\)-th entry of \(\ell\)-channel for each codebook. Then, three separate encoders \(E_{\mu}\), \(E_{C}\) and \(E_{A}\) are utilized to encode the corresponding context information into latent features of \(\mathbf{e}_{\mu} = \{\mathbf{e}_{\mu, i}\}_{i=1}^{T'}\), \(\mathbf{e}_{C} = \{\mathbf{e}_{C, i}\}_{i=1}^{T'}\) and \(\mathbf{e}_{A} = \{\mathbf{e}_{A, i}\}_{i=1}^{T'} \in \mathbb{R}^{T' \times \ell}\), where \(T'\) is the temporal dimension and \(\ell\) is the channel dimension. Notably, we denote the \(i\)-th temporal feature of each motion component as \(\mathbf{e}_{\mu, i}\), \(\mathbf{e}_{C, i}\) and \(\mathbf{e}_{A, i}\). The feature encoding process can be formulated as: \[\begin{align} E_{\mu}(\mu_{(1:T)}) = \mathbf{e}_{\mu}, \quad E_{C}(C_{(1:T)}) = \mathbf{e}_{C}, \quad E_{A}(A_{(1:T)}) = \mathbf{e}_{A}. \label{eq:encode} \end{align}\tag{4}\] Following the pipeline of VQ-VAE [58], we individually quantize \(\mathbf{e}_{\mu}\), \(\mathbf{e}_{C}\) and \(\mathbf{e}_{A}\) by substituting each temporal feature \(\mathbf{e}_{\mu, i}\), \(\mathbf{e}_{C, i}\) and \(\mathbf{e}_{A, i}\) to the nearest codebook entry \(\mathbf{d}_{\mu, m}\), \(\mathbf{d}_{C, m}\) and \(\mathbf{d}_{A, m}\) as: \[\begin{align} \underbrace{\mathbf{e}^{\mathbf{q}}_{\mu} = \mathop{\arg\min}_{\mathbf{d}_{\mu} \in \mathcal{D}_{\mu}} || \mathbf{e}_{\mu} - \mathbf{d}_{\mu} ||}_{\textit{quantize shift-translation }\mu},\;\; \underbrace{\mathbf{e}^{\mathbf{q}}_{C} = \mathop{\arg\min}_{\mathbf{d}_{C} \in \mathcal{D}_{C}} || \mathbf{e}_{C} - \mathbf{d}_{C} ||}_{\textit{quantize covariance matrix }C},\;\; \underbrace{\mathbf{e}^{\mathbf{q}}_{A} = \mathop{\arg\min}_{\mathbf{d}_{A} \in \mathcal{D}_{A}} || \mathbf{e}_{A} - \mathbf{d}_{A} ||}_{\textit{quantize affine transformation }A}, \label{eq:quantizesca} \end{align}\tag{5}\] where \(\mathbf{e}^{\mathbf{q}}_{\mu} = \{\mathbf{e}^{\mathbf{q}}_{\mu, i}\}_{i=1}^{T'}\), \(\mathbf{e}^{\mathbf{q}}_{C} = \{\mathbf{e}^{\mathbf{q}}_{C, i}\}_{i=1}^{T'}\) and \(\mathbf{e}^{\mathbf{q}}_{A} = \{\mathbf{e}^{\mathbf{q}}_{A, i}\}_{i=1}^{T'} \in \mathbb{R}^{T' \times \ell}\) are the quantized code sequence of length \(T'\) for each motion component. The \(i\)-th quantized code of each motion component is denoted as \(\mathbf{e}^{\mathbf{q}}_{\mu, i}\), \(\mathbf{e}^{\mathbf{q}}_{C, i}\) and \(\mathbf{e}^{\mathbf{q}}_{A, i}\), respectively. Finally, three separate decoders \(D_{\mu}\), \(D_C\) and \(D_A\) are leveraged to reconstruct the motion representations of each component as: \[\begin{align} \widehat{\mu}_{(1:T)} = D_{\mu}(\mathbf{e}^{\mathbf{q}}_{\mu}), \quad \widehat{C}_{(1:T)} = D_{C}(\mathbf{e}^{\mathbf{q}}_{C}), \quad \widehat{A}_{(1:T)} = D_{A}(\mathbf{e}^{\mathbf{q}}_{A}). \label{eq:decode} \end{align}\tag{6}\]

Such discrete representation also ease the audio-to-gesture learning (Sec. 3.3): Previous methods predict continuous output as a harder regression problem. While we only need to predict features nearer to the correct codebook entry, which in essence resembles an easier classification problem.

Quantization Design for Valid Motion Representation. To extract valid gesture patterns, we have to preserve certain characteristics of motion representation. Especially, the covariance matrix \(C\) should be symmetric positive definite (Eq. 1 ), and the affine transformation \(A\) is determined by \(C\) through SVD (Eq. 2 ). Therefore, instead of naively quantize each component in Eq. 5 , we propose to only quantize the shift-translation \(\mu\) and covariance matrix \(C\), while derive the affine transformation \(A\) with SVD. The only constraint is to guarantee that the covariance matrix \(C\) is symmetric positive definite. To satisfy such requirement, we use the unique cholesky decomposition theorem [59]:

In this way, we turn to quantize the lower triangular matrix \(L = \begin{pmatrix} l_1 & 0 \\ l_2 & l_3 \end{pmatrix}\), where the constraint is much simpler as \(l_1, l_3 > 0\). The updated quantization scheme with cholesky decomposition is: \[\begin{align} \underbrace{\mathbf{e}^{\mathbf{q}}_{\mu} = \mathop{\arg\min}_{\mathbf{d}_{\mu} \in \mathcal{D}_{\mu}} || \mathbf{e}_{\mu} - \mathbf{d}_{\mu} ||}_{\textit{quantize shift-translation }\mu},\;\; \underbrace{\mathbf{e}^{\mathbf{q}}_{L} = \mathop{\arg\min}_{\mathbf{d}_{L} \in \mathcal{D}_{L}} || \mathbf{e}_{L} - \mathbf{d}_{L} ||}_{\textit{quantize the lower triangular matrix }L}, \label{eq:quantizesl} \end{align}\tag{7}\] where \(\mathbf{e}_{L}\), \(\mathbf{e}^{\mathbf{q}}_{L}\), \(\mathcal{D}_{L}\) and \(\mathbf{d}_{L}\) denote the encoded feature, quantized feature, codebook and codebook entry for factorial covariance \(L\), respectively. A simple transformation of \(l_{1, 3} = \texttt{ReLU}(l_{1, 3}) + \epsilon\) guarantees the diagonal entries to be positive, where \(\epsilon\) is a small positive number. The motion component \(C\) and \(A\) can be further obtained by \(LL^{\mathsf{T}}\) and SVD calculation, respectively.

Position-Irrelevant Motion Pattern. Another problem arises when we inspect the value of the motion representation: Since \(\mathcal{M}\) is extracted in the image pixel space, the object location will affect the element in \(\mathcal{M}\). For example, if a person poses the same gesture at different image regions, the motion component differs yet the underlying motion pattern remains the same. Thus we focus on a image location invariant motion pattern representation. In particular, due to the linear additiveness, the relative shift-translation \(\mu\) between adjacent frames can be represented as \(\Delta \mu_j = \mu_{j} - \mu_{j-1}\), and the relative change of the lower triangular matrix is \(\Delta L_j = L_{j} - L_{j-1}\) for any \(j=2, \dots, N\). Note that with the uniqueness of cholesky decomposition, \((L + \Delta L)\) corresponds to the sole covariance matrix \(C = (L + \Delta L)(L + \Delta L)^{\mathsf{T}}\), which further determines affine matrix \(A\) by SVD. In this way, the term \(\Delta L\) is sufficient to represent any relative affine transformation between two frames. We accordingly update the quantization scheme with position-irrelevant motion pattern representation as: \[\begin{align} \underbrace{\mathbf{e}^{\mathbf{q}}_{\Delta \mu} = \mathop{\arg\min}_{\mathbf{d}_{\Delta \mu} \in \mathcal{D}_{\Delta \mu}} || \mathbf{e}_{\Delta \mu} - \mathbf{d}_{\Delta \mu} ||}_{\textit{quantize relative shift-translation }\Delta \mu},\;\; \underbrace{\mathbf{e}^{\mathbf{q}}_{\Delta L} = \mathop{\arg\min}_{\mathbf{d}_{\Delta L} \in \mathcal{D}_{\Delta L}} || \mathbf{e}_{\Delta L} - \mathbf{d}_{\Delta L} ||}_{\textit{quantize relative lower triangular matrix change }\Delta L}, \label{eq:quantizesldiff} \end{align}\tag{8}\] where \(\mathbf{e}_{\{\Delta \mu, \Delta L\}}\), \(\mathbf{e}^{\mathbf{q}}_{\{\Delta \mu, \Delta L\}}\), \(\mathcal{D}_{\{\Delta \mu, \Delta L\}}\) and \(\mathbf{d}_{\{\Delta \mu, \Delta L\}}\) are the encoded feature, quantized feature, codebook and entry for the relative shift-translation \(\Delta \mu\) and factorial covariance \(\Delta L\), respectively.

Overall Quantized Motion Pattern Learning. With the position-irrelevant motion pattern, the codebook naturally contains reusable common co-speech gesture patterns \(\mathcal{M}^{\textit{pattern}}\). The encoders \(E_{\Delta \mu}\), \(E_{\Delta L}\) and the decoders \(D_{\Delta \mu}\), \(D_{\Delta \mu}\) are jointly learned with the codebooks \(\mathcal{D}_{\Delta \mu}\) and \(\mathcal{D}_{\Delta L}\) via: \[\begin{align} \mathcal{L}_{\textit{VQ}} = || \widehat{\Delta \mu} - \Delta \mu || &+ || \texttt{sg}[\mathbf{e}_{\Delta \mu}] - \mathbf{e}^{\mathbf{q}}_{\Delta \mu} || + \beta_1 || \mathbf{e}_{\Delta \mu} - \texttt{sg}[\mathbf{e}^{\mathbf{q}}_{\Delta \mu}] || + \nonumber \\ || \widehat{\Delta L} - \Delta L || &+ || \texttt{sg}[\mathbf{e}_{\Delta L}] - \mathbf{e}^{\mathbf{q}}_{\Delta L} || + \beta_2 || \mathbf{e}_{\Delta L} - \texttt{sg}[\mathbf{e}^{\mathbf{q}}_{\Delta L}] ||, \label{eq:vqloss} \end{align}\tag{9}\] where sg denotes the stop gradient operation, \(\beta_1\) and \(\beta_2\) are two weight balancing coefficients.

3.3 Co-Speech Gesture GPT with Motion Refinement↩︎

Co-Speech Gesture GPT Network. With the position-irrelevant motion pattern of valid quantization design, each co-speech gesture clip can be transformed into discrete representation. We then learn a co-speech gesture GPT network to map from speech audio \(\mathbf{a}_{(1:T)}\) to quantized code sequences \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (1:T')}\) and \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (1:T')}\). Specifically, we extract audio features \(\mathbf{a}^{\textit{onset}}_{(1:T')}\) with onset strength information, which is more suitable for cross-modal pattern learning [48], [60]. Then, a feature embedding layer with positional embedding is leveraged to obtain the tokens for audio onset features, quantized relative shift-translation and qunatized relative factorial covariance. Further, we encode cross-attention information with a series of transformer layers. Finally, followed by a linear transformation with softmax activation, the \(M\)-dimensional output denotes the probability of each quantization code at that time step. The whole co-speech gesture GPT is trained with cross-entropy loss \(\mathcal{L}_{\textit{CE}}\). Such design enables us to predict and sample future quantization code \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu}\) and \(\mathbf{e}^{\mathbf{q}}_{\Delta L}\) with speech audio.

Motion Refinement by Learning Residuals. Now that we can reconstruct the relative shift-translation \(\widehat{\Delta \mu}\) and factorial covariance change \(\widehat{\Delta L}\) by VQ-VAE decoding. Given \(\mu_1\) and \(L_1\) extracted from the initial image frame \(I_{(1)}\), the absolute shift-translation and factorial covariance for the \(j\)-th frame can be calculated as \(\widehat{\mu_j} = \mu_1 + \sum_{i=2}^{j} \widehat{\Delta \mu_i}\) and \(\widehat{L_j} = L_1 + \sum_{i=2}^{j} \widehat{\Delta L_i}\), respectively. However, since the quantized codebook is designed to only represent the large-scale common motion pattern information, while fine-grained rhythmic details are omitted. Therefore, we propose to refine the co-speech movements by learning residual terms. Concretely, we extract the audio mfcc features \(\mathbf{a}^{\textit{mfcc}}_{(1:T)}\) to encode more contextual audio cues for prosodic dynamics learning. Then a bi-directional LSTM is used to predict the per-frame motion representation residuals \(R\) to the main motion pattern result \(\widehat{\mu}_{(1:T)}\) and \(\widehat{L}_{(1:T)}\), i.e., \(\widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:T)} = \left[R(\widehat{\mu}_{(1:T)}; \mathbf{a}^{\textit{mfcc}}_{(1:T)}); R(\widehat{L}_{(1:T)}; \mathbf{a}^{\textit{mfcc}}_{(1:T)})\right]\). By adding residual terms, the overall co-speech gesture GPT with motion refinement learning can be formulated as: \[\label{eq:refine} \mathcal{L}_{\textit{Residual}} = || \mathcal{M}_{(1:T)} - \widehat{\mathcal{M}}_{(1:T)}(\mathbf{a}) ||, \textit{where } \widehat{\mathcal{M}}_{(1:T)}(\mathbf{a}) = \widehat{\mathcal{M}}^{\textit{pattern}}_{(1:T)}(\mathbf{a}^{\textit{onset}}) + \widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:T)}(\mathbf{a}^{\textit{mfcc}}).\tag{10}\] In this way, we capture both major gesture patterns and subtle rhythmic dynamics for vivid results.

4 Experiments↩︎

4.1 Experimental Settings↩︎

Dataset and Preprocessing. Pose, Audio, Transcript, Style (PATS) is a large-scale dataset of 25 speakers with aligned pose, audio and transcripts [8], [39], [40]. The training corpus contains 251 hours of data with around 84,000 intervals of mean length 10.7s. Notably, the PATS dataset contains three modalities of audio log-mel spectrograms, speech transcripts and per-frame skeletons labeled with OpenPose [18]. To bypass the error accumulation in pose annotation and facilitate co-speech gesture image generation task, we extend PATS with more features: 1) preprocessed image frames and 2) onset strength audio features which are more suitable for co-speech gesture pattern learning.

We conduct the experiments on four speakers’ co-speech video subsets, including Oliver, Seth, Kubinec and Jon. Concretely, 2D skeletons of the image frames are obtained by OpenPose [18] for baseline methods training. The frames are cropped to make the speaker locate at the image center. Since the time span of a meaningful co-speech gesture unit sequence ranges from 4s to 14s [25], [26], we trim invalid videos and finally obtain 1306, 990, 1294 and 1284 clips for four subsets, respectively. The overall mean clip length is 9.8s. We randomly split the segments into 90% for training and 10% for evaluation. The image frames are sampled at 25 fps and further resized to \(256 \times 256\).

Comparison Methods. We compare with recent SOTA works: 1) Speech to Gesture (S2G) [8], a GAN-based pipeline that maps audio to 2D keypoints with a U-Net; 2) Hierarchical Audio to Gesture (HA2G) [24] which captures the hierarchical associations between multi-level audio features and tree-like human skeletons; 3) Speech Drives Template (SDT) [9] which relieves the one-to-many mapping ambiguity by a set of continuous gesture template vectors; 4) Trimodal Context (TriCon) [10], a representative framework that considers the trimodal context of audio, text and speaker identity. Note that all methods could drive 2D human skeletons with speech audio. We train baselines on the PATS image dataset and tune the hyper-parameters by grid search for the best evaluation result. In particular, we also show direct evaluations on the Ground Truth (GT) skeletons for clearer comparison.

Implementation Details. We sample \(T=96\) frame clips with stride \(32\) for training. 1) For the VQ-Motion Extractor: the co-speech gesture pattern codebook size \(M\) for both relative shift-translation \(\Delta \mu\) and factorial covariance change \(\Delta L\) are set to \(512\). The encoders \(E_{\Delta \mu}\), \(E_{\Delta L}\) and the decoders \(D_{\Delta \mu}\), \(D_{\Delta \mu}\) are based on 1D-convolution structure. The channel dimension \(\ell\) of each codebook entry \(\mathbf{d}_{\Delta \mu}\), \(\mathbf{d}_{\Delta L}\) as well as the encoded latent features \(\mathbf{e}_{\Delta \mu}\), \(\mathbf{e}_{\Delta L}\) are \(512\), while the temporal dimension \(T'\) is set as \(T/8=12\) to encode contextual features with downsampling rate of \(8\). The \(\epsilon\) is set as \(1\times10^{-5}\) to guarantee the positiveness of diagonal entries in factorial covariance \(L\). The commit loss trade-offs in \(\mathcal{L}_{\textit{VQ}}\) are empirically set as \(\beta_1 = \beta_2 = 0.1\). We optimize the gesture pattern VQ-VAE with Adam optimizer [61] of learning rate \(3\times10^{-5}\). 2) For the Co-Speech GPT: the Transformer channel dimension is \(768\), and the attention layer is implemented in \(12\) heads with dropout probability of \(0.1\). The onset strength audio features \(\mathbf{a}^{\textit{onset}} \in \mathbb{R}^{426}\) are extracted by Librosa, while the audio mfcc features \(\mathbf{a}^{\textit{mfcc}} \in \mathbb{R}^{28 \times 12}\) are computed with the window size of \(10\) ms. During the GPT training, the \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (1:11)}\), \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (1:11)}\) are used as input while \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (2:12)}\), \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (2:12)}\) serve as supervision labels. 3) For the motion representation \(\mathcal{M}\) and image generator \(G\): we implement as MRAA [27] to use \(K=20\) regions. The motion estimator is pretrained for knowledge distillation. The overall framework is implemented in PyTorch [62] and trained on one 16G Tesla V100 GPU for three days.

4.2 Quantitative Evaluation↩︎

Evaluation Metrics. We adopt 1) Fréchet Gesture Distance (FGD) [10] to evaluate the distance between the real and synthetic gesture distribution. We train an auto-encoder on the PATS image dataset and use the encoder to compute fréchet distance between the real and synthetic gesture in feature space. We also use the 2) Beat Consistency Score (BC) and 3) Diversity (Div.) [24], [63] to account for the speech-motion alignment and the diversity among generated gestures. Specifically, BC is computed as the average temporal distance between each audio beat and its closest gesture beat, and Diversity indicates the difference of multiple audios’ corresponding gestures in the latent space. Note that since all metrics are skeleton-based, we downgrade our method to operate on skeleton data for fair comparison, i.e., we use VQ-VAE w/o cholesky scheme to create 2D skeletons for evaluation.

Evaluation Results. The results are reported in Table 1. It can be seen that the proposed ANGIE achieves the best evaluation results on most metrics. Since our method summarizes reusable co-speech gesture patterns into quantized codebooks and complements subtle rhythmic dynamics, we can cover richer gesture patterns and create diverse results. Note that HA2G [24] tends to generate over-expressive gestures with multi-level audio features, which makes their results on BC even better than the ground truth in some cases. Despite of this, we perform comparable to ground truth on BC metric with stable motion results, showing that we can generate audio-aligned gestures. Besides, we can find that both SDT [9] and ours perform better on FGD and Diversity metrics than other methods due to the explicit modeling of co-speech gesture patterns. However, since the gesture pattern is finite and discrete, our quantized codebook design is more suitable than continuous representation.

4.3 Qualitative Analysis↩︎

User Study. We further conduct a user study to reflect the quality of audio-driven gestures. Concretely, we sample 25 audio clips from the PATS image test set for all methods to generate skeleton results, then involve 18 participants for user study. The Mean Opinion Scores protocol is adopted, which requires the participants to rate three aspects: (1) Realness; (2) Synchrony; (3) Diversity. The rating scale is 1 to 5, with 1 being the poorest and 5 being the best. The results are reported in Table 2, where our method performs the best on all three aspects. Notably, with the help of motion pattern codebook, we achieve comparable diversity result to the ground truth, demonstrating that the quantization design helps to capture common gesture patterns.

Video Generation Results. With the Co-Speech GPT and motion refinement network, we could predict the future quantization code and complement rhythmic motion details based on audio features, which enables us to generate video results. As shown in Fig. 3, the synthesized image sequence results from left to right contain diverse and meaningful gestures that are aligned with speech audio.

Codebook Analysis for Co-Speech Gesture Pattern. We analyze the meaning of the quantized codebooks by asking the question: Does each codebook entry really correspond to a certain type of motion pattern? To answer this, we validate two things in Fig. 4: 1) Different entries represent diverse gesture patterns, hence driving the same image with different quantized codes leads to different gestures (left). 2) Each codebook entry denotes a fixed motion pattern, hence driving different images with the same quantized code show same motions (right). Please refer to demo video for more results.

4.4 Ablation Study↩︎

In this section, we present ablation study of two modules in our framework. Note that except for the skeleton-based metrics, we also use video reconstruction accuracy as a proxy for image quality, including \(\mathcal{L}_1\) and perceptual loss [64] between the reconstructed and GT image; Average Euclidean Distance (AED) evaluates identity preservation by pretrained re-identification networks [27], [65].

VQ-Motion Extractor. We conduct ablation experiments on five settings: 1) w/o Quantization, where we directly infer motion representation with audio; 2) Quantize \(\mu\), \(L\); 3) Quantize \(\Delta \mu\), \(L\); 4) Quantize \(\mu\), \(\Delta L\), where we quantize the absolute or relative difference of the motion components; 5) w/o Cholesky Decomposition, which means the covariance matrix is not guaranteed to be symmetric positive definite. The results are shown in Table 3, which proves that the quantization design with position-irrelevant relative motion pattern could improve the performance. We also find that quantizing \(\Delta \mu\) is more effective than \(\Delta L\), since the shift translation is more correlated with region position. Note that when motion component is invalid, the pipeline fails to generate image due to the numerical instability in calculating affine matrix. Hence the results for the setting 5 are not reported.

Motion Refinement Module. To verify the efficacy of motion residuals, we eliminate the motion refinement for ablation. The results in Table 3 suggest that this module improves beat consistency by capturing subtle rhythmic movements, while the improvement on FGD majorly derives from the quantization design. Such module complements the motion pattern for fine-grained results.

5 Discussion↩︎

Conclusion. In this paper, we propose a novel framework ANGIE to generate audio-driven co-speech gesture video in the image domain. To summarize valid common co-speech gesture patterns, we propose VQ-Motion Extractor with cholesky decomposition based quantization scheme and position-irrelevant design to represent relative motion patterns. Then we propose Co-Speech GPT to refine subtle rhythmic movement details for fine-grained results. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video generation results.

Ethical Consideration. Generating co-speech gesture images facilitates applications such as digital human. However, it could be misused for malicious purposes like forgery generation. We believe that the proper use of this technique will enhance the machine learning research and digital entertainment.

Limitation and Future Work. As an early work towards audio-driven co-speech video generation without structural prior, we notice that our method fails for some challenging cases. For example, if the source image is in a large pose, it is hard to generalize well in such an out-of-domain data. We will explore how to develop a model with higher generalization ability in future work.

6 Acknowledgment↩︎

This work is in part supported by GRF 14205719, TRS T41-603/20-R, Centre for Perceptual and Interactive Intelligence, and CUHK Interdisciplinary AI Research Institute; and in part supported by NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033), and under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

In the supplemental document, we will introduce below contents: 1) proof of Theorem 1 (unique cholesky decomposition theorem) (Sec. 7); 2) more details about Co-Speech Gesture GPT (Sec. 8); 3) model architecture details (Sec. 9); 4) analysis on the limitations and future work (Sec. 10); 5) analysis on the model’s generalization ability (Sec. 11); 6) the standard deviation of use study results (Sec. 12); 7) the details of audio feature extraction (Sec. 13); 8) analysis on extra input modality of text information (Sec. 14); 9) experimental results with error bars (Sec. 15); 10) user study details (Sec. 16); 11) ethical considerations with possible prevention measures against potential negative societal impact (Sec. 17); 12) discussions on the dataset consent and personally identifiable information (Sec. 18); 13) the licenses of existing assets involved in this paper (Sec. 19);

7 Proof of Theorem 1↩︎

In the main paper, to ease the constraint in the quantization process, we use the unique cholesky decomposition theorem [59] to transform the covariance matrix \(C\) to factorial covariance \(L\) by theorem:

Proof. As \(C\) is a symmetric positive definite matrix, it can be represented as \(C=PDP^{\mathsf{T}}\), where \(D\) is a diagonal matrix with positive diagonal entries and \(P\) is a unitary matrix. Suppose \(B^{\mathsf{T}}=D^{\frac{1}{2}}P^{\mathsf{T}}\), it can be factorized as the product of a unitary matrix \(Q\) and an upper triangular matrix \(R\) (according to Matrix QR Decomposition), i.e., \(B^\mathsf{T}=QR\). Thus, we have \[C=PDP^{\mathsf{T}}=BB^\mathsf{T}=R^\mathsf{T}Q^{\mathsf{T}}QR=R^\mathsf{T}R=LL^\mathsf{T},\] where \(L=R^{\mathsf{T}}\) is a lower triangular matrix. Because \(C\) has the positive determinants, the diagonal entries of \(L\) should be non-zero values. Therefore, there exists a lower triangular matrix \(L\) with positive diagonal entries such that \(C=LL^\mathsf{T}\).

Assume \(C=L_1L_1^{\mathsf{T}}=L_2L_2^\mathsf{T}\) where \(L_1, L_2\) are two distinct lower triangular matrices with diagonal positive entries, it shows that \(I=L_1^{-1}L_2L_2^\mathsf{T}L_1^{-\mathsf{T}}=(L_1^{-1}L_2)(L_1^{-1}L_2)^\mathsf{T}\) where \(I\) is the identity matrix. It means that \(L_1^{-1}L_2=(L_1^{-1}L_2)^{-\mathsf{T}}\) which further shows that \(L_1^{-1}L_2\) should be a diagonal matrix. Because \(I=(L_1^{-1}L_2)^2\), it demonstrates that \(L_1^{-1}L_2\) is a diagonal matrix with \(\pm1\) entries. Since both \(L_1, L_2\) have positive diagonal entries, it means \(L_1^{-1}L_2=I\). Therefore, \(L_1=L_2\). 0◻

8 More Details about Co-Speech Gesture GPT↩︎

The target of the Co-Speech Gesture GPT is to predict the quantized motion codes from the audio onset features \(\mathbf{a}^{\textit{onset}}\). With the proposed vector quantization scheme of position-irrelevant representation, we could transform the raw relative shift-translation \(\Delta \mu_{(1:T)}\) and relative factorial covariance change \(\Delta L_{(1:T)}\) into corresponding quantized code sequence \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (1:T')}\) and \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (1:T')}\). In this way, any co-speech gesture sequence can be represented by a sequence of quantized motion codes. The original continuous audio-to-gesture prediction task is reframed as to select the proper motion code from the codebooks \(\mathcal{D}_{\Delta \mu}\) and \(\mathcal{D}_{\Delta L}\) according to the previous motion codes and current speech audio features. The output of the GPT [28] model at the \(t\)-th time step is the probability of choosing each codebook entry, where the entry with the largest probability serves as the predicted motion code of the next time step.

In particular, given the quantized code sequence \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (1:T')} \in \mathbb{R}^{T'}\) and \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (1:T')} \in \mathbb{R}^{T'}\) of temporal length \(T'\), two embedding layers map them to learnable features \(\mathbf{f}_{\Delta \mu} \in \mathbb{R}^{T' \times \ell}\) and \(\mathbf{f}_{\Delta L} \in \mathbb{R}^{T' \times \ell}\), respectively. The onset audio features \(\mathbf{a}^{\textit{onset}}\) are also passed through an embedding layer to obtain audio features \(\mathbf{f}_{a} \in \mathbb{R}^{T' \times \ell}\). After the concatenation over the temporal dimension and supplement the position embedding information, we get the latent input features of shape \((3 \times T') \times \ell\). Then, \(12\) Transformer layers with channel dimension of \(768\) and \(12\) attention heads of dropout probability \(0.1\) are utilized to encode the cross-attention information. Finally, a linear and softmax layer map the encoded latent features to normalized motion probabilities of \(\mathbf{p} \in \mathbb{R}^{(3 \times T') \times M}\), where \(M\) is the codebook size. Notably, \(\mathbf{p}_{t, m}\) denote the probability of motion code \(\mathbf{d}_{\Delta \mu, m}\) and \(\mathbf{d}_{\Delta L, m}\) predicted at the time step \(t + 1\) for relative shift-translation \(\Delta \mu\) and relative factorial covariance change \(\Delta L\), respectively. During the implementation, the probabilities at the audio feature indexes \(\mathbf{p}_{(1:T')}\) are discarded, while the probabilities at the \(\Delta \mu\) and \(\Delta L\) indexes represent the corresponding motion probabilities, i.e., \(\mathbf{p}_{\Delta \mu} = \mathbf{p}_{(T' + 1: 2T')}\) and \(\mathbf{p}_{\Delta L} = \mathbf{p}_{(2T' + 1: 3T')}\) [48]. The attention layer can be formulated as: \[\begin{align} \text{Attention}(\mathbf{Q, K, V, M}) = \text{softmax}(\frac{\mathbf{QK^\mathsf{T}} + \mathbf{M}}{\sqrt{\ell}})\mathbf{V}, \label{eq:attention} \end{align}\tag{11}\] where \(\mathbf{Q, K, V}\) are the query, key and value matrix from input, \(\mathbf{M}\) denote the mask, \(\ell\) is the channel dimension and \(\mathsf{T}\) is the matrix transpose operation. To encode the cross-conditional features of audio features, relative shift-translation features and relative covariance change features, the mask \(\mathbf{M}\) is set as a \(3 \times 3\) repeated block matrix with the lower triangular matrix as its element.

The Co-Speech GPT is trained via cross-entropy loss as a classification problem. Specifically, the motion codes of \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (1:11)}\), \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (1:11)}\) are used as input, while \(\mathbf{e}^{\mathbf{q}}_{\Delta \mu, (2:12)}\), \(\mathbf{e}^{\mathbf{q}}_{\Delta L, (2:12)}\) serve as supervision labels. The overall loss function is: \[\begin{align} \mathcal{L}_{\textit{CE}} = \frac{1}{T'} \sum_{t=1}^{T'} \left\{\texttt{CrossEntropy}\left[\mathbf{p}_{(\Delta \mu, t)}, \mathbf{e}^{\mathbf{q}}_{(\Delta \mu, t + 1)}\right] + \texttt{CrossEntropy}\left[\mathbf{p}_{(\Delta L, t)}, \mathbf{e}^{\mathbf{q}}_{(\Delta L, t + 1)}\right]\right\}, \label{eq:celoss} \end{align}\tag{12}\] where CrossEntropy denotes the cross-entropy loss function; \(\mathbf{p}_{(\Delta \mu, t)}\), \(\mathbf{p}_{(\Delta L, t)}\) are the predicted probability of \(\Delta \mu\), \(\Delta L\) terms at the \(t\)-th time step, respectively; the \(\mathbf{e}^{\mathbf{q}}_{(\Delta \mu, t + 1)}\), \(\mathbf{e}^{\mathbf{q}}_{(\Delta L, t + 1)}\) are the vector quantized code of \(\Delta \mu\) and \(\Delta L\) at the \((t+1)\)-th time step, respectively.

9 Architecture Details↩︎

VQ-Motion Extractor Encoder and Decoder. The encoders \(E_{\Delta \mu}\), \(E_{\Delta L}\) and the decoders \(D_{\Delta \mu}\), \(D_{\Delta \mu}\) are 1D temporal convolution networks. In particular, the encoders have three convolution layers, with each layer’s kernel size of \(3\), padding of \(1\) and stride of \(2\). In this way, the downsampling scale of the encoder is \(8\), which will compress the temporal contextual information. For the decoder, the structure is similar, with three convolution layers of kernel size \(3\), padding \(1\) and stride \(1\), and three upsample layers of upsample scale \(2\) to reconstruct the latent features.

Motion Estimator and Image Generator. For the motion estimator, we follow MRAA [27] that uses a region predictor to extract the articulated human body information. Specifically, we similarly use the U-Net architecture with five “convolution - batch norm - ReLU - pooling” blocks in the encoder and five “upsample - convolution - batch norm - ReLU” blocks in the decoder for both the region predictor and pixel-wise flow predictor. Note that since it is inappropriate to infer the background motion from speech audio cues, we remove the original background motion predictor. In terms of the image generator, it is based on the Johnson architecture [64], with two down-sampling blocks, six residual-blocks, and two up-sampling blocks. The skip connections are added to warp and weight the confidence maps.

Motion Refinement Network. The motion refinement network maps the audio mfcc features to the residual terms to complement the subtle rhythmic movements of the co-speech gesture. Concretely, the audio mfcc features \(\mathbf{a}^{\textit{mfcc}} \in \mathbb{R}^{28 \times 12}\) are first pre-extracted with the window size of \(10\) ms. Then a convolution based audio encoder with a series of linear layers is used to encode the per-frame audio features into \(\mathbf{f}_{a}^{\textit{mfcc}} \in \mathbb{R}^{128}\). The detailed structure is listed in Table 4.

We first follow the pipeline of MRAA [27] to pretrain the motion estimator and image generator via self-reconstruction. Then, the motion estimator module is freezed and serves as supervision in VQ-Motion Estimator training, i.e., we train the VQ-VAE model to reconstruct the motion representation from pretrained MRAA motion estimator.

10 Analysis on Limitations and Future Work↩︎

As an early work towards audio-driven co-speech video generation without structural prior, we notice that our method fails for some challenging cases. For example, if the source image is in a large pose, it is hard to generalize well in such out-of-domain data.

Notably, since our work does not involve any structural human body prior, it is of high freedom without any human skeleton physical constraint. Therefore, for some cases of large pose, the model may generate too large motion field and lead to failure case results. We will explore how to develop a model with higher generalization ability in future work.

Generalizing co-speech gesture avatars to more complex and general settings like social conversation is a promising idea of great practical usage. Currently, the biggest bottleneck could be the lack of high-quality conversational dataset. Although CMU Panoptic contains the multi-view conversation videos, the image quality is poor for co-speech image animation. Besides, since the social co-speech gesture is more diverse, some model designs like VQ codebook size should be well studied. We will delve into this interesting problem in future work.

Besides, since our frames are preprocessed to resolution of \(256 \times 256\), the scale and resolution of facial regions are rather small. We leverage Wav2Lip [57] to sync the lip shape in the demo video for better visualization. Note that such process substantially differs from the post-processing step of [8], [9] that trains an extra pose2image generator: 1) The focus of co-speech gesture generation task is to synthesize coherent upper body poses that are aligned to speech audio, while the lip-sync is not the focus in this work. We merely implement it for better visualization. 2) We could directly use the off-the-shelf Wav2Lip [57] to sync the lip shape without any preprocessing step, post-processing step or re-training. Therefore, it is in essence different from an additional pose2image generator that requires tedious re-training. 3) The task of simultaneously generating audio-consistent upper body gestures and lip shapes is challenging and remains unsolved. The difficulty lies in a) the low-resolution of mouth regions in the large-scale upper body image makes it hard to capture the fine-grained lip-sync information; b) the mouth shape only depends on the phoneme, while the co-speech gesture correlates to the semantic meanings. We will explore this problem in future work.

11 Analysis on Generalization Ability↩︎

As shown in previous studies [8], [40], the co-speech gesture motions and styles vary a lot for different speakers, which is termed as “individual speaking style” [8]. Therefore, it is suitable to train a separate co-speech gesture generation model for each person following the experiment settings of baselines [8], [9], [40]. However, we explore a more challenging task of co-speech gesture video generation in a unified framework without structural prior and achieve superior performance. Even for a single-person subset, it is non-trivial to animate non-rigid human body in image space by speech audio, especially with the interference of complex background scenes.

It is hard to generalize to speakers that are not in the training data with currently available co-speech gesture image datasets. In particular, the commonly used datasets are TED Gesture [10] and PATS [8], [40]. TED Gesture is based on TED Talk videos, while PATS contains 25 speakers of talk shows, lectures, etc. Due to the frequent camera movements and viewpoint shift in TED videos, there lacks clear co-speech gesture clips for image generation. Hence we narrow down the experiments to PATS dataset in this work. A dataset with high-quality co-speech gesture image frames of multiple speakers is needed to learn a model of novel (unseen) person generalization ability. We will strive for this in future work.

We verify the potential generalization ability of our approach: a) We could animate different appearances of a speaker with speech audio (as shown in codebook analysis part of the demo video, we could animate Oliver’s different appearances), while previous studies [8], [9] that resort to off-the-shelf pose2img generator only support a single appearance. b) We additionally implement the experiments of animating with novel (unseen) audio. The evaluated results are reported in Table 5. It shows that the model’s performance is still effective with the input of unseen audio. With the proposed vector quantize design, each codebook entry corresponds to a reasonable co-speech gesture pattern. In contrast to directly mapping to the continuous coordinate space, such technical design guarantees the valid gesture even when generalizing to the unseen audio.

12 The Standard Deviation of User Study↩︎

We additionally provide the standard deviation of user study results in Table 6, which shows that the agreement among participants is highly consistent.

13 Details on Audio Feature Extraction↩︎

Audio Onset Strength Feature. The audio onset strength feature of \(T\) frames is of shape \((T, 426)\), where \(T\) is the temporal dimension (frame number) and \(426\) is the feature channel dimension. It is the concatenation of constant-Q chromagram, tempogram, onset beat, onset tempo and onset strength. Most features are derived from the audio onset strength/envelope and the channel dimension is summed up to \(426\). We utilize the librosa onset functions to extract the features. The audio sample rate is \(16000\), the time lag for computing differences is \(1\), the hop length is \(512\) and the window length is \(384\).

Audio MFCC Feature. The original audio mfcc was calculated as \(12\) mfccs which has the dimension of \((T', 12)\) where \(T'\) is the original audio frame number. In our implementation, we use a \(28\)-dim sliding window to further unfold the mfcc feature into a final shape of \((T, 28, 12)\), where \(T\) denotes the final temporal dimension (video frame number), \(28\) denotes the size of sliding window and \(12\) is the mfcc feature dimension. MFCC feature is extracted with a sample rate of \(16000\), window length of \(25\) ms, window step of \(10\) ms, cepstrum number of \(13\), filters number of \(26\) and FFT block size of \(512\). In the motion refinement module, we use the a certain frame’s mfcc feature of shape \((28, 12)\) and forward a series of convolution and linear layers to extract the per-frame audio feature of dimension \(128\).

14 Analysis on Extra Input Modality of Text Information↩︎

Our main focus and technical contributions are how to animate the co-speech gesture in the image domain, but not on the influence of input modality. We choose the audio-driven setting mainly for two reasons: a) We generally follow the problem setting of baselines [8], [9], [40], where all the compared methods take only the speech audio as input modality for fair comparison. b) We have to pre-process the raw transcripts to be temporally aligned with audio using tools like Gentle. To prevent the potential alignment inaccuracy in the pre-processing step, and to simplify the problem setting for better focusing on audio-driven co-speech gesture image animation, we do not involve the text input in this work.

As proved in Automatic Speech Recognition (ASR) [66], [67] and recent co-speech gesture studies [24], the speech audio actually contains some high-level semantic information. Such implicit semantic information in the speech audio could guide the model to capture some specific co-speech gesture patterns like metaphorics [24]. Besides, as shown in the Kubinec chemistry lecture setting, our model manages to synthesize deictic gestures of pointing to the screen by learning such implicit audio-gesture correlations. We would like to respectfully claim that the generated gestures of our model are not only limited to the beat gestures, but some semantic gestures could also be synthesized.

Typically, language model/text information contains rich semantics, which are beneficial to the learning of semantics relevant gestures like iconics, metaphorics, and deictics. Previous study [10] has also verified the influence of each input modality on co-speech gesture, including audio, text and speaker identity, etc. Therefore, we additionally complement an experiment of using text feature: we encode the Gentle aligned transcripts by TextTCN [68] and further concatenate with audio features. The combined audio text features are fed to Co-Speech GPT with Motion Refinement network to predict the quantized code as well as motion residuals. The results are reported in Table 7, which suggest that the text feature could indeed facilitate the co-speech gesture generation.

As an early attempt exploring audio-driven co-speech video generation, this work could serve as a baseline for further studies in the research community. However, how to effectively fuse the multiple modality information (including audio and text) and better map to the implicit motion representation remains an open problem. We will explore this in future work.

15 Experimental Results with Error Bars↩︎

Since the Diversity metric involves randomness, which may fluctuate due to different sampling clips. Therefore, we randomly sample \(400\) pairs to get robust evaluation results. To verify the steadiness, we conduct the evaluation \(5\) times (create random samples \(5\) times with different random seeds). The results are listed in Table 8. We can see that the fluctuation is rather small, which shows the robustness of such evaluation.

16 User Study Details↩︎

The study involves \(18\) participants of \(9\) females and \(9\) males. In particular, the users are unaware of which generated result corresponds to which method for a fair comparison. Before they rate the quality of synthesized results, the participants will first be shown what the Ground Truth (original raw video) is to help them make accurate scoring. The participants are asked to judge the three perspectives of the generated portraits: (1) Realness; (2) Synchrony; (3) Diversity. All participants are paid for \(15\) USD for about \(40-60\) minutes’ rating process. We use the Fleiss’s-Kappa statistic to measure the participants’ scoring disagreement, which is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items.

17 Ethical Considerations↩︎

Our method could synthesize co-speech gesture images, which is envisioned to facilitate extensive applications like digital human and human-machine interaction. On the other hand, however, such techniques could be misused for malicious purposes such as forgery generation. As part of our responsibility, we strongly advocate all safeguarding measures against malicious use of co-speech gesture images and feel obliged to share our generated results with the deepfake detection community to improve the method’s robustness. We believe that the proper use of this technique will enhance positive societal development in both machine learning research and human’s daily entertainment.

18 Discussions on Dataset Consent and Information↩︎

The PATS Image Dataset is collected and preprocessed based on the PATS dataset. We download the YouTube videos by the provided video links, shared under the “CC BY - NC - ND4.0 International” license. This license allows for non-commercial use. Since all our code, data, models and results are kept for academic use only, hence despite of the contained personally identifiable information in the collected data, we will use them for research use only and will not allow any commercial use.

19 Assets License↩︎

In the Table 9, we list the licenses of all the existing assets we have used in this work.

1mm

References↩︎