CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos
January 15, 2026
Abstract
In this paper, we find that the generation of 3D human motions and 2D human videos is intrinsically coupled. 3D motions provide the structural prior for plausibility and consistency in videos, while pre-trained video models offer strong generalization capabilities for motions. Based on this, we present CoMoVi, a co-generative framework that generates 3D human motions and videos synchronously within a single diffusion denoising loop. However, since the 3D human motions and the 2D human-centric videos have a modality gap between each other, we propose to project the 3D human motion into an effective 2D human motion representation that effectively aligns with the 2D videos. Then, we design a dual-branch diffusion model to couple human motion and the video generation process with mutual feature interaction and 3D-2D cross attentions. To train and evaluate our model, we curate CoMoVi-Dataset, a large-scale real-world human video dataset with text and motion annotations, covering diverse and challenging human motions. Extensive experiments demonstrate that our method generates high-quality 3D human motion with a better generalization ability and that our method can generate high-quality human-centric videos without external motion references. Project page: https://igl-hkust.github.io/CoMoVi/.
1 Introduction↩︎
The generation of 3D human motion and realistic video sequences is essential to understanding human behaviors and coherent visual dynamics, with broad downstream applications including character animation, VR/AR, and gaming.
Traditional text-driven motion generation models learn to generate 3D human motion given textual descriptions [1]–[12]. However, these approaches are often constrained by the bottleneck of high-quality 3D motion data, which leads to limited generalization capabilities and low prompt fidelity of these methods. Recently, advanced approaches [13]–[15] attempt to overcome these limitations by first generates human videos and then applying video-based motion capture algorithms to recover 3D human motion. While video generation models generalize well, they often struggle with highly structured objects such as human bodies, frequently producing implausible motions with inconsistent body structure, which in turn corrupts the recovered 3D motion.
On the other hand, although recent Video Diffusion Models (VDMs) [16]–[20] demonstrate remarkable performance and strong generalization capabilities, the generation of a video about a specific person performing a particular action with high fidelity remains challenging. Previous human-centric video generation works introduce such priors by pre-extracting 2D pose or 3D motion as driving signals to guide the video generation process [21]–[29], achieving promising results with high quality. However, these methods all require external videos and motions as reference to provide such a guidance, and it remains unexplored how to generate high-quality video without such external video and motion references.
Through our discussion above, it is revealed that there exists a strong coupling relationship between 3D human motion and video generation. High-quality 3D motion can derive high-fidelity generated videos, and, conversely, the powerful prior of VDMs can enhance the generalization capabilities of 3D motion generation. However, as illustrated in Fig. 2, existing works are cascaded, either in a motion-to-video or a video-to-motion framework, which are suboptimal. In this paper, we introduce CoMoVi, a novel framework for the co-generation of 3D human motion and human video synchronously. This co-generative framework allows mutual information exchange between the motion and video generation processes, enabling generalization enhancement for motion generation and structure guidance for video generation.
To achieve this synchronous co-generation, we first need to address the inherent modality gap between 3D human motion and 2D video. A natural approach to bridging this gap is to render 3D human motion into 2D feature maps, which allows for the synchronous generation of both motion and video utilizing VDMs. However, effectively formulating this 2D representation remains a critical challenge. Utilizing normal maps as the representation struggles to disambiguate symmetrical or overlapping body parts (e.g., distinguishing the left hand from the right) and lacks explicit structural semantics. Conversely, semantic maps provide part-level awareness, but inherently discard crucial 3D geometric information. To overcome these limitations, we propose a novel 2D human motion representation that successfully encapsulates both rich semantic distinctions and precise 3D structural geometry into the pixel space. As demonstrated in our experiments, this unified representation is pivotal to effectively aligning the two modalities and serves as the cornerstone of our co-generation framework.
Building upon this representation, we design a unified pipeline that takes an input image and a text description to synchronously generate human motions and the corresponding video sequences within a single diffusion denoising loop. Specifically, we develop a dual-branch VDM extended from Wan2.2-I2V-5B [19]. This architecture tightly couples the denoising processes of 2D motion videos and RGB videos through mutual feature interactions, ensuring that the video generation is explicitly guided by robust human motion priors and that the motion generation shares the generalization ability of video generation. Furthermore, instead of solving for a 3D motion from our generated 2D motion representations with optimization, we design a module to combine the features from both videos and 2D motion maps for direct 3D motion estimation. Thus, CoMoVi can directly output both 3D human motions and 2D human-centric videos.
Finally, effective training and evaluation of our co-generation framework inherently demand a dataset that simultaneously provides comprehensive text descriptions, high-quality video sequences, and precise 3D motion annotations. Existing datasets often fall short of these stringent requirements: while large-scale datasets like Motion-X++ [30] suffer from compromised video quality and low resolution, high-fidelity datasets such as HumanVid [31] are limited in scale and rich 3D annotations, with less than 20k real-world videos. To bridge this critical data gap, we contribute the CoMoVi-Dataset. Leveraging a meticulously designed data processing pipeline, we curate a collection of approximately 50K high-resolution, real-world human videos. Our dataset significantly surpasses HumanVid [31] in scale while offering vastly superior video quality compared to Motion-X++ [30]. Furnished with reliable text and motion annotations, the CoMoVi-Dataset covers a diverse array of challenging human dynamics, serving as a robust foundation for our model and future research in this domain.
To demonstrate the effectiveness of CoMoVi, we conduct comprehensive experiments on the Motion-X++ dataset [30], VBench benchmark [32]–[34], and our CoMoVi-Dataset for both motion generation and video generation tasks. Qualitative and quantitative results validate that our approach is effective in generalizable 3D human motion generation, outperforming state-of-the-art (SoTA) text-to-motion (T2M). For human-centric video generation, CoMoVi achieves high-quality video generation without relying on external reference videos and pre-extracted motion signals.
2 Related Works↩︎
2.0.0.1 Text-driven Human Motion Synthesis.
Along with the emergence of large-scale human motion datasets with natural language labels [12], [30], [35]–[51], T2M methods step a lot forward in both diffusion [1]–[6] and autoregressive directions [7]–[11], [52]. However, the scarcity of high-quality 3D motion data constraints their diversity and generalization capabilities. To overcome this limitation, recent works leverage multi-view diffusion models [53]–[56] to generate 2D motion sequences and then triangulate to 3D [13], [14]. Though effective, these approaches separate 2D generation and 3D reconstruction into two independent processes and only consider human motion as 2D joint coordinates, neglecting the coupling relationship between 3D motion and 2D frames. Our work proposes to encode 3D motion into the same space pre-trained VDMs [16]–[20], [57] to generate 3D motion and 2D video synchronously.
2.0.0.2 Image-based Human Animation.
Based on powerful image generation, pioneering image animation works animate static images by fine-tuning or adding additional motion adapters [58], [59] to pre-trained text-to-image (T2I) models [60]. Inspired by the success of ControlNet [61], a series of motion-driven methods are developed, utilizing 2D pose [62] to drive human video generation [21]–[25]. Recent works incorporate more multimodal control signals such as 3D parametric body model [26]–[29], [63]–[65], camera trajectory [31], [66], [67], optical flow [68], [69], 3D scene geometry [70], [71], novel background [72]–[75], and audio [76]–[78] to empower multi-subject interaction animation [79], [80], video subject replacement [81], and promptable animation [82], [83]. Additionally, researchers also explore how to implicitly transfer high-level motion patterns directly from reference to target videos [84]–[87], bypassing the extraction of explicit driving motion signals. Yet, such methods still require extra reference videos to drive the animation and are not capable of co-generating motions and videos.
2.0.0.3 Joint Generation of Human Motion and Video.
In order to achieve co-generation and remove the dependency of driving sources, latest works follow a cascaded generation pipeline by tailoring one generative model to the other. Motion-to-video methods generate human videos conditioning on pre-generated 2D pose [24], [88], [89], 3D motion [90]–[94] or optical flow [69] sequences, and video-to-motion frameworks [95] first generate human video and then refine it using the motion estimation results. Nevertheless, cascaded pipelines propagate defects in the upstream model and overlook the coupling relationship of two generative processes. In the field of multimodal generation and understanding, advanced approaches co-generate RGB videos and normals [79], [96], [97], depths [79], [96]–[102], segmentations [96], [97], optical flow [103], and motion maps [104]. However, the co-generation of 3D human motions and 2D human videos has not been well studied, and the few concurrent works [15], [105], [106] also reveal the promising prospects of this research direction.
3 Method↩︎
3.1 Overview↩︎
Our goal is to co-generate 3D human motion \(\left\{\boldsymbol{m}_i\in \mathbb{R}^{J\times 3}\right\}_{i=0}^F\) and RGB video \(\left\{\boldsymbol{s}_i\in \mathbb{R}^{H\times W\times 3}\right\}_{i=0}^F\) sequence in \(F\) frames given a starting image \(\boldsymbol{s}_0\) and a text description \(\boldsymbol{p}\). In which \(J\) is the number of body joints defined by SMPL [63], and \(H\times W\) is the resolution of generated videos. As illustrated in Fig. 3, we first estimate the 3D human motion \(\boldsymbol{m}_0\) of \(\boldsymbol{s}_0\) using CameraHMR [107] and render the 3D SMPL mesh posed in \(\boldsymbol{m}_0\) as our 2D human motion representation \(\boldsymbol{k}_0\in \mathbb{R}^{H\times W\times 3}\) according to vertex normals and body part semantics (Sec. 3.2). Then, \(\boldsymbol{s}_0\) and \(\boldsymbol{k}_0\) are zero-padded to \(F\) frames and fed into each branch of our dual-branch diffusion model, together with \(\boldsymbol{m}_0\) and \(\boldsymbol{p}\) to generate human video \(\left\{\boldsymbol{s}_i\right\}_{i=0}^F\), 2D motion map \(\left\{\boldsymbol{k}_i\right\}_{i=0}^F\), and 3D human motion \(\left\{\boldsymbol{m}_i\right\}_{i=0}^F\) sequence synchronously (Sec. 3.3). Our training data is prepared as introduced by Sec. 3.4.
3.2 2D Human Motion Representation↩︎
In this section, we aim to design a 2D motion representation composed of normal and semantic maps for the co-generation, which not only carries rich 3D information but also aligns with the 2D video. Existing works [26] treat normal and semantic maps as separate conditions. This leads to inherent limitations: normal maps lack the semantic information necessary to distinguish between different body parts, while semantic maps are devoid of 3D structural cues. To address this limitation, we propose an improved motion representation that cohesively integrates both types of information.
To be specific, given the \(i\)-th vertex \(\boldsymbol{ve}_i=\left(ve_x,ve_y,ve_z\right)\) of SMPL mesh, the vertex normal \(\boldsymbol{vn}_i=\left(vn_x,vn_y,vn_z\right)\) satisfies \[vn_z = \pm\sqrt{1-vn_x^2-vn_y^2},\] providing that \(vn_x\) and \(vn_y\) are known and only \(\operatorname{sign}(vn_z)\) is undetermined. Thus, we can combine this sign with body part semantics and encode both into a single vertex color channel. As illustrated in Fig. 4, we first encode \(vn_x\) and \(vn_y\) as the Blue and Green channels, respectively. Then, suppose that SMPL can be segmented into \(R\) body parts, we define a color list containing \(2R\) candidate values uniformly sampled in the range \(\left[0,1\right]\) for the Red channel. The Red channel value for \(\boldsymbol{ve}_i\) of part \(r\) is assigned as \[\mathrm{Red}(\boldsymbol{ve}_i) = \begin{cases} \mathrm{RedList[r]} & \text{if } \operatorname{sign}(vn_z) \geq 0 \\ \mathrm{RedList[r+1]} & \text{if } \operatorname{sign}(vn_z) < 0\;. \end{cases}\] This strategy enables effective compression of 3D surface normals and body part semantics into a single RGB image, which preserves essential 3D structural information and can also be embedded in the latent space of VDMs seamlessly.
3.3 Co-Generation of Human Motion and Video↩︎
To achieve co-generation of human motion and video, we adopt a dual-branch diffusion architecture and train it in multiple stages progressively. In the following, we use \(\boldsymbol{x}_t^\text{motion}\) and \(\boldsymbol{x}_t^\text{video}\) to represent the latents of \(\mathcal{D^{\text{motion}}}\) and \(\mathcal{D^{\text{video}}}\) at noise level \(t\), and the clean latents are \(\boldsymbol{x}_0^\text{motion}\) and \(\boldsymbol{x}_0^\text{video}\) respectively.
3.3.0.1 Adapt Pre-trained VDM to 2D Motion Domain.
Our first training stage is to finetune \(\mathcal{D^{\text{motion}}}\) to adapt its weights from the RGB video to our 2D motion representation domain. This is necessary because significant appearance shifts occur when directly applying the pre-trained VDM to our 2D motion representation sequence (see Fig. 5), corrupting the essential color patterns that encompass rich 3D information. In this stage, only \(\mathcal{D^{\text{motion}}}\) is tunable, which learns to denoise \(\boldsymbol{x}_t^\text{motion}\) to \(\boldsymbol{x}_0^\text{motion}\) conditioning on the text prompt and the starting frame.
3.3.0.2 Couple Motion and Video Generation Process.
In the second training stage, we introduce mutual feature interactions between dual branches to guide human video generation, and 3D-2D cross-attention module to output 3D human motion from video latents. Only \(\mathcal{D^{\text{video}}}\) is freezed in this stage.
3.3.0.3 Mutual Feature Interactions.
For the \(i\)-th diffusion block, we insert a group of two zero-linear modules after it between dual branches to obtain \[\begin{align} \boldsymbol{x}_t^\text{fused} &= \boldsymbol{x}_t^\text{motion} + \operatorname{ZeroLinear}_{i}(\boldsymbol{x}_t^\text{video}) \notag \\ \boldsymbol{x}_t^\text{video} &= \boldsymbol{x}_t^\text{video} + \operatorname{ZeroLinear}_{i+1}(\boldsymbol{x}_t^\text{motion}). \label{eq:fuse} \end{align}\tag{1}\] Then, we pass \(\boldsymbol{x}_t^\text{motion}\) and \(\boldsymbol{x}_t^\text{video}\) to the next \((i+1)\)-th diffusion block of \(\mathcal{D}^{\text{motion}}\) and \(\mathcal{D}^{\text{video}}\) respectively. The fused latent \(\boldsymbol{x}_t^\text{fused}\) will serve as the key and value in our 3D-2D cross-attention module.
3.3.0.4 3D-2D Cross-Attention Module.
To generate 3D human motion synchronously, we propose a 3D-2D cross-attention module \(\mathcal{A}\) composed of 6 layers of self-attention, cross-attention, and feed-forward network combination. We first initialize all \(F\)-frame SMPL poses with the known initial pose \(\boldsymbol{m}_0\), and apply a linear SMPL embedding layer to project them into \(F\)-frame embeddings \(\boldsymbol{q}\in\mathbb{R}^{F\times C}\). Subsequently, these initial pose embeddings are used as 3D queries going through self-attention layers as \[\boldsymbol{q} = \operatorname{SelfAttention}(\boldsymbol{q}).\] Then \(\boldsymbol{q}\) will interact with \(\boldsymbol{x}_t^{\text{fused}}\) to generate human poses for all frames. Since \(\boldsymbol{x}_t^{\text{fused}}\) is derived from 2D video latents which are compressed along temporal axis by the VAE encoder \(\mathcal{E}\) with compression ratio as 4 [19], 1 frame of \(\boldsymbol{x}_t^{\text{fused}}\) corresponds to 4 frames of \(\boldsymbol{q}\). Therefore, we re-organize \(\boldsymbol{q}\) per 4 frames to get \(\boldsymbol{q}'\in\mathbb{R}^{\frac{F-1}{4}\times4\times C}\) and apply 3D-2D cross attention as \[\boldsymbol{q} = \operatorname{CrossAttention}(\boldsymbol{q}',\boldsymbol{x}_t^{\text{fused}}).\] Finally, our model generates 3D human motion \(\left\{\boldsymbol{m}_i\right\}_{i=1}^F\) by a feed-forward network and a final output projection layer to map \(\boldsymbol{q}\) back to SMPL parametric dimension.
3.3.0.5 Losses.
Our total training loss is defined as \[\mathcal{L}^{\text{total}} = \mathcal{L}^{\text{motion}} + \mathcal{L}^{\text{video}} + \mathcal{L}^{\text{smpl}},\] where \(\mathcal{L}^{\text{motion}}\) and \(\mathcal{L}^{\text{video}}\) have the same flow matching objective formulation [108] \[\mathcal{L} = \mathbb{E}_{\boldsymbol{x}_0,\epsilon,t,\boldsymbol{p}}\left[ \left\Vert \mathcal{D}(\boldsymbol{x}_t,t,\boldsymbol{p}) - \boldsymbol{v}_t \right\Vert_2^2 \right],\] and \(\mathcal{L}^{\text{smpl}}\) is defined as \[\mathcal{L}^{\text{smpl}} = \frac{1}{F-1}\sum_{i=1}^{F-1}\left\Vert\boldsymbol{m}_i-\operatorname{GT}(\boldsymbol{m}_i)\right\Vert_2^2.\] During training, the gradients from \(\mathcal{L}^{\text{smpl}}\) backpropagate to the 2D motion branch \(\mathcal{D}^{\text{motion}}\) and eventually update the information injected to RGB video branch \(\mathcal{D}^{\text{rgb}}\). While in the inference phase, the primary function of \(\mathcal{A}\) is to generate 3D human motion from the latents of VDM, which does not iteratively guide the denoising process.
| Dataset | Clips | Motion Type | Resolution |
|---|---|---|---|
| Motion-X++ [30] | 120,500 | 3D SMPL | - |
| HumanVid* [31] | 19,688 | 2D Pose | 1080P+ |
| MotionVid [88] | 1.2M | 2D Pose | - |
| CoMoVi-Dataset | 54,053 | 3D SMPL | 720P+ |
22.2pt
3.4 CoMoVi-Dataset↩︎
Training our co-generative framework requires a high-quality dataset of triplets containing human videos, 3D human motions, and text annotations. Different from previous datasets that focus on single-modality taks such as motion-only or video-only generation, we collect real-world videos paired with 3D motion for the co-generation task. Concurrent works [15], [106] also make efforts and reveal the promising prospects of this research direction. As reported in Tab. 1, existing datasets [15], [30], [31], [88], [109], [110] fail to simultaneously meet our demands for high-quality video and sufficient 3D motion. Therefore, we curate such a dataset called CoMoVi-Dataset. Specifically, we source high-resolution human videos from Koala-36M [111], HumanVid [31], and publicly available Internet videos, and employ a carefully designed filtering pipeline to select clips featuring single-person motion based on Qwen3 [112], Qwen2.5-VL [113] and YOLO [114]. For 3D human motion annotations, we obtain pseudo-labels using SoTA in-camera human mesh recovery method CameraHMR [107] for the best re-projection alignment. A smoothing post-processing procedure is tailored to further refine the frame-wise annotations. For text descriptions, we utilize the advanced Gemini-2.5-Pro to generate precise motion captions for each video. More details about our data filtering hierarchy and prompts are provided in the supplementary material.
4 Experiments↩︎
In this section, we first introduce our implementation and evaluation details. Then, we compare our approach with SoTA T2M and I2V models for motion generation and video generation tasks, respectively. Finally, we conduct ablation studies to validate the effectiveness of our proposed motion representation and model architecture designs.
4.1 Implementation Details↩︎
We train our model on the training set of our proposed dataset using 24 A100-SXM4-40G GPUs with per GPU batch size 1 and gradient accumulation steps 4 for 6,000 optimization steps. We use the ZeRO-3 [115] strategy and AdamW optimizer with learning rate 2e-5. Our training data are unified to resolution \(H\times W=704\times1280\), \(F=81\) frames in 16 fps. More implementation details can be found in the supplementary material.
4.2 Evaluation Datasets and Metrics↩︎
For comprehensive evaluations on both motion generation and video generation tasks, we take the widely-used Motion-X++ dataset [30], VBench benchmark [32]–[34], and also our testing set to assess performance of all compared models. For the motion generation task, we adhere to the evaluation protocol defined by MoMask [10] and use Frechet Inception Distance (FID), R-Precision (1? and @3), and MultiModal Distance (MMDist) metrics. Since our method generates motions and videos from an input image and a text prompt, we adopt the Motion-X++ benchmark [30] containing texts, images, and 3D human motions. Though HumanML3D [38] is widely used in motion generation evaluation, it lacks images, so not a good fit with our method. For the video generation task, we calculate “Subject Consistency (SC)”, “Background Consistency (BC)”, “Motion Smoothness (MS)”, “Aesthetic Quality (AQ)”, and “Imaging Quality (IQ)” metrics provided by VBench [32]–[34]. The “Dynamic Degree (DD)” is excluded since it is aimed at measuring scene-level dynamics, on our testing set. For the following tables, the numbers marked in bold and underlined represent the first and the second best, respectively.
4.3 Comparisons on Motion Generation Task↩︎
4.3.0.1 Baselines.
We compare with SoTA T2M methods, including diffusion-based MDM [1] and autoregressive MotionGPT [9], Momask [10], and Go-to-Zero [12]. Additionally, since we explore a new task of co-generating 3D human motions and videos from text prompts together with an input image, there does not exist a T2M baseline that perfectly matches our setting currently. Therefore, we build and compare with a simple yet meaningful baseline Wan2.2-I2V-5B+CameraHMR [19], [107] that first generates human video from a starting image and a text prompt, and then captures the 3D motion of the person in that generated video. It is noteworthy that we sample test sequences from Motion-X++ [30] on our own since no official train/test split is provided, which might be included in the training set of Go-to-Zero [12].
![Figure 6: Qualitataive comparison of 3D human motion generation with SoTA T2M models [1], [9], [10], [12]. Wan2.2-I2V-5B+CameraHMR [19], [107] is a simple yet meaningful baseline composed a video generation model followed by a video motion capture model. We present motion keywords in text prompts for simplicity.](/article/2601.10632/figure/mogen_comp.png "fig:")
4.3.0.2 Results.
As presented in Fig. 6, CoMoVi generates 3D human motions with high prompt fidelity and dynamic smoothness, while baselines often produce jittery motions, unrelated contents, and implausible body movements. Quantitative evaluations in Tab. 2 also validate that our co-generative framework outperforms SoTA T2M models on the testing set of CoMoVi-Dataset, and also generalizes well, achieving comparable performance on the unseen MotionX++ dataset [30] to Go-to-Zero [12]. According to the results of Wan+CameraHMR [19], [107], incorporating additional reference images helps achieve promising performance by providing structural cues. Advancing more, our co-generative framework effectively couples motion and video modalities, enabling them to mutually guide one another during the denoising process, leading to further improvements in both generated motion and video.
| CoMoVi-Dataset | Motion-X++ [30] | |||||||
|---|---|---|---|---|---|---|---|---|
| 2-5 (lr)6-9 Method | FID \(\downarrow\) | R@1 \(\uparrow\) | R@3 \(\uparrow\) | MMDist \(\downarrow\) | FID \(\downarrow\) | R@1 \(\uparrow\) | R@3 \(\uparrow\) | MMDist \(\downarrow\) |
| MDM [1] | 27.740 | 0.287 | 0.540 | 4.848 | 20.871 | 0.332 | 0.406 | 4.803 |
| MotionGPT [9] | 26.079 | 0.316 | 0.569 | 4.486 | 27.084 | 0.305 | 0.529 | 5.276 |
| MoMask [10] | 24.521 | 0.385 | 0.606 | 4.943 | 19.365 | 0.316 | 0.569 | 4.915 |
| Go-to-Zero-3B* [12] | 3.175 | 0.440 | 0.605 | 3.585 | 15.474 | 0.311 | 0.692 | 3.751 |
| Go-to-Zero-7B* [12] | 1.641 | 0.459 | 0.608 | 3.429 | 14.470 | 0.384 | 0.699 | 3.578 |
| Wan+CameraHMR [19], [107] | 0.815 | 0.539 | 0.591 | 3.056 | 21.538 | 0.302 | 0.464 | 3.735 |
| Ours | 0.349 | 0.565 | 0.640 | 3.035 | 16.728 | 0.377 | 0.571 | 3.507 |
2.4pt
![Figure 8: Qualitataive comparison of human video generation with SoTA open-souce I2V models [18], [19], and a baseline composed of SoTA T2M [12] and motion-driven video generation model [26].](/article/2601.10632/figure/vigen_comp.png "fig:")
4.3.0.3 Extending to World-Centric Motion Generation.
As shown in Fig. 7, despite that CoMoVi generates in-place motion, it can be easily extended to world space via a kinematic post-processing. Details are provided in the supplementary materials.
| Method | SC \(\uparrow\) | BC \(\uparrow\) | MS \(\uparrow\) | AQ \(\uparrow\) | IQ \(\uparrow\) |
|---|---|---|---|---|---|
| CogVideoX1.5-I2V-5B [18] | 0.933 | 0.947 | 0.991 | 0.512 | 0.716 |
| Wan2.2-I2V-5B [19] | 0.948 | 0.960 | 0.993 | 0.529 | 0.713 |
| Go-to-Zero-7B+Champ [12], [26] | 0.931 | 0.934 | 0.980 | 0.479 | 0.663 |
| Ours | 0.955 | 0.963 | 0.993 | 0.533 | 0.718 |
11.2pt
4.4 Comparisons on Video Generation Task↩︎
4.4.0.1 Baselines.
We choose to compare with two leading open-source I2V models, CogVideoX1.5-I2V-5B [18] and Wan2.2-I2V-5B [19], to which the model size of CoMoVi is comparable. Additionally, we construct a cascaded baseline composed of SoTA text-driven motion synthesis [12] and motion-driven video generation [26] to ensure a fair and meaningful comparison between different models and paradigms in similar scale and capability.
4.4.0.2 Results.
As depicted in Fig. 8, CoMoVi gets benefits from our 2D motion representation, which encapsulates rich 3D motion information, generating realistic human videos with more consistent body structure, higher prompt fidelity, and anatomically plausible motions. While CogVideoX1.5 [18] and Wan2.2-I2V-5B [19] struggle with motion-prompt misalignment, distorted body shapes, and background maintenance. Though the motion-driven video generation methods achieve promising performances as shown in previous works [21], [26], [31], cascading a SoTA T2M model Go-to-Zero [12] and a SoTA motion-controllable video generation model Champ [26] leads to inferior results due to the limited generalization capability, low prompt fidelity, and misalignment with the video frames to guide video generation. Details are provided in the supplementary material. In contrast, CoMoVi achieves high-quality video generation by the co-generation of aligned motion and videos, avoiding reliance on external video and motion reference.
The quantitative results detailed in Tab. 3 demonstrate the advantage of our co-generative framework in all evaluation dimensions defined by VBench [32]–[34]. It is noteworthy that VBench metrics [32]–[34] prioritize consistency of subject identity and background, smoothness of pixel flow, and imagery clarity, but do not fully show the human motion quality and plausibility. As evidenced by Fig. 8 and Tab. 3, our method significantly outperforms baselines in body structure consistency and prompt fidelity, but the metric differences are within \(\pm 0.05\) due to no significant variation or mutation exhibited in baseline results.
4.5 Ablation Study↩︎
We further conduct extensive ablation studies to validate the significance of our 2D motion representation and the effectiveness of our model architecture design.
4.5.0.1 Ablation of 2D Motion Representations.
As introduced in Sec. 3.2, our 2D human motion representation integrates surface normals with body part semantics. Therefore, we experiment with “normal only” and “body semantic only” settings, and also a commonly used 2D pose representation “DWPose” [62]. Tab. 4 shows that directly fine-tuning Wan2.2-I2V-5B [19] on RGB videos only without our co-generative framework (“w/o motion”) leads to significant performance degradation. While the absence of any factor among normals, body semantics, and surface rendering results in suboptimal performance in both generation tasks.
4.5.0.2 Ablation of Model Architecture Designs.
In designing our model architecture, we explore several approaches inspired by VideoJAM [103] and VACE [116], ultimately adopting a dual-branch diffusion model with the full copy strategy.
4.5.0.3 Dual-Branch Architecture.
Specifically, we first refer to VideoJAM [103] to concatenate RGB videos and our 2D motion representations along the latent channel dimension, and double the dimension of the patch embedding layer and the output head layer of Wan2.2-I2V-5B [19]. However, we observe that this severely disrupts the pre-trained VDM’s latent space during the initial training phase, requiring a computationally expensive reconstruction process. Consequently, this method fails to outperform baseline models with a limited training budget.
| Motion Generation | Video Generation | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 2-5(lr)6-10 Ablation Configs | FID \(\downarrow\) | R@1 \(\uparrow\) | R@3 \(\uparrow\) | MMDist \(\downarrow\) | SC \(\uparrow\) | BC \(\uparrow\) | MS \(\uparrow\) | AQ \(\uparrow\) | IQ \(\uparrow\) |
| Motion Representation | |||||||||
| w/o motion | 0.758 | 0.458 | 0.556 | 4.079 | 0.937 | 0.944 | 0.980 | 0.520 | 0.709 |
| Normal only | 0.415 | 0.538 | 0.556 | 3.003 | 0.918 | 0.946 | 0.988 | 0.521 | 0.710 |
| Body semantic only | 0.553 | 0.499 | 0.589 | 3.536 | 0.940 | 0.944 | 0.994 | 0.528 | 0.708 |
| DWPose | 0.503 | 0.464 | 0.551 | 3.396 | 0.929 | 0.951 | 0.981 | 0.522 | 0.701 |
| Architecture | |||||||||
| VideoJAM joint latent space | 0.755 | 0.459 | 0.559 | 3.534 | 0.868 | 0.942 | 0.987 | 0.529 | 0.705 |
| VACE distributed copy | 0.625 | 0.500 | 0.572 | 3.148 | 0.927 | 0.949 | 0.994 | 0.522 | 0.715 |
| Pass \(\bm{x}_t^{\text{fused}}\) to \(\mathcal{D}^{\text{motion}}\) | 0.718 | 0.466 | 0.552 | 4.157 | 0.911 | 0.955 | 0.985 | 0.530 | 0.718 |
| Ours w/o \(\mathcal{L}^{\text{smpl}}\) | - | - | - | - | 0.951 | 0.961 | 0.993 | 0.532 | 0.714 |
| Ours w/o \(\mathcal{D}^{\text{motion}}\) | - | - | - | - | 0.949 | 0.940 | 0.988 | 0.517 | 0.709 |
| Ours | 0.349 | 0.565 | 0.640 | 3.035 | 0.955 | 0.963 | 0.993 | 0.533 | 0.718 |
2pt
4.5.0.4 Full Copy of DiT Blocks.
Therefore, we adopt dual-branch diffusion architecture following VACE [116], which effectively preserves the integrity of the latent space of pre-trained VDM. Nevertheless, we identify that the distributed copy strategy proposed by VACE [116] causes the copied branch to completely lose the prior knowledge of pre-trained VDM. At the first training step, the output of the copied branch is totally noised, indicating that it essentially discards priors and requires to be trained from scratch. The quantitative performance shown in Tab. 4 demonstrates that our dual-branch diffusion architecture with the full copy strategy achieves better performance in both generation tasks.
4.5.0.5 Inject Information from \(\mathcal{D}^{\text{rgb}}\) to \(\mathcal{D}^{\text{motion}}\).
Furthermore, we experiment with passing the fused latent feature \(\boldsymbol{x}_t^{\text{fused}}\) defined in Eq. 1 rather than \(\boldsymbol{x}_t^{\text{motion}}\) to the 2D motion diffusion branch \(\mathcal{D}^{\text{motion}}\). Intriguingly, it is found that the direct feature injection from RGB to 2D motion latents significantly disturbs the denoising process of \(\mathcal{D}^{\text{motion}}\). We analyze that 2D motion representation is far more sparse than RGB videos containing rich appearances and background details, which results in chaotic and fluctuating artifacts in 2D motion generation, degrading the preservation of essential 3D information. And eventually, the effectiveness of motion guidance is weakened, leading to the suboptimal performance (see Fig. 9).
4.5.0.6 3D Regularization \(\mathcal{L}^{\text{smpl}}\).
As stated in Sec. 3.3, the additional 3D regularization term \(\mathcal{L}^{\text{smpl}}\) is introduced to reinforce the alignment between 2D latents and 3D representation, ensuring structure consistency and visual plausibility of generated human videos. Without \(\mathcal{L}^{\text{smpl}}\), as we quantitatively evaluate in Tab. 4, the quality of video generation results degrade especially in “Subject Consistency” dimension.
4.5.0.7 Directly Use 3D Motion without \(\mathcal{D}^{\text{motion}}\).
Finally, we validate the importance of our 2D motion branch \(\mathcal{D}^{\text{motion}}\). Tab. 4 demonstrates that discarding our 2D motion representation and \(\mathcal{D}^{\text{motion}}\) causes performance degradation in all five evaluation dimensions on video generation task. We analyze that, different from motion-driven video generation methods like Animate Anyone [21] and Champ [26] which require a reference video and pre-extracted 2D pose or 3D motion sequence from it, co-generation of human motion and video necessiates a tight coupling the generation process of the two modalities to ensure the alignment of gerated 3D motions and 2D videos. Consequently, our 2D motion representation is not only pixel-aligned with RGB video, but also encodes rich 3D structure and semantic information, which is incorporated in a shared denoising process with VDM via \(\mathcal{D}^{\text{motion}}\). This design thus achieves superior performance to directly using 3D motion conditioning without \(\mathcal{D}^{\text{motion}}\).
5 Conclusion↩︎
In this work, we propose a novel framework called CoMoVi for co-generation of 3D human motions and realistic videos. Our key idea is to couple the denoising process of 3D motions and 2D videos, enabling synchronous generation of both within a single diffusion loop. We first introduce a novel 2D motion representation that encodes surface normals and body part semantics of 3D SMPL mesh into RGB images, allowing it to directly inherit priors from pre-trained VDMs. Then, we develop a dual-branch diffusion model with mutual feature interactions and 3D-2D cross-attentions, providing motion guidance for video generation while propagating VDM’s generalization capability to 3D motion generation. Moreover, we contribute the CoMoVi-Dataset, a large-scale human video collection annotated with high-quality text and motion labels to support versatile video-based and motion-related tasks. Comprehensive experiments on multiple benchmarks demonstrate our method’s effectiveness in both motion and video generation.
6 More Implementation Details↩︎
| Hyper-parameter | Value |
|---|---|
| Batch Size / GPU | 1 / 24 |
| Accumulate Step | 4 |
| Optimizer | AdamW |
| Weight Decay | 3e-2 |
| Learning Rate | 2e-5 |
| AdamW \(\beta_1\) | 0.9 |
| AdamW \(\beta_2\) | 0.999 |
| Scheduler | Constant with Warmup |
| Warmup Steps | 100 |
| Training Steps | 6,000 |
| Resolution | \(704\times 1280\) |
| Max Gradient Norm | 0.5 |
| Weighting Scheme | Uniform |
| Pre-trained Model | Wan2.2-I2V-5B |
| Sampler | Flow Unipc |
| Sample Steps | 50 |
| CFG Scale | 6.0 |
| Training Device | A100-SXM4-40G\(\times\)24 |
| Training Strategy | ZeRO-3 |
| Gradient Checkpointing | True |
| CPU Offload | True |
| Precision | bf16 |
| Training Speed | 280s per accumulation |
| Inference Speed | 15min per video in 5s |
r0.48 0.01pt
The main paper outlines the primary training and experimental parameters. This section provides a comprehensive description of multiple stages of our model training, and the detailed procedures of the comparative experiments on motion and video generation tasks.
6.0.0.1 Training.
Our model training is conducted in two stages. The first stage adapts the copied motion DiT branch \(\mathcal{D}^{\text{motion}}\) to our 2D motion representation domain using \(\mathcal{L}^{\text{motion}}\) only. During this stage, only \(\mathcal{D}^{\text{motion}}\) is enabled with gradient calculation and weight update, trained for 2,000 steps. In the second stage, we incorporate the mutually interactive zero-linear layers and 3D-2D cross-attention modules into the training process, using the total loss \(\mathcal{L}^{\text{total}}\) to supervise it. To enhance training efficiency and reduce GPU memory consumption, at each training step we randomly select the latent features from three DiT layers, while consistently including the final layer of \(\mathcal{D}^{\text{video}}\) and \(\mathcal{D}^{\text{motion}}\), to operate feature fusion and perform cross-attention with the 3D motion query. The pre-trained weights of RGB DiT branch \(\mathcal{D}^{\text{video}}\) are frozen, while the remaining components are updated over 4,000 steps at the second training stage. In total, the model is trained for 6,000 steps across both stages. Detailed hyperparameter configurations are provided in Tab. 5.
6.0.0.2 Experiments of Motion Generation.
For 3D human motion evaluation, we follow the 263-dimensional representation and use the pretrained motion and text encoder provided by HumanML3D [38]. The metrics reported in the main paper are determined by the average of 20 independent inference results with a confidence interval of 95% for each model.
None
Figure 11: Prompt instruction for Qwen3 [112] to analyze dense video captions..
None
Figure 12: Prompt instruction for Qwen2.5-VL [112] to analyze the first frame of video..
None
Figure 13: Prompt instruction for Gemini2.5-Pro to caption human motion in videos..
6.0.0.3 Experiments of Video Generation.
For fair comparative experiments of video generation task, all models are evaluated at their optimal resolutions (\(768\times 1360\) for CogVideoX1.5-I2V-5B [18], and \(704\times 1280\) for Wan2.2-I2V-5B [19] and ours), and each model is configured to generate video sequences of a unified length of 81 frames. All results are generated once using random seed 42 and CFG scale 6.0 without any cherry-picking.
7 CoMoVi-Dataset Curation↩︎
In the main paper, we give an overview of our dataset construction process. Here, we elaborate on the detailed procedures. As illustrated in Fig. 10, firstly, we curate a collection of high-quality videos sourced from the Internet and the Koala-36M dataset [111], and employ LLMs/VLMs [112], [113] to selectively retain only those depicting single-person movements (Sec. 7.1). Subsequently, we apply 2D human pose detection model [114], [117] to filter out videos where the subject is largely outside the frame or severely occluded (Sec. 7.2). The filtered videos are finally captioned (Sec. 7.3) and labeled with 3D motion (Sec. 7.4). We provide specific prompt instructions used at each filtering stage and our confirmation of dataset ethics (Sec. 7.5) below.
7.1 Multimodal Video Filtering↩︎
Given a large collection of publicly available videos, we ask the Qwen3 model [112] to determine if the video contents satisfy our criteria according to the corresponding dense captions using the prompt instruction shown in Fig. 11. We only keep the videos that are judged as “" in this initial text-based filtering stage, and then process them using the Qwen2.5-VL model [113], which takes the first frame of each video and the prompt instruction specified in Fig. 12 to make a visual confirmation. This multimodal filtering procedure effectively filters out non-human videos, multi-person videos, and contents such as movies, animations, and video games that may contain human-like characters but do not represent real-world human motions.
7.2 Human Tracking Filtering↩︎
To ensure a balanced data distribution and avoid overwhelming clips from long videos, we segment each video into non-overlapping 5-second clips, with a maximum of two clips retained per video. Subsequently, we apply human tracking utilizing YOLO [114] and ViTPose [117] for each clip. A series of confidence thresholds are established to flag frames with low-confidence human detections throughout a sequence. Clips containing an excessive number of such low-confidence frames are filtered out to ensure data quality.
7.3 Video Captioning↩︎
For all retained 5-second clips, the human motion captioning is performed by querying the Gemini-2.5-Pro API at 1 fps, adhering to the official recommendation. The prompt instruction provided for motion captioning is shown in Fig. 13. We also experiment with higher frame rates, which do not yield improvement in caption quality but substantially increase the cost of annotation.
7.4 3D Human Motion Annotation↩︎
We estimate the SMPL parameters [63] for each frame of the filtered human videos using CameraHMR [107]. Since these per-frame estimations are independent, which can lead to motion jitter due to occlusions or motion blur in videos, we apply the smoothing curve in Blender to post-process the estimated body motions to ensure temporal coherence. For consistency of body shape across the video sequence, the SMPL shape parameters estimated from the first frame are applied to all subsequent frames.
7.5 Dataset Ethics↩︎
We strictly adhere to the conference ethics guidelines and only collect publicly available videos from academic datasets and open social media platforms. We confirm that all collected data are used only for research purposes. This condition will also be explicitly stated upon the release of our dataset. We fully respect the privacy of the individuals appearing in the videos, so no personal information or metadata is retained. Furthermore, in compliance with video ownership rights, only video identifiers will be publicly released, rather than the original video files.
8 More Discussions↩︎
8.1 How to Extend CoMoVi Results to World-Space↩︎
As shown in Fig. 7 in the main paper, a simple and deterministic kinematic approach can be employed to “lift” in-place motion generated by CoMoVi to world space via physical constraints of locomotion that leverages the zero-velocity assumption of the stance foot. Specifically, for each time step \(t\), we identify the stance joint \(k\) (e.g., the ankle or toe) as the end-effector exhibiting the minimal vertical elevation relative to the root. Assuming the stance foot remains static in the world coordinate system (i.e., \(\mathbf{v}_{k}^{\text{world}} \approx 0\)), the global linear velocity of the root joint, \(\mathbf{v}_{\text{root}}\), can be derived by inverting the relative velocity of the stance foot: \(\mathbf{v}_{\text{root}} \approx - \dot{\mathbf{p}}_{k}^{\text{local}}\), where \(\mathbf{p}_{k}^{\text{local}}\) denotes the position of the stance foot in the root-relative coordinate system. The final global trajectory is obtained via temporal integration of these frame-wise velocities \[\begin{align} \mathbf{v}_{\text{root}}^{(t)} \approx - \dot{\mathbf{p}}_{k}^{\text{local}} &= - \left( \mathbf{p}_{k}^{(t)} - \mathbf{p}_{k}^{(t-1)} \right) / \Delta t\\ \mathbf{T}^{(t)} &= \mathbf{T}^{(t-1)} + \mathbf{R}^{(t-1)} \mathbf{v}_{\text{root}}^{(t)} \Delta t, \end{align}\] where \(\mathbf{T}\) and \(\mathbf{R}\) represents the root translation and orientation respectively.
8.2 Denoising Guidance of 3D Motion during Inference↩︎
During inference, the 3D guidance is provided by our 2D motion representation (see Fig. 14), which is 3D-aware (containing both body surface normals and semantics information) and pixel-aligned with the RGB video. We introduce the 3D-2D cross-attention module to learn to output 3D motion from DiT latents, achieving synchronous geneartion of 3D motion and 2D video. This design is also inspired by REPA [118], regularizing the diffusion training process via additional 3D regularization \(\mathcal{L}^{\text{smpl}}\).
8.3 Evaluating on Motion-X++ Benchmark↩︎
In the field of text-driven human motion generation, both HumanML3D [38] and Motion-X++ [30] are commonly used benchmarks. However, HumanML3D [38] focuses solely on text-motion pairs, whereas Motion-X++ [30] is a tri-modal dataset of text, image, and 3D human motion, which aligns the task investigated in this work. Moreover, the research problem of this work is co-generating 3D human motion and realistic video, which means synthetic data is undesired and real-world video data is required. We therefore adopt the Motion-X++ benchmark [30] for our evaluation of the motion generation task.
9 Limitations and Future Work↩︎
While our method offers significant advantages, it is constrained to generating fixed-length motion sequences, lacking the capacity for variable-length or infinite-length generation. Additionally, due to the inherently denser nature of video latents compared to 3D human motion data, our inference speed is relatively slower. Promising future directions include extending our framework to human-object interaction scenarios, employing distillation techniques for generation acceleration, and enabling the generation of variable-length sequences.