¹

1 Introduction↩︎

Multimodal whole-body human motion generation plays critical roles in animation [10], gaming [11], virtual reality [12], and embodied intelligence [13] across diverse input conditions, including text, audio, and trajectory, etc. The limited multimodal data and task-specific model designs prevent existing motion generation methods from supporting multimodal whole-body human motion generation. Due to the high cost of mocap data collection and labeling, most datasets focus on single domains such as Text-to-Motion (T2M) [14], [15], Music-to-Dance (M2D) [16], [17], Speech-to-Gesture (S2G) [18], [19], Human-Object Interaction (HOI) [20]–[22], Human-Scene Interaction (HSI) [23], and Human-Human Interaction (HHI) [24], [25], each with inconsistent data formats (e.g., BVH, SMPL-(H/X) and 3D Keypoints) and control conditions (e.g., Text captions, Audio, and Trajectories). To tackle motion generation across diverse scenarios, it is crucial to build a unified framework that utilizes large-scale, diverse data to achieve more generalized representations.

Although recent works [2]–[4], [8], [26] attempt to unify multitasks in one model, they meet challenges in multimodal control modeling, versatile tasks, and high-quality motion generation (as shown in Tab. 1): (1) Independent Model Training. Previous approaches train separate models for each modality, limiting simultaneous control across inputs [8]. (2) Additional Control Branches. Some methods add separate control branches for each condition, limiting interaction between them [2], [4]. (3) Conflict Granularity Training. Existing methods use mixed training by combining high-level semantic conditions with low-level controls, which hampers effective control at different levels and leads to optimization challenges [3], [7], [26]. Similar phenomena have also been observed in video generation [27], [28]. In addition to modeling challenges, relevant works also introduce large-scale motion datasets from multiple tasks and modalities [3], [26], [29]–[31], they still exhibit the following significant shortcomings (see comparisons in Tab. 2): (1) Low Motion Quality. Datasets expanded with non-mocap motion estimation exhibit “garbage in, garbage out" effects, leading to poor-quality motion [3], [29], [31]. (2) Text Inconsistency. Inconsistent text annotations or expanded LLM descriptions lead to uneven text quality and hallucination issues [26], [30], [31]. and (3) limited tasks. They focus on common tasks, limiting applicability in diverse ones like HOI, HSI, and HHI [3], [4], [31]. These limitations hinder the development of a unified, high-quality dataset for multimodal whole-body human motion generation across diverse scenarios.

To address existing challenges, we propose OmniMotion-X, a unified framework for multimodal whole-body human motion generation, and OmniMoCap-X, the largest unified multimodal mocap motion dataset. Specifically, OmniMotion-X employs Diffusion Transformers (DiTs) DIT?, incorporates multimodal conditions by concatenating condition tokens as prefix context, and adopts a progressive weak-to-strong mixed conditions training strategy to gradually constrain motion from high-level semantics to dense spatial-temporal alignment. Notably, unlike previous methods, OmniMotion-X introduces a novel generation paradigm that utilizes reference motion (e.g., user-provided or model-predicted motion) as a special condition. This significantly enhances generated motion quality and achieves consistency between reference and generated motion, creating an effective clip-by-clip autoregressive motion diffusion. This enables OmniMotion-X to support autoregressive interactive generation with strong temporal alignment. Furthermore, OmniMotion-X unifies various Global Spatial-Temporal Controllable Generation tasks through spatial-temporal masking strategies. To guarantee high-quality motion generation training, we collect high-quality mocap datasets that support diverse motion generation tasks, unify them under the SMPL-X [32] format with standard world coordinate systems, and automatically generate hierarchical text captions by rendering motions into videos and annotating them with vision language models (VLMs). This dataset contains about \(286.2\) hours and integrates multimodal control conditions, supporting versatile tasks, including T2M, M2D, S2G, HOI, HSI, and HHI.

In summary, our core contributions are as follows:

• We propose OmniMotion-X, a multimodal autoregressive diffusion transformer for versatile whole-body motion generation. By introducing reference motion, OmniMotion-X significantly enhances consistent content, style, and temporal dynamics generation.

• We propose a progressive weak-to-strong mixed conditions training strategy to effectively handle multi-granular constraints.

• We construct OmniMoCap-X, the largest unified multimodal mocap motion dataset with a unified SMPL-X format, and provides consistent, detailed, and structured text captions to serve versatile motion generation tasks.

• Extensive experiments show that OmniMotion-X achieves state-of-the-art performance across various tasks, including T2M, S2G, M2D, and GSTC. These evaluations were conducted on our more challenging test sets comprising \(280\) samples uniformly sampled from our OmniMoCap-X across diverse scenarios.

2 Related Work↩︎

2.1 Human Motion Generation↩︎

Existing methods are typically categorized based on input conditions into three main types: single-modal, cross-modal, and multimodal. Single-modal motion generation uses control conditions from the same modality as the motion, including motion prediction [33], [34], in-betweening [35], joint/trajectory-guided synthesis [9], [36], [37], and body-shape-conditioned motion generation [38], [39]. Cross-modal motion generation uses control conditions from different modalities, including text in T2M [1], [5], [6], [30], [40]–[44], music in M2D [16], [45]–[47], speech in S2G [18], [19], [48], [49], target object positions in HOI [21], [50], [51], scene layouts in HSI [23], [52], [53], and partner movements in HHI [24], [25], [54]. However, these approaches typically rely on task-specific architectures, significantly limiting their cross-task generalization capabilities and practical applications. Recently, researchers [2]–[4], [7], [8], [55] have begun exploring multimodal motion generation, with approaches falling into three primary categories:

• Separate Model Training. These approaches [8] typically train independent models for different conditional modalities, making multimodal motion generation fundamentally unattainable.

• Additional Control Branch. These approaches [2], [4] incorporate separate control branches into the backbone architecture, each dedicated to a specific condition, resulting in limited interaction between different conditions.

• Unified Multimodal Condition Modeling. These approaches [3], [26], [56], [57] use a single model to learn mappings from diverse modalities to target motions, enhancing multi-modal adaptability. However, challenges arise from different constraints across modalities: text provides semantic guidance, spatial-temporal controls impose physical constraints, while audio inputs enforce rhythmic alignment. These disparate constraints often result in compromised controllability and optimization difficulties.

2.2 Human Motion Dataset↩︎

Current human motion generation datasets are predominantly task-specific. Different tasks rely on separate collections: text-to-motion (T2M) uses datasets with text captions [11], [14], [15], [29], [58], music-to-dance (M2D) requires music-annotated datasets [16], [17], [45], [59], and speech-to-gesture (S2G) employs datasets with paired speech [18], [19], [48], [60]. Similarly, interaction datasets remain disconnected across human-object [20], [21], [50], [61]–[64], human-scene [23], and human-human interaction [24], [25]. This fragmentation constrains multimodal motion generation capabilities. Integrating these datasets presents challenges due to inconsistent motion formats. Datasets vary widely in their representations: some use SMPL [17], [20], [24], [65], others employ SMPL-X [18], [19], [48], while many rely on BVH [66]–[70] or keypoint representations [71], [72]. This diversity makes format standardization extremely difficult. Recent works [3], [26], [29]–[31] have attempted to integrate multiple datasets, increasing scale and incorporating some multimodal conditions. However, they still exhibit several limitations:

• Low-quality Motions: Most integrated datasets derive from non-mocap sources with significant estimation errors [3], [26], [29], [31], [65], creating a “garbage in, garbage out" effect that compromises motion generation quality.

• Text Inconsistency: Existing datasets [3], [30], [31] use inconsistent text annotations (e.g., action labels versus semantic descriptions) or rely on LLMs for text refinement without visual motion data, causing variable text quality and hallucinations [26], [29].

• Limited Generation Scenarios: Most datasets lack support for critical interaction tasks including HOI, HSI [3], [26], [29]–[31], and HHI [3], [29], [31], severely limiting their applicability to complex scenarios.

3 OmniMoCap-X Dataset↩︎

To address the data challenges—the scarcity of high-quality multitask motion datasets, inconsistent textual annotations and motion representations—we implement a three-pronged approach: (1) integrating diverse motion capture datasets with multitask support, (2) establishing a unified motion representation, and (3) developing high-quality motion captions derived from visual-textual annotations.

3.1 Mocap Dataset Composition↩︎

While existing approaches prioritize dataset scale over quality by incorporating substantial amounts of non-mocap data [3], [26], [29], [31], leading to degraded motion generation quality ("garbage in, garbage out"), our work emphasizes data quality. We systematically curate open-source mocap datasets and classify them into five acquisition categories: Marker with manual correction, Marker (Vicon), IMU, Multi-View RGB, and Single-View RGB (detailed in Tab. 3). Our dataset supports diverse motion generation tasks, including text-to-motion, motion-to-dance, speech-to-gesture, and various interactions (human-object, human-scene, and human-human), providing a high-quality multimodal foundation. Comprising \(64.3\) million frames and \(286.2\) hours of data, our approach achieves comprehensive task coverage while maintaining superior data quality. Visualizations of specific interaction scenarios are showcased in the supplementary material.

3.2 Unified Motion Representation↩︎

We standardize diverse motion formats (BVH [77], FBX [78], and SMPL (-H) [79]) into the SMPL-X [32] format to support our unified multimodal model. First, we convert these formats into the Motion-X [29] SMPL-X format, comprising root orientation, pose parameters (body, hand, and jaw), facial expressions, facial shape, translation, and body shape parameters. We then normalize the translation scale and initial root orientation across datasets to establish a consistent coordinate system. Additionally, we resample all datasets to 30 frames per second (FPS) to enhance the model’s ability to capture temporal patterns. The specific conversion process and normalization for different datasets are provided in the supplementary material.

To enhance generalization ability, we extend the widely-used body-only representation [14] to a whole-body format. Specifically, the pose of the \(i\)-th frame is a tuple \(\mathbf{p}_i = ({\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, \mathbf{j}^p, \mathbf{j}^v, \mathbf{c}^f, \mathbf{f}})\), where \(\dot{r}^a\in \mathbb{R}\) is root angular velocity around the Y-axis; \(\dot{r}^x, \dot{r}^z \in \mathbb{R}\) are root linear velocity in the XZ plane; \(r^y \in \mathbb{R}\) is root height, \(\mathbf{j}^p \in \mathbb{R}^{3N-1}\) are local joint positions; Notably, \(\mathbf{j}^r \in \mathbb{R}^{6N'}\) are joint 6D rotations of SMPL-X; \(\mathbf{j}^v \in \mathbb{R}^{3N}\) are joint velocities; \(\mathbf{c}^f\) is binary features obtained by thresholding the heel and toe joint velocities to emphasize the foot ground contacts. Here, \(N\) refers to 127 whole-body joints extracted from SMPL-X [32], while \(N'\) represents 53 joints spanning the body, hands, and jaw. For facial representation, we adopt the Flame Format [80], encoding facial features as \(\mathbf{f} \in \mathbb{R}^{100}\).

3.3 Consistent Visual-textual Motion Captions↩︎

Current datasets employ three annotation approaches:

• LLM Refinement [29], [31] employs LLMs like GPT-4 [81] to enhance textual descriptions. However, since LLMs cannot directly perceive motion data and rely solely on existing text, they frequently introduce hallucinations and fail to capture precise action details.

• Manual Annotation [14], [26], [58] effectively eliminates hallucinations but proves costly and difficult to scale, resulting in limited annotations with simple descriptions.

• Motion-to-Text Model-Based Annotation [82], [83] generates captions directly from raw motion. While these methods effectively capture low-level kinematic details (e.g., left elbow bends 45 degrees), they lack the understanding of high-level semantics and action context.

To overcome these limitations, we propose an automated approach that integrates both visual and textual information for comprehensive motion annotation. Our method renders motion into videos and combines them with existing textual annotations (e.g., descriptions, action labels, and task categories) as input to the state-of-the-art vision-language models (VLMs), GPT-4o [81]. This multimodal approach generates structured, hierarchical, and precise motion captions that ensure both annotation quality and expression richness. The quality of the texts was ensured through iterative prompt optimization and a comparative evaluation of various VLMs, with specific examples and detailed caption statistics provided in the supplementary material.

4 Method↩︎

The overview of OmniMotion-X is illustrated in Fig. 1. We propose a unified multimodal, autoregressive transformer-based diffusion model for whole-body human motion generation across multiple tasks. The model takes textual description \(\mathbf{c}_{t}\) for semantic guidance, global motion \(\mathbf{c}_{g}\) for spatial-temporal control, and speech \(\mathbf{c}_{s}\) and music conditions \(\mathbf{c}_{m}\) to ensure rhythmic and stylistic coherence. Furthermore, it incorporates reference motion \(\mathbf{c}_{r}\) as a motion prior, derived from either previously generated clips or user-designed motion, thus providing fine-grained details unavailable in other conditions. This reference input is optional and can be set to null, which is essential for generating the initial motion when no user-specified prior exists. In addition, the mixed-condition setting allows multiple conditions to occur simultaneously during training, enabling the model to handle complex scenarios with diverse inputs. In this section, we present our approach in two parts: a unified framework for multimodal modeling and a progressive weak-to-strong mixed conditions training strategy that ensures precise motion control.

4.1 Unified Multimodal Modeling↩︎

We integrate multimodal conditions using Diffusion Transformers (DiTs) DIT? by concatenating condition tokens as prefix contexts, allowing the model to process and fuse multimodal information efficiently.

Multimodal Conditions. Our framework integrates multiple condition modalities: text \(\mathbf{c}_t\) providing semantic guidance; global motion \(\mathbf{c}_g\) ensuring spatial-temporal consistency; speech \(\mathbf{c}_s\) synchronizing gestures and lip movements with rhythm; music \(\mathbf{c}_m\) supplying beat and style information for dance; and reference motion \(\mathbf{c}_r\) serving as a motion prior. Notably, this reference motion condition—overlooked in previous multimodal motion generation [2]–[4]—enables our model to maintain precise spatial-temporal pattern consistency with the reference, substantially enhancing motion quality and coherence. Crucially, a reference motion provides rich spatio-temporal context that a single starting pose cannot. By typically spanning a duration similar to the target sequence, it captures fine-grained details and style, which are absent in a momentary posture. To fully leverage each modality and facilitate interaction between different conditions, we employ modality-specific encoders (e.g., T5-XXL [84] for text, a wav encoder [18] for speech, Librosa [85] for music and body-wise encoding [4] for motion) to extract features from each modality. These features are aligned to match motion embedding dimensions using learnable linear projections, allowing us to concatenate all condition tokens as prefix context with noisy motion tokens during processing.

Multimodal Modeling. Overall, the unified multimodal modeling is formulated as follows: \[\scalebox{0.78}{c = [h_t ( f_t (c_t)), h_g (f_g (c_g)), h_s ( f_s (c_s)), h_m ( f_m (c_m)), h_r ( f_r (c_r))],} \label{eq:c95representation}\tag{1}\] where \(f\) and \(h\) denote the modality-specific encoder and projection layer, respectively. The concatenated representation \(c\) is then fed into our DiT backbone as a conditioning prefix context, where attention mechanisms are employed to learn the correspondences among the various modalities. To constrain the physical properties of motion, we follow previous works [1], [10] by directly predicting motion \(\hat{x}_0\) rather than noise. Consequently, our diffusion objective is defined as follows: \[\mathcal{L}_{\text{simple}} = \mathbb{E}_{x_0 \sim q(x_0 | c), t \sim [1,T]} \left[ \left\| x_0 - G(x_t, t, c) \right\|_2^2 \right], \label{eq:simple95loss}\tag{2}\] where \(q(x_0 | c)\) represents the data distribution, \(T\) is the maximum diffusion step. \(G\) denotes the learned denoising function, while \(x_t\) represents the noisy motion at step \(t\), expressed as \(x_t = [p_0^t, p_1^t, \dots, \mathbf{p}_N^t]\), where each \(\mathbf{p}_i^t\) corresponds to the \(i_{th}\) pose in motion at step \(t\). Further details are available in the supplementary material.

4.2 Weak-to-Strong Progressive Training Strategy↩︎

Considering our diverse multi-granularity conditioning inputs, we empirically observed that employing all conditions within a single-stage training paradigm leads to difficulties in directly learning the correlation between motion and conditions. Moreover, the model tends to overfit strongly constrained low-level control signals such as fine-grained reference motion and spatiotemporal joint control. While these signals appear dominant, they can suppress other modalities such as text, ultimately compromising overall controllability. To address this, we implement weak-to-strong mixed conditions training strategy: as shown in Fig. 2 initial text-conditioned learning establishes motion-semantic alignment, followed by progressive integration of finer-grained conditions, including reference motion, global motion, and audio signals. This progressive approach enables the model to effectively adapt to different conditions, ensuring high-quality and flexible motion generation while precisely adhering to multimodal conditions.

5 Experiments↩︎

5.1 Implementation Details↩︎

Our method employs a Transformer Encoder architecture [86] with 8 layers and 8 attention heads, featuring a hidden dimension of \(d_{\text{model}} = 1536~(128 \times 12)\), where \(128\) represents the embedding size per each of the \(12\) body parts, and a feedforward dimension of \(3072\). Training is performed on a single H800 GPU through progressive conditioning: starting with text-only training for \(460K\) steps, then adding reference motion for another \(460K\) steps, followed by global spatiotemporal control for \(230K\) steps, and finally incorporating full audio conditions for \(920K\) steps. The corresponding batch sizes are \(48\), \(48\), \(48\), and \(16\), respectively. We optimize with AdamW using an initial learning rate of \(1 \times 10^{-4}\) (reset for new conditions) and a cosine schedule decaying to \(1 \times 10^{-5}\) within the first \(460K\) steps. The default length of motion reference and prediction is \(150\). During training and testing, the reference motion \(\mathbf{c}_{r}\) is provided by the preceding segment of the ground-truth sequence, which shares the same temporal length as the motion to be generated, or is set to null. At inference, this capability is extended to accommodate user-defined motions, enabling customized and interactive generation. More implementation details are provided in supplementary material.

5.2 Quantitative Results↩︎

We evaluate OmniMotion-X on four tasks: Text-to-Motion (T2M), Global Spatiotemporal Controllable Generation (GSTC), Music-to-Dance (M2D), and Speech-to-Gesture (S2G). For T2M and GSTC evaluations, test sets are uniformly sampled across all datasets (\(10\) samples from each). The M2D benchmark integrates test sequences from AIST++ [17], FineDance [16], and PhantomDance [73], with S2G evaluation conducted on BEAT2 [18].

T2M. Tab. 4 shows that previous methods, constrained by small-scale datasets, struggle to generalize in generating diverse and complex motions. In contrast, OmniMotion-X demonstrates a significant advantage, outperforming not only baseline methods trained on small datasets (MDM [1], MLD [40], MoMask [5], MotionCraft [4]) but also MoMask* and MotionCraft*, which are trained on our OmniMoCap-X. Compared to MoMask* and MotionCraft*, OmniMotion-X uses stronger text encoder (T5-XXL [84]), which models complex texts more effectively than CLIP [87]. In addition, it adopts a unified multimodal framework with reference motion during training, enabling the model to learn more detailed whole-body motion representations, leading to better generation quality and generalization. We observed a significant performance enhancement when conditioning on the preceding ground-truth motion at test time, which validates the reference motion’s role as a novel signal for preserving consistency in content, style, and temporal dynamics.

GSTC. We adopt the cross-joint setup from OmniControl [36], simulating spatially dense control by controlling all joints. As shown in Tab. 5, due to the small dataset size, OmniControl [36] struggles to generalize in GSTC task. In contrast, our method demonstrates clear advantages, effectively following spatially dense control signals.

M2D and S2G. We directly compare our method with the MotionCraft [4] on the S2G and M2D tasks. As shown in Tab. 6, OmniMotion-X achieves competitive performance. The slightly higher FID is primarily because the test sets for the S2G and M2D tasks are relatively small and have a limited distribution. In contrast, OmniMotion-X  is trained on OmniMoCap-X, which comprises data from diverse tasks including T2M, M2D, S2G, HHI, HOI, and HSI. This diverse training endows our method with enhanced generative diversity, causing a distributional gap with the test sets of the S2G and M2D tasks.

5.3 Ablation Study↩︎

The ablation study in Tab. 7 reveals two key findings: (1) Removing the progressive weak-to-strong conditioning training strategy compromises text-conditioned alignment, indicating that finer-grained controls can override coarse semantic constraints; (2) Mixed-condition training strategy impairs spatiotemporal control by introducing optimization conflicts due to physical constraints, which underscores the necessity of adopting a weak-to-strong conditioning strategy. Furthermore, our progressive training strategy not only resolves these conflicts but also significantly reduces computational overhead in the initial stages. This is attributed to its design, where multimodal conditions are introduced incrementally as training progresses, resulting in a lower computational load in the early phases. In contrast, a conventional end-to-end approach that jointly trains on all conditions from the outset converges more slowly and faces greater challenges in achieving convergence.

5.4 Qualitative Results↩︎

As shown in Fig. 3, OmniMotion-X handles various multimodal generation, including T2M, M2D, S2G, and GSTC (covering motion prediction, in-betweening, and joint guidance). When combined with reference motion, the model generates conditioned motions that align with the reference. More visualizations are in the supplementary video.

6 Conclusion and Discussion↩︎

This work introduces the OmniMotion-X and the OmniMoCap-X. OmniMotion-X is an autoregressive diffusion model that integrates reference motion and multimodal conditions for precise whole-body motion control. It employs a progressive weak-to-strong mixed conditions strategy to effectively handle multi-granular constraints. OmniMoCap-X is the largest multimodal MoCap dataset, comprising \(286.2\) hours from \(28\) motion capture datasets, unified in SMPL-X with structured and consistent captions across 10 tasks. Experiments demonstrate that OmniMotion-X outperforms baselines, laying a strong foundation for large-scale multimodal motion generation.

Limitation and Future Work. While our method demonstrates versatility across multimodal conditions, it has key limitations. A primary limitation is the absence of explicit conditioning for physical interactions with scenes, objects, or other persons. Consequently, the model struggles to generate plausible interactive motions in complex environments. Future work should thus focus on incorporating interaction-aware conditions, such as conditioning on scene geometry and enforcing explicit physical constraints, to enhance realism and versatility. Furthermore, inference speed remains a significant concern. Our model (27.22M parameters) requires 2.14 seconds per sample, which is slower than text-conditioned diffusion models of a comparable scale. Notably, this inference time only increases to 3.00s when scaling the model tenfold to 355.13M parameters. This indicates the primary bottleneck is not the model’s parameters but the extensive context length arising from aggregating diverse multimodal conditions. Future research should therefore explore more efficient architectures for handling these long-sequence multimodal interactions to accelerate inference.

Acknowledgement↩︎

This work was supported by the Science, Technology and Innovation Project of Shenzhen Longhua District (No. 20260309G23410662).

References↩︎