1 Introduction↩︎

In recent years, the advent of deep learning has driven rapid advancements in generative modeling, significantly impacting multimedia fields [1]–[9]. Data synthesis using generative models is crucial for a range of applications [10]–[16]. One of the most captivating applications of generative modeling is music-to-dance generation [17]–[20], which involves translating musical elements such as tempo, rhythm, and melody into coherent and expressive dance sequences. Recent advancements in deep learning architectures and generative models have made it possible to achieve this translation, enabling the creation of synchronized and harmonious dance movements. However, existing solutions [18]–[22] often lack an interactive and engaging experience, limiting their potential to inspire creativity and facilitate learning.

To address this gap, we present DanceDuo, an novel application designed to harness the power of generative models for in-the-wild music-to-dance generation. By leveraging cutting-edge diffusion models [17], DanceDuo provides users with an interactive platform to create dance sequences synchronized with a wide range of musical tracks. The application also features a diverse selection of humanoid models, allowing users to visualize AI-choreographed movements in rich detail. Beyond simply showcasing generated dances, DanceDuo also allows for the importation of personal dance videos, enabling users to compare their own movements with those generated by the AI by integrating human pose estimation models, thus facilitating an engaging learning experience that inspires users to explore their dancing potential. We also design a scoring formula specifically that fosters engagement and motivation for all users, regardless of their dance proficiency, to encourage people to do excises via dancing. Moreover, this interaction helps in promoting creativity, as users can reflect on their own dance styles in conjunction with AI-generated sequences.

To evaluate the efficacy and user experience of our DanceDuo, we conducted a comprehensive user study. The study utilized an open-ended survey format to gather qualitative feedback on various aspects of the application, including usability, interface design, and feature effectiveness. Findings revealed that participants generally perceived DanceDuo as intuitive and engaging, especially praising the dance comparison feature for its interactive and motivational qualities.

The main contributions of this work are as follows:

• Integration of a music-to-dance model facilitates the creation of an interactive platform that enhances user engagement and creativity in dance.

• Exploration of interactive features, such as the ability to compare a user’s dance performance with AI-generated dances, provides valuable insights for skill improvement.

• Scoring formula is designed to foster engagement and motivation for all users, regardless of their dance proficiency.

• Comprehensive quantitative and qualitative evidences showcase the capabilities of DanceDuo.

2 Related Work↩︎

2.1 Music-Driven Dance Generation↩︎

Early research in music-driven dance generation primarily employed retrieval-based techniques [23], [24]. However, recent methodologies have shifted towards motion synthesis approaches, leveraging advanced network architectures such as CNNs [25], [26], RNNs [27], [28], GANs [29], [30], and Transformers [21], [31], [32]. Notably, the VQ-VAE framework [33] has been employed to generate temporally coherent dance sequences [20], [34]. Recently, diffusion models have shown significant promise in generating high-quality images, videos, and motion sequences [14], [35]–[40], with the EDGE framework demonstrating a music-conditioned motion denoising approach [18].

2.2 Dance Interactive Systems↩︎

The integration of digital tools in dance creation has expanded significantly with technological advancements. Notable examples include Merce Cunningham’s “Biped” [41], which combined computer-captured dance movements with hand-drawn graphics to create animated and abstract dance characters. Burton et al..[42]’s work on Laban Movement Analysis (LMA) highlights the importance of understanding movement dynamics in enhancing the relationship between dancers and technology.

Several studies have developed interactive environments that support the creation, practice, and performance of choreography. For instance, a virtual reality-based tele-immersive environments [43] allows dancers to engage in immersive experiences, while interactive augmented reality systems enhance live performances [44]. Additionally, Sheppard et al..[45] introduced an application enabling multiple participants to interact regardless of physical distance, leading to the concept of tele-immersive dance (TED), which fosters a highly interactive collaborative environment.

Within the realm of human-computer interaction (HCI), various technologies have been employed to capture and analyze dance movements. These include vision-based motion capture systems [46], [47], depth cameras like Microsoft Kinect [48], [49], and inertial motion sensors [50]. Moreover, a significant body of research has focused on translating movement assessment results into effective feedback [51], [52]. By integrating these technological advancements, the field of dance continues to evolve, offering new opportunities for creativity and expression.

2.3 3D Pose Estimation↩︎

Estimating human body pose in3D from a single image or video is a significant challenge in computer vision. Methods can be categorized into multi-stage and single-stage techniques. Multi-stage methods involve detecting individuals and then estimating body pose parameters [53], [54]. Recent advancements leverage supervisory signals to improve geometric and dynamic consistency [55], [56]. Alternative single-stage solutions [57], [58] directly estimate body joint positions and group them into individuals, such as the one-stage approach proposed by ROMP [59].

3 DanceDuo Design↩︎

3.1 Overview↩︎

DanceDuo is a versatile platform that leverages the power of diffusion models to deliver an interactive experience centered around AI-generated dance movements. By harnessing the power of these models, DanceDuo generates real-time dance sequences that synchronize seamlessly with a diverse selection of music tracks across various genres. The platform encompass a wide range of dance styles, including Break, Lock, LA Style Hip-hop, Waack, Street Jazz, Pop, Middle Hip-hop, House, Krump, and Ballet Jazz. These categories align with the AIST++ dataset [21], which is widely recognized for its applications in music-to-dance generation tasks.

Users can explore these AI-choreographed movements through different humanoid models, including Boy, Girl, Mousy, and Robot (Fig. 1), offering a rich and diverse visualization of the generated dance sequences and allowing them to tailor their dance performance to their preferences.

In additionally, the platform also enables users to import their personal dance videos. This feature allows users to compare their own movements with those produced by the AI. This comparison can inspire and motivate users to engage more actively in dancing, fostering a deeper connection between the user and the art of dance.

3.1.1 Findings on Music to Dance Generation↩︎

Generating realistic and engaging music-conditioned dance sequences is a complex challenge, which requires addressing two crucial aspects. The first is the temporal challenge, where the generated dance sequences should synchronize with the given music’s beat and rhythm while maintaining naturalness. Equally important is the spatial challenge, which demands that the generated dance moves be both physically realistic and aesthetically pleasing.

To address these challenges, various methods have been developed, which can be divided into two main categories: autoregressive and non-autoregressive approaches. Autoregressive methods [22], [27], [60] generate future motions based on the previous ones, effectively capturing the flow of the dance over time. However, they often struggle to produce long-term sequences, resulting in a freezing effect where movements become repetitive and static. On the other hand, non-autoregressive methods [18], [19], [61] handle each motion segment independently and use simple temporal smoothing techniques to smooth out the transitions between segments, but can lead to abrupt and unnatural transitions between poses.

To overcome these limitations, we utilize DanceFusion [17], a conditional diffusion model designed for generating realistic dance movements that are conditioned on music as well as past and future motions. Particularly, the diffusion process is represented as a Markov chain process, gradually adding noise to real data. At each step of this process, Gaussian noise is injected, with the amount of noise controlled by a predefined schedule, \(\{a_t \in (0, 1)\}_{t=1}^T\). With latent \(\{z_t\}_{t=0}^T\) that follow a forward noising process \(q(z_t|x)\), where \(x\sim p(x)\) is drawn from the data distribution, the forward noising process is mathematically defined as: \[\label{eq:4461} q(z_t | x) \sim N(z_t; \sqrt{a_t}x, (1 - a_t)I).\tag{1}\]

In the setting with music conditioning \(m\), past motions \(x_p\) and future motion \(x_f\), the author reverse the forward diffusion process by learning to estimate \(\hat{x}_\theta (z_t, t, m, x_p, x_f) \approx x\) with model parameters \(\theta\) for all \(t\). They optimize \(\theta\) with the "simple" objective introduced in Ho et al..[62]: \[\mathcal{L}_{\text{simple}} = \mathbb{E}_{x, t} \left[ \left\| x - \hat{x}_\theta (x_t, t, m, x_p, x_f) \right\|_2^2 \right].\]

DanceFusion effectively combines the strengths of autoregressive and non-autoregressive techniques, allowing for the creation of arbitrarily long performances. The authors of this model have also demonstrated its effectiveness in enhancing the quality of generated dance sequences. This capability makes DanceFusion suitable for applications involving diverse and spontaneous music selections. By leveraging DanceFusion, we aim to generate high-quality dance sequences that are both temporally and spatially coherent, providing a more engaging and realistic experience for users.

3.1.2 Motion Representation↩︎

To capture the complexities of human movement, we need an effective motion representation. Common parameterizations like Euler angles, axis angles, and quaternions describe 3D rotations but may miss subtle body deformations during movement.

To overcome this, we use statistical mesh models and adopt the SMPL model [63], [64], chosen for its compatibility with DanceFusion, our dance generation model, and its detailed representation of human form, capturing body position, orientation, and surface details, as shown in Fig. 2.

The SMPL model relies on two parameter sets: pose and shape. Pose parameters (\(\theta\)) define each joint’s orientation within a 24-joint hierarchy, where each joint’s rotation is relative to its parent.

3.2 Implementation↩︎

3.2.1 Motion Importing↩︎

The process of importing and visualizing the dance files generated by DanceFusion within the Blender environment is a crucial step. To facilitate this process, we utilize a specialized Blender add-on known as “SMPL to FBX,”². This add-on enables us to seamlessly import the SMPL model files generated by DanceFusion into Blender, where we can further refine and visualize the dance movements.

After refining the dance sequences in Blender, we export the files in FBX format, which is a widely supported format for 3D models and animations. Finally, we import the FBX files into Unity for visualization and integration with our application DanceDuo.

3.2.2 Motion Estimation from Video↩︎

To estimate the SMPL pose from video, we employ the ROMP model, a state-of-the-art, one-stage method for real-time, monocular, multi-person 3D mesh recovery [59]. ROMP’s streamlined architecture enables efficient estimation of SMPL 3D pose parameters from a single image, making it an ideal choice for our application.

To optimize computational efficiency and processing speed, we adopt a selective frame extraction approach. Instead of extracting SMPL parameters from every frame, we extract them at regular intervals and interpolate the intermediate frames. This strategy significantly reduces the computational load while maintaining the required processing speed for our application.

Let \(\theta_{j,f}\) denotes the rotation of joint \(j\) at frame \(f\). To estimate the rotation of a joint at frame \(f\), we first estimate the pose at the preceding frame \(f - 1\) and the succeeding frame \(f + 1\), obtaining \(\theta_{j, f - 1}\) and \(\theta_{j, f + 1}\), respectively. We then calculate the rotation of joint \(j\) at frame \(f\) using the average of the two estimated rotations:

\[\theta_{j, f} = \frac{\theta_{j, f - 1} + \theta_{j, f + 1}}{2}.\]

By applying this interpolation technique to all 24 joints, we obtain the interpolated pose at frame \(f\). In cases where \(f + 1\) exceeds the total number of frames, we estimate the pose at frame \(f\) using the available information.

3.2.3 Scoring System↩︎

For the scoring system in DanceDuo, we aimed for more than simple similarity calculations, focusing on user engagement and motivation regardless of dance skill level. We designed a formula that scores between \(6\) and \(8\) out of \(10\), helping users feel accomplished even while learning.

Our approach transforms the raw "similarity score" into a beta distribution, and making it like a normal curve. This keeps most scores in a balanced range, avoiding extremes like \(0\) or \(1\), which could be discouraging.

We calculate accuracy based on two primary metrics:

1. Mean Per Joint Position Error (MPJPE): MPJPE measures the average distance between corresponding joints in predicted and ground truth poses. Given a set of predicted joint positions \(P\) and ground truth joint positions \(G\), the MPJPE is calculated as follows: \[\text{MPJPE}(P, G) = \frac{1}{N} \sum_{i=1}^{N} \| P_i - G_i \|,\] where \(N\) is the number of joints, and \(\| P_i - G_i \|\) denotes the Euclidean distance between the \(i\)-th joint’s predicted position \(P_i\) and the ground truth position \(G_i\).

2. Mean Per Joint Angle Error (MPJAE): MPJAE measures the average angular error between corresponding joints’ orientations in predicted and ground truth poses. Given the predicted joint angles \(A_p\) and ground truth joint angles \(A_g\), the MPJAE is calculated as follows: \[\text{MPJAE}(A_p, A_g) = \frac{1}{N} \sum_{i=1}^{N} \theta(A_{p_i}, A_{g_i}),\] where \(N\) is the number of joints, and \(\theta(A_{p_i}, A_{g_i})\) represents the angular difference between the \(i\)-th joint angle’s predicted orientation \(A_{p_i}\) and the ground truth orientation \(A_{g_i}\). The angular difference \(\theta\) can be computed using the dot product of unit vectors representing the joint angles: \[\theta(A_{p_i}, A_{g_i}) = \cos^{-1} \left( \frac{A_{p_i} \cdot A_{g_i}}{\|A_{p_i}\| \|A_{g_i}\|} \right).\]

We map the combined similarity score to a Beta distribution using the Percentage Point Function (PPF). The Beta distribution offers two key benefits: flexibility in shaping the curve with parameters \(\alpha\) and \(\beta\), allowing us to create a bell-shaped curve similar to the normal distribution, and its ability to map values to the \([0, 1]\) range, ideal for scoring. We set \(\alpha = 3\) and \(\beta = 1\) to achieve a well-balanced bell curve.

To calculate the accuracy at each frame, we consider both MPJPE and MPJAE. For each frame \(f\), we compute the accuracy by averaging MPJPE and MPJAE: \[\text{accuracy}_f = 1 - \frac{\text{MPJPE}_f + \text{MPJAE}_f}{2}.\]

The raw accuracy score is then refined using the Beta PPF: \[\text{score}_f = \text{PPF}(\text{accuracy}_f).\]

Finally, the total score for a sequence is the average of all frame scores, scaled to a 100-point system.

4 User Study↩︎

4.1 Setup↩︎

Five participants were invited to interact with the application and complete a survey on their user experience. The survey featured open-ended questions on usability, interface design, and feature effectiveness.

4.2 User Workflow↩︎

The application follows a user-centric design for an intuitive experience, as illustrated in Fig. 3, showing the journey from launch to final options.

Users start by selecting a music track and a humanoid model in a gallery interface, as shown in Fig. 4. They then watch the generated dance performance, as shown in Fig. 5 (a) and can record and upload their own dance video. The app processes this video, compares it to the AI-generated dance, and provides a similarity score. Users can view their score and watch the AI and their own videos side-by-side, as shown in Fig. 5 (b).

4.3 Results↩︎

The user study provided valuable insights into the application’s strengths and potential areas for improvement:

4.3.0.1 Usability and Interface Design

Participants generally found the application intuitive and user-friendly, praising its simple design and seamless navigation. However, some users suggested providing more detailed instructions or context-sensitive help to enhance user understanding and interaction.

4.3.0.2 Interactive Features

The dance comparison feature was consistently identified as one of the most engaging aspects of the application. Users praised its interactive nature and motivational impact, highlighting the value of comparing their own dance performances with AI-generated dances. Some users suggested that providing more specific, actionable feedback could further improve the feature.

4.3.0.3 Variety and User Engagement

Participants appreciated the diverse options available within the application, particularly the wide array of music genres and humanoid models. Users found the ability to customize these elements entertaining and meaningful, enhancing their engagement and enjoyment.

4.3.0.4 Output Quality

Feedback on the quality of the generated dance animations was generally positive, with users satisfied with the fluidity and the visual representation of the movements. However, some participants pointed out areas for improvement, particularly regarding synchronization between the music and the generated dance movements. Refining this aspect could further enhance the overall quality of the output.

5 Conclusion↩︎

In this paper, we introduced DanceDuo, a novel platform designed to generate in-the-wild, AI-driven dance sequences synchronized with music tracks, using diffusion models. By integrating a diverse selection of dance styles and humanoid models, DanceDuo enables users to explore and engage with AI-generated choreography in a personalized and interactive manner. The platform’s ability to import user-generated dance videos for comparison with AI-choreographed movements further enhances user engagement, promoting learning, creativity, and skill improvement.

The user study demonstrated that DanceDuo provides an intuitive and engaging experience, particularly highlighting the effectiveness of its dance comparison feature. While the feedback was generally positive, it also revealed several areas for improvement, including the need for real-time features, enhanced usability guidance, and support for custom audio tracks.

In future work, we aim to address these limitations by integrating real-time processing capabilities and expanding the platform’s functionality to include more customizable features. These enhancements will further improve the user experience, encouraging deeper engagement and creativity in the intersection of music, dance, and AI.

Acknowledgment↩︎

This research is funded by University of Science, VNU-HCM, under grant number CNTT 2024-16.

References↩︎

---

1. Corresponding author.↩︎

2. https://github.com/softcat477/SMPL-to-FBX↩︎