1 Introduction↩︎

The rapid development of generative music models [1]–[3] has led to a surge in AI-generated music, creating an urgent need for automatic evaluation aligned with human aesthetic preferences. The rapid expansion of AI-generated music not only creates diverse application scenarios for automatic aesthetic evaluation, but also elevates it to a critical component in modern music generation pipelines, with applications in content filtering, recommendation, and reinforcement learning from human feedback (RLHF).

Despite these advances, current models often fail to produce music that aligns with human expectations, exhibiting issues such as structural incoherence and limited expressiveness [1], [4]. This gap highlights the need for reliable aesthetic evaluation models that can accurately reflect human perception.

Music aesthetic assessment is inherently challenging. It is subjective and exhibits substantial inter-individual variability, while also requiring the modeling of relatively stable and generalizable criteria grounded in domain knowledge. Moreover, aesthetic evaluation in music relies heavily on expertise: trained musicians can provide fine-grained and technically grounded assessments, explaining not only whether a piece is preferred but also why it is perceived as aesthetically effective or flawed. This dual requirement of subjectivity and expertise makes large-scale, high-quality annotation particularly challenging.

However, existing datasets for music aesthetic evaluation remain limited in scale, number of annotators, and coverage of perceptual dimensions, and often rely on single-score annotations without richer supervision such as multi-dimensional ratings or textual feedback.

To address these challenges, we make the following contributions:

A multidimensional music aesthetic annotation framework. We define an evaluation framework with one overall score and 10 fine-grained perceptual dimensions, supported by annotation guidelines to ensure consistency.

A large-scale music aesthetics dataset and benchmark. We construct MADB, a dataset of 9999 tracks annotated by trained annotators, each rated by 9–11 annotators across all dimensions, with additional textual comments.

A unified benchmark for music aesthetic assessment. We evaluate multiple pretrained models, including CLAP-based approaches, under a unified framework. Results show that current models capture only partial aesthetic information, revealing the difficulty of the task and the need for improved representations.

2 Related Work↩︎

2.1 Music Aesthetic Evaluation and Datasets↩︎

Early studies on music aesthetics in the MIR community mainly focused on perceptual modeling with handcrafted musical features. For instance, Marias et al.  adopted musical tension dynamics as a surrogate indicator of aesthetic quality. Subsequent work by Xin Jin et al. [5], [6] further combined traditional feature engineering with neural networks to regress aesthetic scores from hand-designed acoustic features. Nevertheless, these methods are constrained by conventional MIR paradigms and exhibit limited representation capacity compared to modern Transformer-based models.

In contrast, aesthetic evaluation in image and video domains has achieved remarkable progress via Transformer architectures trained on large-scale expert-annotated corpora. Xin Jin’s team has built comprehensive aesthetic models for visual content [7]–[9] and constructed large-scale datasets annotated by professionals from art and film institutions, enabling the learning of high-level aesthetic representations.

In the music domain, several benchmark datasets have been proposed, such as MusicEval [10] and SongEval [11], which provide basic benchmarks for music aesthetic evaluation. However, these datasets suffer from limited data scale, insufficient annotator participation, and incomplete coverage of perceptual dimensions. Most existing datasets only provide single-value aesthetic ratings and lack rich supervisory signals such as multi-dimensional perceptual scoring and textual subjective feedback, making them inadequate for capturing the complexity of human music aesthetics.

2.2 Music Representation Learning↩︎

Self-supervised learning has substantially advanced music representation learning in recent years [12], [13]. Pre-trained models including MuQ [14] and MERT [15] learn universal audio representations from large-scale music corpora and yield promising transfer performance across downstream MIR tasks. Moreover, models such as PANNs [16] and HTSAT [17] have also been widely adopted. These pre-trained models can effectively capture diverse acoustic and musical characteristics, including timbre, rhythm, and structural patterns, which are inherently correlated with human aesthetic perception.

However, they are not explicitly optimized for perceptual or aesthetic modeling, nor do they incorporate structured aesthetic dimensions and subjective human evaluation signals. It remains an open problem to what extent general-purpose music representations can align with fine-grained human aesthetic judgments.

2.3 Music-Text Cross-Modal Learning↩︎

Multimodal research has made considerable progress in aligning music and textual representations. CLAP [18] learns unified audio-text embeddings via contrastive learning. As a foundational framework, CLIP [19] also verifies the efficacy of large-scale cross-modal alignment across perceptual domains. Extended frameworks such as CLAMP3 [20] further integrate structured textual attributes to enhance fine-grained cross-modal alignment.

Music-text multimodal models offer strong capability to associate audio content with semantic and contextual descriptions. Most existing methods primarily concentrate on coarse-grained attributes such as music genre and general mood. By contrast, rich textual subjective feedback, which is rarely exploited in current literature, holds great potential for modeling fine-grained perceptual and aesthetic traits, highlighting a promising direction for injecting human-centered perceptual knowledge into music understanding systems.

3 MADB Dataset and Benchmark↩︎

3.1 Aesthetic Annotation Framework↩︎

To model the multi-dimensional nature of music perception, we design a structured annotation framework consisting of 10 perceptual dimensions and one overall score. These dimensions capture complementary aspects of musical experience, including melody, arrangement, rhythm, structure, performance, and production-related attributes.

Not all dimensions are applicable to every track. For non-applicable cases (e.g., singing-related attributes for instrumental music), annotators assign a score of 0. These entries are retained as non-applicable labels and excluded from downstream model optimization.

The 10 dimensions cover key aspects of music production, including melody perception and emotion, rhythm and structure, arrangement quality and expressiveness, performance-related attributes such as emotion and technical skill, and post-production effects. These dimensions jointly reflect both low-level acoustic qualities and high-level perceptual judgments.

The dimensions are organized to align with the major stages of the music production pipeline, including composition, arrangement, performance, and post-production. This formulation provides a structured decomposition of music aesthetics and enables systematic analysis of different perceptual factors.

In addition to fine-grained annotations, we collect an overall aesthetic score representing the annotator’s holistic judgment. This score is not defined as a fixed aggregation of individual dimensions, but instead reflects a higher-level perceptual integration process.

3.2 Semantic Annotations↩︎

In addition to numerical ratings, MADB includes semantic annotations to enrich the representation of music aesthetics. Specifically, each track is associated with short textual comments and tag-based labels.

Annotators are required to provide brief textual comments(around 50 words) describing their aesthetic impressions of the music. These comments serve as complementary explanations for the numerical scores and provide insight into the reasoning process behind human aesthetic judgments. The comments were originally written in Chinese and subsequently translated into English using Qwen2.5-Instruct. Comments those less than 10 words will be ignored.

We further define two tag-based annotation dimensions: genre and emotion. Each track is assigned up to two genre labels and up to two emotion labels from 10 genre labels and 6 emotion labels. These tags provide high-level semantic context and facilitate future research on multi-modal learning and cross-modal alignment. The complete list of tags is provided in the Appendix.

The combination of numerical ratings, textual comments, and semantic tags enables MADB to support both perceptual modeling and multi-modal aesthetic analysis.

3.3 Data Sources↩︎

The MADB dataset is constructed from four complementary sources to ensure diversity in musical style, production quality, and generation paradigm.

The first subset consists of 2,799 manually collected tracks curated from online platforms. This subset includes both professionally produced music (e.g., canonical works such as famous classical musics) and widely consumed popular content (e.g., short-video platform music), as well as tracks with varying production quality such as royalty-free or publicly available music. This design aims to capture a broad spectrum of real-world listening scenarios.

The second and third subsets consist of AI-generated music. Specifically, 1,000 tracks are generated by the Suno system, including a subset conditioned on existing song lyrics, and 1,800 tracks are generated by the Levo system. These samples introduce diverse generative styles and enable the evaluation of aesthetic perception in AI-generated content.

The fourth subset consists of 4,400 tracks from the Muchin dataset[21], which further expands coverage across musical genres and compositional styles.

This multi-source construction results in a dataset that spans both human-composed and AI-generated music, covering a wide range of production qualities, stylistic variations, and emotional characteristics. Such diversity is critical for establishing a robust and generalizable benchmark, enabling evaluation across different creation paradigms and varying aesthetic characteristics.

We provide all annotation data and a large subset(7030 songs)of publicly available audios, includes 2630 songs generated by AI, which can be downloaded on our huggingface page¹, and 4400 songs from open-source Muchin dataset, which can be downloaded on Muchin’s github page². Other copyrighted audios are not allowed to be released due to licensing constraints.

3.4 Annotators↩︎

The dataset is annotated by 30 annotators, each treated as an independent rating source in our analysis.

Each music track is independently evaluated by an average of 10 annotators, enabling robust estimation of inter-rater agreement and stable aggregation of aesthetic scores. This multi-annotator design helps reduce individual bias and supports reliable statistical analysis of perceptual consistency.

To ensure annotation quality, we adopt a three-stage quality control pipeline. The first stage consists of the primary annotation process. In the second stage, 20% of the annotations are randomly sampled for review and consistency checking. In the third stage, an additional 10% of the annotations are randomly selected for further validation by a different reviewer. The review and validation stages are conducted independently to reduce confirmation bias and improve reliability.

Annotators were recruited from multiple music conservatories, media universities, and industry partners across several institutions. All annotators had received at least three years of formal professional music training, and all held a bachelor’s degree or higher. This ensured that raters possessed sufficient musical knowledge and listening expertise to provide consistent and informed evaluations. Quality control and secondary review were conducted by senior experts with formal music training and professional evaluation experience, each with more than ten years of professional music education background.

3.5 Annotation Statistics and Distribution↩︎

We analyze the distribution of aesthetic scores across all dimensions. As shown in Fig. 2, scores are predominantly concentrated in the mid-to-high range (approximately 2.5–4.5), with relatively few extreme values. This reflects a common characteristic of human aesthetic evaluation, where most music is perceived as moderately good rather than strongly polarized. Such a distribution implies a limited dynamic range, making fine-grained regression particularly challenging.

We further evaluate annotation consistency using the Intraclass Correlation Coefficient (ICCK) The results are all around 0.8, indicate strong multi-rater reliability across most dimensions, supporting the stability and validity of the aggregated scores.

4 Experiments↩︎

4.1 Pretraining and Evaluation Framework↩︎

We build upon the CLAP framework [18] by incorporating semantic supervision from both textual comments and structured tags (e.g., genre and mood).

All comments are originally written in Chinese and translated into English using Qwen2.5-Instruct [22] for CLAP’s training, both original Chinese comments and translated English comments are provided. For each audio sample, multiple annotator comments are encoded and aggregated via mean pooling to form a unified textual representation. In parallel, genre and mood tags are embedded into low-dimensional vectors and projected into the same embedding space. The two representations are fused through a learnable gating mechanism to produce a joint semantic embedding.

To enhance audio–semantic alignment, we perform a pre-adaptation stage based on contrastive learning. Specifically, the audio encoder is trained to align with the fused semantic representation using a cosine similarity-based objective. This stage relies only on comments and tags, without using aesthetic scores, thereby avoiding label leakage while encouraging the model to capture perceptual and semantic information.

For downstream evaluation, we adopt a unified regression framework across all models. Audio representations extracted from pretrained encoders are temporally pooled into fixed-length embeddings, followed by a Transformer-based regression head to predict aesthetic scores. Models are trained using mean squared error (MSE) as the primary objective, optionally combined with a correlation-based loss to better align predictions with human perceptual rankings.

We evaluate multiple encoder variants, including MuQ, MERT, and CLAP, as well as CLAP adapted with comment-only and comment–tag supervision. All models share the same downstream architecture, differing only in encoder and pretraining strategy, enabling a controlled comparison of representation quality. The encoders are frozen during downstream training.

We randomly split the dataset into training and validation sets with a fixed ratio. All models are optimized using Adam with a consistent set of hyperparameters across experiments. During training, zero values indicate non-applicable dimensions and are excluded from optimization, with separate models trained when necessary. All clap-based experiments are conducted with 4 fixed random seeds(42,120,5,2500) to ensure reproducibility and to reduce the influence of stochastic factors such as data shuffling, parameter initialization, and mini-batch sampling. Muq and Mert experiments are under seed 42.

All experiments are conducted on NVIDIA A100 GPUs for embedding extraction and pretraining, and NVIDIA Tesla V100 GPUs for downstream training.

2pt

4.2 Result Performance↩︎

Table 2 presents the performance of different models across 11 aesthetic dimensions under four evaluation metrics with mean and std. Bold indicates the best result among all methods, and underlined values denote the best performance within CLAP-based methods.

Overall, models based on MuQ and MERT significantly outperform CLAP-based variants across all metrics, achieving substantially lower MSE and higher correlation scores (LCC, SRCC, and KRCC). In particular, MuQ consistently achieves the best performance on most dimensions, indicating that current audio representation models specifically designed for music understanding are more suitable for aesthetic prediction tasks.

In contrast, CLAP-based models achieve comparatively lower performance, particularly on perceptual and structural dimensions such as arrangement and structure perception.

Despite the use of strong pretrained encoders, the overall performance remains far from saturation, indicating that multi-dimensional aesthetic prediction is a challenging task.

4.3 Effect of Comment and Tag-Based Adaptation↩︎

We further compare three CLAP-based[18] variants: the original pretrained CLAP, CLAP adapted using textual comments (CLAP+Com), and CLAP adapted using both comments and tag fusion (CLAP+Tag&Com). Across most dimensions and metrics, both adapted variants outperform the original CLAP model, as indicated by consistently better MSE and higher correlation values.

This improvement demonstrates that incorporating human-provided semantic signals—especially free-form comments—provides useful supervisory information for aesthetic prediction. Compared to comment-only adaptation, the addition of structured tag information further improves performance on several dimensions (e.g., melody and rhythm perception), indicating that tags offer complementary and more stable semantic priors.

Importantly, this adaptation is performed without using any aesthetic scores, ensuring that the observed improvements are not due to label leakage but rather to enhanced representation learning. These findings confirm the value of combining textual and structured annotations, and suggest that multi-source semantic supervision is a promising direction for improving music aesthetic modeling.

4.4 Multimodal LLM-Based Aesthetic Prediction↩︎

To analyze the contribution of different modalities to music aesthetic evaluation, we conduct a zero-shot probing experiment using Qwen2-Audio-7B-Instruct. We consider three input configurations: Audio only, Comment&Tag, and Audio + Comment&Tag.

For each sample, the model predicts an overall aesthetic score (1–5) based on an instruction prompt aligned with the perceptual dimension definitions in Table 1. To ensure fair comparison, each configuration uses a separate input pipeline, explicitly controlling which modalities are provided without describing missing inputs.

4pt

4.5 Analysis of Multimodal Input Configurations↩︎

As shown in Table 2, the Audio+Comment&Tag and Comment&Tag setting significantly outperforms Audio-only, achieving strong correlation. This indicates that MADB’s human textual feedback provides highly informative signals for aesthetic judgment. This is because comments already encode high-level perceptual judgments and evaluative semantics produced by human annotators, effectively serving as a compressed representation of aesthetic reasoning that is closely aligned with the target scores. English comments are used in these experiments, which prove the reliability of translations.

Combining modalities (Audio + Comment&Tag) yields further improvements on several metrics, suggesting that audio provides complementary information beyond textual descriptions. However, the relatively small gains also highlight the difficulty of extracting high-level aesthetic judgments directly from raw audio.

Overall, these results show that (1) human comments serve as a strong supervision signal, and (2) multimodal inputs lead to the most consistent performance. Notably, all results are obtained in a zero-shot setting, reflecting intrinsic modality informativeness rather than dataset-specific training.

4.6 Discussion↩︎

The gap between CLAP-based models and music-specific encoders (MuQ, MERT) suggests that aesthetic evaluation requires not only semantic alignment but also strong intrinsic audio representation capabilities.

At the same time, the consistent gains from comment and tag-based adaptation, as well as the complementary signals from LLM-derived text representations, indicate that human annotations provide meaningful information that can partially bridge this gap.

Overall, the results validate the proposed benchmark as both challenging and informative, revealing clear limitations of existing models while highlighting the potential of leveraging richer annotation sources and lightweight language modeling for future improvements.

5 Limitations↩︎

Despite its scale and richness, MADB has several limitations. First, the dataset exhibits an imbalanced distribution of music styles, with a predominance of popular genres. While this reflects real-world listening patterns, it may bias models and limit generalization to underrepresented styles.

Second, textual comments are originally written in Chinese and translated into English for compatibility with pretrained language models, which may introduce noise or subtle semantic shifts.

Third, annotations are provided at the whole-track level without temporal granularity. In practice, aesthetic quality often varies across different segments within a piece, and the absence of segment-level annotations limits the ability to model such dynamic perceptual variations. Incorporating fine-grained temporal annotations is therefore an important direction for future work.

Fourth, the current tag system is relatively coarse and does not capture fine-grained stylistic distinctions (e.g., subgenres such as house or post-rock). Expanding the tag taxonomy with more detailed and hierarchical categories is an important direction for improving semantic expressiveness.

Finally, the dataset does not explicitly capture causal relationships between perceptual dimensions and overall aesthetic judgments, leaving this as an open problem for future research.

6 Conclusion↩︎

We study music aesthetic assessment, an increasingly important problem in the era of generative music. Our analysis shows that it is inherently challenging due to its multi-dimensional, subjective, and temporally dependent nature, with concentrated score distributions, moderate inter-annotator agreement, and complex relationships across perceptual dimensions.

To support this direction, we introduce MADB, a large-scale dataset and benchmark with multi-dimensional ratings, textual comments, and semantic tags. Based on this benchmark, we show that while modern pretrained models capture partial aesthetic signals, a substantial gap to human perception remains.

These findings suggest that music aesthetic evaluation requires not only strong audio representations but also deeper integration of perceptual, semantic, and contextual information. Textual comments provide complementary signals but are insufficient alone, highlighting the need for multimodal approaches.

We hope MADB will advance research in human-centered music understanding and improve both generative models and evaluation frameworks. Future work may explore tighter integration with large language models and more fine-grained temporal annotations to better capture human aesthetic reasoning. It also opens opportunities for studying structured aesthetic reasoning and controllable generation grounded in human perception. While offering benefits for recommendation and creative tools, such systems may introduce bias and should be used responsibly.

7 Technical appendices and supplementary material↩︎

7.1 Inter-Annotator Agreement Across Dimensions↩︎

Figure 4 presents the ICCK values across all perceptual dimensions. Overall, the agreement is consistently high, with most dimensions exceeding 0.80, indicating strong multi-rater reliability.

Structurally grounded dimensions, such as melody_emotion and enunciation_and_singing_skill, achieve relatively higher agreement (e.g., above 0.83), suggesting that annotators share consistent judgments on well-defined perceptual attributes. In contrast, more subjective or context-dependent dimensions, such as sound_effect_perception, exhibit comparatively lower agreement (0.72), reflecting the inherent ambiguity in evaluating production-related effects.

Notably, the overall score also demonstrates high reliability (ICCK = 0.84), indicating that despite its holistic nature, annotators maintain consistent global judgments. These results support the stability and validity of the annotation process.

7.2 Score Distribution Across Data Sources↩︎

Figure 5 illustrates the mean overall scores across different data sources. We observe noticeable variation in average ratings: internet and suno samples achieve relatively higher scores (around 3.5), while levo samples receive lower ratings (approximately 3.18).

Despite these differences in mean values, all sources exhibit non-negligible standard deviations, indicating substantial intra-source diversity. This suggests that each source contains a mixture of high- and low-quality samples, rather than being dominated by uniformly strong or weak content.

These findings highlight that while source-level biases exist, the dataset maintains sufficient variability within each source, contributing to the overall complexity and realism of the benchmark.

7.3 Annotators Background↩︎

7.4 Tags Definition↩︎

References↩︎