Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluation
June 24, 2026
Abstract
Automatic singing quality assessment (SQA) requires evaluating lyrical correctness and musical fidelity while handling expressive variations. However, existing systems largely rely on either acoustic cues or lyric transcriptions exclusively, limiting holistic performance evaluation. Furthermore, their integration is non-trivial due to challenges in robust singing transcription amid melisma, vibrato, and tempo elasticity. To this end, we propose MusicJudge, a modality-guided framework for automated SQA that performs block-aligned multimodal analysis by coupling lyric correctness with pitch–rhythm fidelity. It detects semantically meaningful lyric blocks using multi-signal matching that integrates semantic embeddings, lexical similarity, and phonetic alignment. To improve singing audio transcription, we introduce Modality-Guided LoRA for ASR fine-tuning. Experiments across datasets demonstrate strong agreement with human expert judgments and validate the generalizability of MusicJudge.
1 Introduction↩︎
Singing quality assessment (SQA) is a multifaceted problem involving lyrical accuracy, pitch intonation and rhythmic timing. Human experts evaluate vocal performances based on correct lyric pronunciation and adherence to the underlying melodic and rhythmic structure of the music (e.g., Raag in Indian classical music). However, objective evaluation is challenging because singers often introduce acceptable variations, including pronunciation changes and deliberate creative improvisations, further compounded by singer-to-singer differences in vocal timbre, pitch range, and expressive style. Existing computational SQA use isolated metrics like pitch deviation or lyric transcription accuracy (via automatic speech recognition or ASR), which fail to capture the holistic judgments by human evaluators. Moreover, signal-level similarity measures penalize musically valid improvisations, while text-based lyric matching overlooks phonetic and ordering variations in sung content. In this work, we propose a block-aligned multimodal framework for automated SQA that is resilient to stage performance nuances like audience noise, in media res, bridge entry, etc. We analyze performances at semantically meaningful temporal segments (e.g., verses, choruses) in two complementary dimensions: content fidelity and musical quality. These signals are aggregated to produce an interpretable unified singing performance score. Unlike rigid pitch-threshold or transcript-matching systems, our MusicJudge models acceptable expressive variations while preserving musical structure. Our main contributions are:
We present the first block-aligned multimodal SQA framework that jointly models lyrical grounding and music-aware pitch–rhythm fidelity, producing interpretable scores with strong human expert correlation.
We introduce a multi-signal lyric alignment and scoring mechanism that integrates semantic embeddings, fuzzy lexical matching, and phonetic similarity, allowing robust detection and evaluation of sung lyric segments even under ASR errors, pronunciation variation, and melismatic singing.
We introduce Modality-Guided LoRA (MG-LoRA), a music-aware fine-tuning strategy for ASR that integrates pitch, timing, and alignment cues, significantly improving lyric transcription robustness.
Related Work: Early SQA methods rely on handcrafted acoustic features and shallow models [1], with later neural approaches introducing
temporal modeling for vocal dynamics [2], [3]. However, these methods remain
largely limited to acoustic analysis and do not integrate musical structure with lyrical content. Musical representation learning methods capture pitch, tonality, and rhythm through structured audio embeddings and harmonic context [4]–[6], but they do not address lyric-aligned
transcription or melismatic tokenization in singing ASR. Recent singing transcription approaches adapt transformer-based ASR and benchmark robustness under musical variability [7], [8], yet they do not explicitly model pitch continuity or onset cues to handle melisma-induced segmentation errors.
More recent work leverages self-supervised audio representations for singing assessment [9], but acoustic modeling and lyric
decoding remain largely decoupled.
In contrast, our approach integrates pitch contour, duration stability, and onset alignment into ASR fine-tuning objective, enabling segmentation-aware transcription aligned with musical structure and linguistic decoding. To the best of our knowledge, this
is the first work to jointly model these aspects for SQA. Experiments on our SwaraLyrics dataset demonstrate strong agreement with human expert judgments (Spearman correlation of 0.683, 32%\(\uparrow\), Kendall’s \(\mathbf{\tau}\) of 0.499, 41%\(\uparrow\)), while results on Jamendo [10] and SingMOS-Pro further demonstrate the generalizability of MusicJudge.
2 Problem Formulation↩︎
Let \(x(t)\) denote a singing performance waveform defined over \(t \in [0,T]\), where \(T\) is the total duration. Let \(\mathcal{G}=\{\ell^{*}(t), \mathcal{Z}^{*}\}\) denote the global reference comprising ground-truth lyrics and canonical musical structure (e.g., tonal framework). Source separation yields vocal and accompaniment streams: \(x(t) \rightarrow (x_v(t), x_a(t))\).
2.1 Segmentation Under Structural Uncertainty↩︎
2.1.1 ASR-Derived Proto-Segments↩︎
Let \(\{\tilde{S}_n\}_{n=1}^{N}\) denote temporal proto-segments obtained from transcription of \(x_v(t)\): \(\tilde{S}_n = \{ t \mid \tilde{t}_n^{(s)} \le t \le \tilde{t}_n^{(e)} \}\). Here \(\tilde{t}_n^{(s)}\) and \(\tilde{t}_n^{(e)}\) are start and end times induced by ASR token or lyric-line boundaries (proto-segment boundaries). Due to singing-specific phenomena (e.g., vowel elongation, melisma, vibrato), these boundaries may not align with musically coherent units.
2.1.2 Sliding-Window Block Candidates↩︎
To mitigate segmentation uncertainty, overlapping candidates are formed: \(W_m = \bigcup_{n=m}^{m+L-1} \tilde{S}_n\), \(m=1,\dots,N-L+1\), where \(L\) is the window length (in proto-segments).
2.1.3 Block Selection↩︎
Final evaluation blocks \(\mathcal{B}=\{B_k\}_{k=1}^{K}\) are selected from \(\{W_m\}\) based on multi-signal structural coherence, i.e., \(B_k \in \{W_m\}\). Each selected \(B_k\) corresponds to a linguistically and musically coherent unit (e.g., verse, chorus, bridge, or alaap) and admits a temporal representation, \(B_k = \{ t \mid t_k^{(s)} \le t \le t_k^{(e)} \}\), where \(t_k^{(s)}\) and \(t_k^{(e)}\) denote the inferred start-end times obtained from the selected window \(W_m\). Subsequent evaluation operates on \(B_k\).
2.2 Block-Level Fidelity Measures↩︎
For each block \(B_k\), we extract – (a) transcribed lyrics \(\hat{\ell}_k\) from \(x_v(t)\), (b) pitch contour \(\mathbf{p}_k(t)\) and vocal onsets \(\mathbf{o}_k\) from \(x_v(t)\), (c) beat sequence \(\mathbf{b}_k\) from \(x_a(t)\), and (d) global key \(\mathcal{K}\) estimated once from \(x_a(t)\) and shared across blocks. Let \(\ell_k^{*}\) denote reference lyrics aligned to \(B_k\).
Content Fidelity, \(\mathcal{C}_k = \sum_i \alpha_i \, s_i(\ell_k^{*}, \hat{\ell}_k)\), \(\quad \sum_i \alpha_i = 1\), where \(s_i(\cdot)\) denote complementary semantic, lexical, and phonetic similarity measures and \(\alpha_i\) denote weighting coefficients. Pitch Fidelity, \(\mathcal{P}_k = 1 - \frac{1}{|B_k|} \int_{B_k} \rho_p(\delta_p(t; \mathcal{K})) \, dt\), where \(\delta_p(t; \mathcal{K})\) denotes deviation relative to the performance-intrinsic global key \(\mathcal{K}\), and \(\rho_p(\cdot)\) is a bounded expressive penalty function. The specific construction of \(\delta_p(\cdot)\) is described in Sec. 3. Rhythmic Fidelity, \(\mathcal{R}_k = 1 - \frac{1}{|\mathbf{o}_k|} \sum_{o_i \in \mathbf{o}_k} \rho_r(\delta_r(o_i))\), where \(\delta_r(o_i)\) denotes normalized deviation between a vocal onset \(o_i\) and its nearest beat in \(\mathbf{b}_k\). The musical score for block \(B_k\) is: \(\mathcal{M}_k = \beta_1 \mathcal{P}_k + \beta_2 \mathcal{R}_k, \quad \beta_1 + \beta_2 = 1\).
2.3 Structured Aggregation↩︎
Let \(|B_k| = t_k^{(e)} - t_k^{(s)}\) denote the duration of block \(B_k\). We define duration weights: \(w_k = \frac{|B_k|}{\sum_{j=1}^{K} |B_j|}, \hfill\sum_{k=1}^{K} w_k = 1\). The overall performance score is then given by: \[\mathcal{S}(x,\mathcal{G}) = \sum\nolimits_{k=1}^{K} w_k \left[ \gamma_{{}_\mathcal{C}} \mathcal{C}_k + \gamma_{{}_\mathcal{M}} \mathcal{M}_k \right] \label{eq:overall95performance95score}\qquad{(1)}\]
2.3.1 Probabilistic Interpretation↩︎
Each penalty \(\rho(\delta)\) corresponds to a negative log-likelihood under an implicit expressive noise model. Under conditional independence across blocks and modalities, \(\mathcal{S}(x,\mathcal{G})\) is proportional to the log-likelihood of the observed performance given the reference structure, with block boundaries treated as latent structural variables inferred through multi-modal consistency.
3 Methodology↩︎
Our framework targets SQA through block-aligned multi-modal analysis, integrating lyrics-aware content scoring and pitch–rhythm modeling as depicted in Fig. 1.
3.1 Dataset Curation and ASR Adaptation↩︎
To improve lyric transcription robustness, we fine-tune whisper-large-v3 on singing data. Many existing datasets [8],
[11] lack pitch-related information, while some have partial lyrics content (\(\sim4.3k\) English lines in [8]). There is also a significant data gap in the coverage of musical performance recordings alongside their reference lyrics, which is further constrained due to copyright restrictions. So, for
this work, we also curate SwaraLyrics, a corpus of 420 samples (train/val/test: 70/15/15), comprising (a) singing performances (including audience noise, judge commentary), (b) authoritative playback audio, and (c)
native-script lyrics. Here, (b) and (c) serve as ground-truth references during evaluation. Portion used for fine-tuning is either locally recorded by the authors’ institutional band or appropriately licensed. To improve robustness to acoustic variability,
we apply data augmentation (noise mixing, tempo perturbation). SwaraLyrics primarily consists of Indian music, particularly solo songs, spanning diverse moods, genres, eras, and singer demographics.
3.2 Source Separation↩︎
Given an input performance audio, we do source separation using Demucs [12] to obtain vocal and accompaniment. Vocal stream
supports lyric-pitch analysis, while accompaniment stream supports beat-tonal estimation.
3.3 Lyrics Pipeline: Reference-Guided Block Detection and Scoring↩︎
The lyrics evaluation pipeline follows a reference-guided but ASR-driven progression. Block-wise analysis accommodates live performances that may begin from arbitrary song sections or reorder structural parts such as intro, verse, and chorus. We select Whisper [13] as our base ASR model to leverage its inherent pause-based segmentation, which is likely to yield segments parallel to musical phrases. Each segment comprises a raw transcript with a timestamp.
3.3.1 Modality-Guided Fine-Tuning for Singing ASR↩︎
To improve temporally stable and lyrics-faithful transcription under singing-specific acoustic variations, we fine-tune whisper-large-v3 on curated music data (Sec. 3.1),
leveraging Low-Rank Adaptation (LoRA). With MG-LoRA, model is optimized using a composite objective that combines the standard sequence-to-sequence cross-entropy loss, \(\mathcal{L}_{\text{ASR}}\), with authoritative lyrics
as targets, augmented by singing-aware regularization terms. Specifically, we optimize \[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{ASR}} + \lambda_d \mathcal{L}_{d} +
\lambda_p \mathcal{L}_{p} + \lambda_a \mathcal{L}_{a} + \lambda_o \mathcal{L}_{o} \label{eq:whisper95ft95loss95fn}\tag{1}\]
Here, \(\mathcal{L}_{d}\) penalizes unstable token duration in sustained segments, \(\mathcal{L}_{p}\) discourages token boundary proliferation within acoustically smooth fundamental frequency regions, \(\mathcal{L}_{a}\) enforces monotonic alignment consistency, and \(\mathcal{L}_{o}\) encourages token boundaries to coincide with detected vocal onset structure. The coefficients are selected by tuning on a small validation set. For our experiments, these are \(\lambda_d = 0.10\), \(\lambda_p = 0.15\), \(\lambda_a = 0.10\), and \(\lambda_o = 0.05\).
Augmentation strategies (e.g., additive noise, tempo perturbation) are applied to improve robustness to performance variability and background interference (e.g., audience reactions). We then convert it to Faster-Whisper for lower inference latency. The separated input vocal stream is then transcribed using the fine-tuned Whisper model. The output consists of time-aligned tokens grouped into proto-segments. Since no explicit structural labels (e.g., verse, chorus) are available at inference time, song structure is treated as latent.
3.3.2 Sliding ASR Window Creation↩︎
Boundary distortions caused by singing-specific acoustic effects (e.g., vowel elongation, melisma, vibrato) may misalign ASR token boundaries from musically coherent units. Furthermore, Whisper operates in 30s windows, further introducing undesired boundaries. To mitigate this, we group proto-segments into overlapping sliding windows to generate candidate temporal text blocks.
3.3.3 Multi-Signal Block Detection↩︎
Each candidate window is compared against reference lyrics using complementary similarity signals: (a) embedding similarity: sentence-level semantic embeddings measure contextual alignment. (b) fuzzy lexical matching: normalized edit-distance captures surface-form correctness. (c) phonetic matching: grapheme-to-phoneme conversion enables pronunciation-aware alignment. Windows are assigned to reference blocks based on joint multi-signal coherence, thereby refining block boundaries through reference-guided alignment rather than fixed segmentation.
3.3.4 Line-Level Ordered Matching↩︎
Within each detected block, line-level sequential alignment is done to ensure correct progression. We employ HIT/MISS to detect missing lines, repeated/spurious lines, and ordering inconsistencies. This captures structural coverage and lyrical flow beyond surface similarity.
3.3.5 Block Content Scoring↩︎
For each block, three normalized measures are computed – (a) coverage: proportion of reference lines correctly detected, (b) correctness: lexical and phonetic fidelity, and (c) flow: sequential consistency and order preservation. These are combined to produce a block-level content score, \(\mathcal{C}_k\), which contributes to the overall lyrics score after aggregation across blocks.
3.4 Musical Quality Scoring↩︎
We design MusicJudge to reward adherence to melodic principles without penalizing creative deviations. Notably, we do not constrain the performance to be grounded to \(\mathcal{Z}^{*}\). Next, we describe the pitch and rhythm deviation terms \(\delta_p(\cdot)\) and \(\delta_r(\cdot)\) introduced in Sec. 2.
3.4.1 Global Key Estimation↩︎
A single global key \(\mathcal{K}\) is estimated from the accompaniment signal \(x_a(t)\) using chroma-based tonal profile matching. \(\mathcal{K}\) is inferred once per performance and shared across all blocks \(\{B_k\}\). This is done from the accompaniment of the input performance rather than the playback reference \(\mathcal{Z}^{*}\) to avoid penalizing intentional transposition while still enforcing intra-performance tonal consistency.
3.4.2 Pitch Deviation Construction↩︎
For each block \(B_k\), the vocal pitch contour \(\mathbf{p}_k(t)\) is extracted from \(x_v(t)\) using Probabilistic YIN (pYIN) [14]. Voiced frames are retained via pYIN masking. We compute three complementary components – (a) in-key deviation: minimum circular distance between pitch class \(c(t)\) and the scale induced by \(\mathcal{K}\), (b) stability: short-term variance within sustained regions, and, (c) voiced rate: proportion of voiced frames within \(B_k\).
The aggregated pitch deviation for block \(B_k\) is defined as: \(\delta_p^{(k)} = \lambda_1 \overline{d}_{\text{scale}}^{(k)} + \lambda_2 \overline{\sigma}^{(k)} + \lambda_3 (1 - v_k), \quad \sum_i \lambda_i = 1\), where \(\overline{d}_{\text{scale}}^{(k)}\), \(\overline{\sigma}^{(k)}\), and \(v_k\) denote block-averaged in-key distance, stability, and voiced-frame ratio, respectively. The pitch fidelity score is obtained via bounded normalization: \(\mathcal{P}_k = 1 - \rho_p\left(\delta_p^{(k)}\right)\), where \(\rho_p(\cdot)\) applies clipping-based normalization to ensure \(\mathcal{P}_k \in [0,1]\).
3.4.3 Rhythmic Deviation Construction↩︎
For each onset \(o_i \in \mathbf{o}_k\), we compute normalized beat-alignment deviation: \(\delta_r(o_i) = \frac{|o_i - \text{NN}(o_i; \mathbf{b}_k)|}{\tau_k}\), where \(\tau_k\) is the local inter-beat interval. For block \(B_k\), three complementary rhythm statistics are computed – (a) absolute timing error: mean \(|\delta_r(o_i)|\), (b) signed bias: mean \(\delta_r(o_i)\), and (c) stability: standard deviation of onset-level deviations \(\delta_r(o_i)\) within block \(B_k\).
The aggregated rhythmic deviation is: \(\delta_r^{(k)} = \eta_1 \, \overline{|\delta_r|}^{(k)} + \eta_2 \, \mathrm{Std}^{(k)} + \eta_3 \, |\overline{\delta_r}^{(k)}|, \quad \sum_i \eta_i = 1\), where within block \(B_k\), \(\overline{|\delta_r|}^{(k)}\) denotes the mean absolute onset deviation, \(\mathrm{Std}^{(k)}\) or \(\mathrm{Std}\left(\{\delta_r(o_i)\}_{o_i \in \mathbf{o}_k}\right)\) denotes the stability metric, and \(\overline{\delta_r}^{(k)}\) denotes the signed mean deviation (bias). The rhythm fidelity score is defined via bounded normalization: \(\mathcal{R}_k = 1 - \rho_r\left(\delta_r^{(k)}\right)\), ensuring \(\mathcal{R}_k \in [0,1]\).
3.4.4 Block-Level Musical Consistency↩︎
The resulting \(\mathcal{P}_k\) and \(\mathcal{R}_k\) are fused to compute the block-level musical score as described in Sec. 2.2.
3.5 Overall Performance Evaluation↩︎
| Method | \(\mathcal{C}\) | \(\mathcal{M}\) | SingMOS-Pro [8] | |||||||||
| \(\mathbf{\rho} \uparrow\) | \(\mathbf{\tau} \uparrow\) | MSE \(\downarrow\) | MAE \(\downarrow\) | MedAE \(\downarrow\) | \(\mathbf{\rho} \uparrow\) | \(\mathbf{\tau} \uparrow\) | MSE \(\downarrow\) | MAE \(\downarrow\) | MedAE \(\downarrow\) | |||
| SingMOS [20] | - | - | - | - | - | \(0.091\) | \(0.062\) | \(0.56212\) | \(0.60370\) | \(0.45039\) | ||
| UTMOS [21] | - | - | - | - | - | \(0.120\) | \(0.076\) | \(0.24000\) | \(0.39200\) | \(0.29400\) | ||
| DNSMOS [22] | - | - | - | - | - | \(0.201\) | \(0.137\) | \(0.07560\) | \(0.22000\) | \(0.16500\) | ||
| Whisper [13] | \(0.518\) | \(0.350\) | \(0.00960\) | \(0.08010\) | \(0.06250\) | \(0.326\) | \(0.241\) | \(0.06829\) | \(0.20020\) | \(0.16600\) | ||
| + MG-LoRA | \(\mathbf{0.626}\) | \(\mathbf{0.459}\) | \(\mathbf{0.00685}\) | \(\mathbf{0.06073}\) | \(\mathbf{0.04250}\) | \(\mathbf{0.483}\) | \(\mathbf{0.379}\) | \(\mathbf{0.04275}\) | \(\mathbf{0.15129}\) | \(\mathbf{0.10799}\) | ||
| SWIPE [23] | \(0.455\) | \(0.320\) | \(0.00910\) | \(0.07600\) | \(0.06500\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | ||
| CREPE [24] | \(0.482\) | \(0.345\) | \(0.00870\) | \(0.07400\) | \(0.06300\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | ||
| pYIN [14] | \(\mathbf{0.495}\) | \(\mathbf{0.354}\) | \(\mathbf{0.00836}\) | \(\mathbf{0.06673}\) | \(\mathbf{0.03600}\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) | ||
| \(\mathbf{0.683}\) | \(\mathbf{0.499}\) | \(\mathbf{0.00564}\) | \(\mathbf{0.05514}\) | \(\mathbf{0.03633}\) | \(\mathbf{0.483}\) | \(\mathbf{0.379}\) | \(\mathbf{0.04275}\) | \(\mathbf{0.15129}\) | \(\mathbf{0.10799}\) | |||
0.52
| Configuration | ||
|---|---|---|
| Musical Score \(\mathcal{M}\) only | \(0.495\) | \(0.00836\) |
| Content Score \(\mathcal{C}\) only | \(0.626\) | \(0.00685\) |
| Both (\(\mathcal{C} \land \mathcal{M}\)) | \(\mathbf{0.683}\) | \(\mathbf{0.00564}\) |
0.46
| Variant | \(\alpha_{\text{embed}}\) | \(\alpha_{\text{fuzzy}}\) | |
|---|---|---|---|
| NO_EMBED | \(0.00\) | \(0.5\) | \(0.495\) |
| NO_PHONETIC | \(0.70\) | \(0.3\) | \(0.560\) |
| NO_FUZZY | \(0.70\) | \(0.0\) | \(0.608\) |
| FULL_ALL | \(0.55\) | \(0.2\) | \(\mathbf{0.626}\) |
![Figure 2: Singing ASR Performance. Lower is better.: hubert-large-ls960-ft [25],B: wav2vec2-large-960h-lv60 [26],C: whisper-medium,D: whisper-large-v3,E: D + MG-LoRA.](/article/2606.26451/_pdflatex/rqiufxmz.png "fig:")
Quantitative Aggregation: Block-level content and musical scores are aggregated using Eq. ?? . For our experiments, we set \(\gamma_{{}_\mathcal{C}} = 0.55\) and \(\gamma_{{}_\mathcal{M}} = 0.45\), placing slightly higher emphasis on lyrical fidelity. These weights have been selected empirically based on validation-set correlation with human expert ratings.
Natural Language (NL) Feedback Generation: In addition to the scalar score, we generate structured natural-language feedback. We provide an LLM with: (a) ordered sequence \(\{ \mathcal{C}_k \}_{k=1}^{K}\), (b) ordered sequence \(\{ \mathcal{M}_k \}_{k=1}^{K}\), (c) ASR transcription \(\hat{\ell}(t)\), and (d) reference lyrics \(\ell^{*}(t)\). Block-wise score sequences preserve localized performance variations (e.g., weaker chorus, stronger verse), enabling the production of section-aware NL rather than relying solely on the global aggregate.
0.46
| Genre | Base | MG-LoRA | ||
| WER | CER | WER | CER | |
| Classical | \(0.800\) | \(0.671\) | \(\mathbf{0.689}\) | \(\mathbf{0.563}\) |
| Folk | \(0.742\) | \(0.624\) | \(\mathbf{0.497}\) | \(\mathbf{0.405}\) |
| Ghazal | \(0.682\) | \(0.592\) | \(\mathbf{0.571}\) | \(\mathbf{0.482}\) |
| Bhajan | \(0.642\) | \(0.534\) | \(\mathbf{0.529}\) | \(\mathbf{0.421}\) |
| Pop | \(0.562\) | \(0.423\) | \(\mathbf{0.451}\) | \(\mathbf{0.319}\) |
0.50
| Language | Base | MG-LoRA | ||
| WER | CER | WER | CER | |
| English | \(0.4052\) | \(0.2627\) | \(\mathbf{0.2218}\) | \(\mathbf{0.2234}\) |
| Mandarin | \(0.7400\) | \(0.1990\) | \(\mathbf{0.6100}\) | \(\mathbf{0.1062}\) |
| Hindi | \(0.7477\) | \(0.4850\) | \(\mathbf{0.5474}\) | \(\mathbf{0.4382}\) |
| Punjabi | \(0.9431\) | \(0.6347\) | \(\mathbf{0.6705}\) | \(\mathbf{0.3854}\) |
| Bengali | \(0.9375\) | \(0.5153\) | \(\mathbf{0.7500}\) | \(\mathbf{0.4365}\) |
4 Experimental Results↩︎
Configuration: We conduct experiments on Linux workstation equipped with 2\(\times\) NVIDIA Tesla V100-SXM2 GPUs (32 GB each), using GPU acceleration for ASR fine-tuning and inference. We fine-tune
whisper-large-v3 using parameter-efficient LoRA adapters (\(r=16\), \(\alpha=32\), dropout \(=0.05\)) applied to the attention projection layers
(q_proj, k_proj, v_proj, out_proj). Audio inputs are limited to 12 s with a maximum transcription length of 256 tokens. Training is performed for 10 epochs with a learning rate of \(10^{-4}\), batch-size 1 with 16-step gradient accumulation. The lyrics pipeline time-based ASR windows (\(L=28\) s, stride \(=10\) s) over proto-segments,
discarding windows with \(<25\) characters, block matching uses embedding/lexical/phonetic weights \((0.55,0.20,0.25)\) with threshold \(0.72\). Musical
analysis uses pYIN (\(C2\)–\(C6\), frame \(=2048\), hop \(=256\)) and onset detection (\(\text{pre\_max}=3\), \(\text{post\_max}=3\), \(\delta=0.15\)).
Quantitative Validation: We sample a sequence of 120 vocal performances, scored by \(\ge3\) human expert judges independently on a scale of 1-10 with the final score computed as the per-clip mean across
judges and then assess these performances using MusicJudge to derive an overall score. Then, we rank them based on these two score sequences and derive two orderings. Table 1 shows how
closely the ordering inferred by MusicJudge correlates with human expert ground truth. Evaluation on SingMOS-Pro is limited to models supporting lyrics/content evaluation (\(\mathcal{C}\)), as
it lacks ground-truth for music score evaluation (\(\mathcal{M}\)). On SwaraLyrics, NL feedback via gpt-oss-120b [27] (Sec. 3.5) yields a all-MiniLM-L6-v2 [28] cosine similarity of 63.97 with expert comments.
Component impact analysis: Table ¿tbl:tab:ablation95study? presents key ablations. Table [tbl:tab:lyrics95pitch95rhythm95ablation] showcases the impact of content and musical components on SQA. In an exemplary instance, where content \(\mathcal{C}\), pitch \(\mathcal{P}\), and rhythm \(\mathcal{R}\), singularly emit scores 0.829, 0.490, and 0.491 respectively, the overall score is computed as: (a) 0.829 (for \(\mathcal{C}\) only), (b) 0.490 (for \(\mathcal{M}\) only; \(55\mathpunct{:}45\) weightage), and (c) 0.677 (for \(\mathcal{C} \land \mathcal{M}\); \(55\mathpunct{:}25\mathpunct{:}20\) weightage). Aggregation of (c) bears the highest \(\rho\) of 0.683. Table [tbl:tab:lyrics95pipeline95ablation] further breaks down the components of the lyrics pipeline, proving that multi-signal block detection approach outperforms individual signals. Fig. 2 shows that the singing transcription accuracy improves by \(29.87\%\) due to MG-LoRA over the second best (averaged across SwaraLyrics, SingMOS-Pro, and Jamendo [10]). The base ASR \(\rho\) of 0.518 improves to 0.583 (+\(\mathcal{L}_{\text{ASR}}\)), 0.597 (+\(\mathcal{L}_{d}\)), 0.616 (+\(\mathcal{L}_{p}\)), 0.622 (+\(\mathcal{L}_{a}\)), 0.626 (+\(\mathcal{L}_{o}\)), showing a maximum benefit due to \(\mathcal{L}_{p}\) after \(\mathcal{L}_{\text{ASR}}\).
Generalization of MG-LoRA: Tables [tbl:tab:genre95consistency], [tbl:tab:language95generalization] present evaluations across the top-5 SwaraLyrics genres and 5 languages representing Whisper performance extremes.
Qualitative Analysis: Table 2 shows that MG-LoRA improves transcription in cases like sustained note prolongation, melisma, ornamentation (like gamakas), portamento. Our NL feedback is shown in Supplementary: https://neelam472.github.io/MusicJudge/Supp.pdf .
| Ground Truth Lyrics | Whisper (Base) | Whisper + MG-LoRA (Ours) |
|---|---|---|
| (Uljhan meri suljha de, chahoon main aana) | (Uljham meri sulja de chaho main aana) | (Uljhan meri sulja de, chahoon main ya na) |
| (Mera koi ehsaas hai jaise) | (Mera koi ahsaas hai jese) | (Mera koi ehsaas hai jaise) |
| डरते हो क्यूँ? ज़रा पास तो आओ ना (Seene se tum mere aa ke lag jao na darte ho kyun? Zara paas to aao na) | दर्दे हो क्यों जरा पास तो आओ ना (Seene se tum mere aakhe lag jao na darde ho kyon jara paas to aao na) | सीने से तुम मेरे आ के लग जाओ ना,दरते हो क्यूँ,ज़रा पास तो आओ ना (Seene se tum mere aa ke lag jao na, darte ho kyun, zara paas to aao na) |
5 Conclusion↩︎
We introduce MusicJudge for automatic SQA, providing a practical foundation for assistive training tools, synthetic music evaluation, and scalable judging support in music competitions. On SwaraLyrics and SingMOS-Pro, MusicJudge achieves Spearman \(\rho = 0.683|0.483\), outperforming lyric-only and music-only baselines by \(+31.9\%|+48.2\%\) and \(+38.0\%|-\), respectively. Coupling linguistic and musical cues yields >9.1% more reliable SQA than single-modality evaluation. Proposed multi-signal block detection further improves intra-song boundary localization (\(\rho = 0.626\), \(+2.96\%\) over the second best). Further, MG-LoRA improves lyric transcription robustness across genres (\(20.1 \pm 7.52\%\) WER\(\downarrow\)) and languages (\(27.7 \pm 10.87\%\) WER\(\downarrow\)). Future work may explore diarization-aware multi-singer MG-LoRA modeling.
6 Generative AI Use Disclosure↩︎
Research usage of gpt-oss-120b is for natural language feedback generation as described in Sec. 3.5 (examples presented in Supplementary). Other generative AI usage is strictly
limited to permitted re-formatting of tables/plots.