1 Introduction↩︎

Music exists digitally across three modalities: audio recordings, symbolic encodings (e.g., MIDI, MusicXML, ABC notation), and sheet music images. Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in audio and visual understanding [1]–[3], and their application to music is an active area of MIR research [4], [5]. Evaluating how well these models handle musical tasks is both timely and consequential: it determines how much of our field’s research agenda can be delegated to general-purpose models, and where purpose-built MIR systems remain indispensable. Yet rigorous evaluation across diverse repertoires, modalities, and research contexts remains an open methodological challenge.

Existing benchmarks for evaluating MLLMs on music suffer from three major limitations. First, large static benchmarks are impractical: they are resource-intensive to evaluate; accuracy values age quickly as new models are released; results may not transfer beyond the styles and genres included; benchmark data may leak into future training corpora, invalidating comparisons; human annotation of ground truth is extremely costly while LLM-assisted annotation raises quality concerns; no single benchmark can cover music’s diversity, and new models arrive faster than evaluation can keep pace; and copyrighted material creates legal constraints on what data can be included.

Second, some benchmarks claiming to evaluate music understanding (e.g., by asking questions about audio) do not require actual perception of musical content, as their questions are answerable from textual cues alone [5], [6].

Third, most benchmarks cover a single modality —audio [5], [7], [8], sheet music images [9], [10], or symbolic encodings [11], [12] — with only a few recent exceptions spanning two modalities [13]–[15]. This prevents cross-modal comparison and leaves a key question open: does poor MLLM performance on music reflect modality-specific perceptual limitations, or deeper conceptual deficits independent of representation?

To address these limitations, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench). Rather than a fixed benchmark, MusICA-MetaBench automatically generates on-demand benchmarks from user-provided musical data. Given a piece in symbolic representation (MusicXML), we programmatically extract ground-truth answers for predefined piece-level question templates using the music21 library [16], yielding question–answer pairs grounded in that specific piece.

1.1 MusICA-MetaBench Design Principles↩︎

Question templates are grounded in music-theory feature-recognition skills (see 1), framing music perception as the identification and relation of pitch and rhythmic patterns, cross-culturally universal features [17]–[19].

When input data is multimodal and piece-aligned (the same piece represented simultaneously as audio, sheet music image, and symbolic file), the same question can be posed across all three modalities with an identical ground-truth answer, enabling direct cross-modal comparison on the same underlying musical content.

We experimentally determine optimal benchmark sizes to be large enough to yield statistically significant comparisons at a desired effect size, yet small enough to keep evaluation fast and inexpensive. By comparing model performance against a no-input condition (question only, no musical file) and a noise-input condition (random noise in place of the musical file), we verify that solving the benchmark requires musical perception.

We demonstrate MusICA-MetaBench by applying it to two multimodal datasets, ChoraleBricks [20] and ChoralSynth [21], and evaluating 8 state-of-the-art MLLMs on the generated benchmarks.

The intended use of MusICA-MetaBench is not objective MLLM comparison, but actionable diagnostics for stakeholders applying MLLMs to their own musical data, for example, MIR researchers assessing model readiness for specific tasks. By enabling evaluation on user-provided repertoire, MusICA-MetaBench reveals not only whether MLLMs fail, but where: across modalities, perceptual dimensions, and repertoires.

The pipeline (benchmark generation and response evaluation) is programmatic (LLM-independent), ensuring controllability and interpretability, and restricting the role of (M)LLMs solely to that of evaluation subjects.

We regard this as a first step rather than a complete solution, but believe this paradigm makes MLLM evaluation on music faster, cheaper, domain-flexible, and more interpretable. We release all code and generated benchmarks.¹

The paper is organized as follows: 2 surveys recent music benchmarks for LLMs; 3 describes the question templates and benchmark generation pipeline; 5 details benchmark-size calibration; 6 demonstrates that perception is required to solve the benchmark; 7 demonstrates practical usage and provides guidelines for custom datasets; and 8 concludes with contributions, limitations, and future directions.

2 Related Work↩︎

Recent benchmarks for music understanding (2024–2025) predominantly evaluate (M)LLMs via question answering (QA) [5], [7], [9]–[11], [13], [14], [22], with multiple-choice (closed QA) being the dominant format.

A known weakness of closed QA is that some benchmarks can be partially solved without perceiving any musical content, through reasoning over answer options alone [5], [6]. [5] demonstrated that a text-only LLM reaches 56% accuracy on MuChoMusic, far above the 25% random baseline. Proposed mitigations include filtering out questions solvable by text-only LLMs [6] and systematic distractor (incorrect option) generation designed to match the plausibility of the correct answer [5].

Most benchmarks assess a single musical modality: audio [5], [7], [22], [23], visual notation (PDF/JPG score) [6], [9], [10], [24], or symbolic representation, either ABC notation [11], [12] or multiple symbolic formats (ABC, Humdrum, MEI, MusicXML) [25]. A few recent benchmarks cover two modalities: SSMR-bench [13] and MSU-bench [14] jointly target symbolic and visual notation, while MuseBench [15] covers audio, sheet images, and textual knowledge questions, though cross-modality performance is not comparable due to differing question types. To our knowledge, no benchmark evaluates MLLMs across more than two musical modalities.

Benchmark construction involves a trade-off between scale and quality. Human-curated benchmarks are small (e.g., 9 questions in [25]; 372 QAs in MusicTheoryBench [12]), costly to produce, and may still be solvable without perception [8]. An alternative to manual annotation are automated approaches, such as converting caption datasets to QA via LLMs [7], using predefined templates with manual [14], [15] or programmatic ground truth [13], or generating questions from synthetic musical data [10], which yield larger benchmarks (e.g., MusiXQA with 130k QAs [10]), but raise questions about validity of the QAs (when LLMs are involved in their creation) or transferability to real music (when synthetic musical data is used). Benchmark size is rarely discussed explicitly: human-curated sets are too small for stable estimates, while automatically generated sets are often too large to permit cost-effective evaluation of expensive models.

Finally, all existing benchmarks are static. Results may not transfer to music beyond the included examples, benchmark content can leak into future models’ training data invalidating comparisons, and large benchmarks are resource-intensive to run — a significant problem given how quickly new MLLMs are released. For a domain as diverse as music, a truly universal benchmark may be unattainable; even if one were constructed, new models would likely emerge faster than they could be evaluated.

3 Methodology↩︎

MusICA-MetaBench is organized around a predefined set of question templates (Tab.1), each with an implemented function that extracts the ground truth and distractor options from a MusicXML. The resulting question-option pairs apply across symbolic files, sheet images, and audio, provided the dataset contains piece-aligned data in those formats, enabling cross-modal performance comparison.

Unlike SSMR-bench [13], which also programmatically extracts ground truth from symbolic notation (ABC notation), MusICA-MetaBench does not produce a single static benchmark. Instead, it is a pipeline for benchmark instance generation over arbitrary user-provided data, and extends evaluation to the audio modality.

3.1 Question Template Design↩︎

We ground our question templates in music theory, specifically in the foundational knowledge that marks the transition from intuitive listening to formalized structural awareness [26]. This analytical threshold, corresponding to the “analysis” level in Bloom’s taxonomy [27], is central to major music pedagogy systems (Kodály [28], ABRSM, GCSE) and provides a shared vocabulary for musical communication.

One author with formal musical training selected skills reflecting standard music learning evaluation structures (see 1), then designed question templates satisfying four requirements: (i) answerable from all modalities (audio, image, symbolic); (ii) ground truth programmatically extractable from MusicXML; (iii) instantiable into multiple questions per piece; (iv) piece-level, i.e., answerable with the whole piece as context.

The skill categories also carry diagnostic value: e.g., low accuracy on audio pitch recognition makes successful audio-to-score transcription unlikely; performance on the image modality may indicate OMR capability, etc.

0.2em

3.2 Meta-Benchmark Generation Pipeline↩︎

The pipeline (see 1) takes three inputs: the set of predefined question templates (see 1), an ontology (a dictionary of possible values for evaluated musical elements²), and most importantly, the musical data from which the benchmark is built. In step 1, question templates are instantiated into questions by replacing wildcards with all combinations of ontology values. In step 2, a Cartesian product of questions and musical pieces is formed. For each (question, piece) pair, a Python extraction function receives the piece’s MusicXML representation and the instantiating ontology values, and returns a ground truth answer and a pool of distractor options.³ In step 3, filtering removes invalid items (e.g., those querying a note index that does not exist in a given piece), subsampling balances the benchmark by skill/category and reduces the benchmark size (see 5), and finally, each (question, piece) pair is crossed with all modalities — symbolic (ABC), audio, and sheet image — so the same question is posed with the musical material in each modality, enabling direct cross-modal performance comparison.

Each benchmark item is a 4-tuple: (question, musical file, correct answer, 4 distractor options),⁴ where musical files are unimodal — no item combines, e.g., audio and image simultaneously.

3.3 Evaluation Methodology↩︎

We adopt a closed-question multiple-choice paradigm (reliability and reproducibility) while partially mitigating cue-reliance via a “none of the other options is correct” (NOTA) option.

All experiments use 5 options (1 true, 3 distractors + NOTA). NOTA is the correct answer in 20% of questions (matching the \(1/5\) uniform prior), where ground truth is replaced by an additional distractor. Options are shuffled randomly at generation time [7], [8], as (M)LLMs are sensitive to option ordering [29]. Responses are parsed to options (A–E) using regex matching and prompt formatting from MMMU-Pro [6].

Models were selected for ease of use, performance, modality coverage, cost, and speed, and accessed primarily via the OpenRouter API (except Qwen3 Omni 30B, run locally). The evaluated MLLMs are: Gemini 2.0 FL, 2.5 FL, 3.1 FL (Flash Lite; 3.1 in preview), Gemini 3.1 Pro (preview), and Qwen3 Omni 30B (Qwen3 Omni 30B A3B Thinking). For GPT and Mistral, which lack a single model covering all modalities, we use complementary pairs reported as unified systems prefixed “agg-”: agg-GPT-4o (gpt-4o-audio-preview / gpt-4o), agg-GPT-5 (gpt-audio-mini / gpt-5-image-mini), and agg-Mistral (voxtral-small-24b-2507 / mistral-small-3.2-24b-instruct), where the first model handles audio and the second handles visual/symbolic input.

4 Dataset: ChoraleBricks↩︎

We use the ChoraleBricks dataset [20] as input for MusICA-MetaBench for most of the experiments. It includes multitrack recordings of ten chorales, each with soprano, alto, tenor, and bass parts played by a variety of wind instruments, along with sheet music in MusicXML and PNG, ideal for MusICA-MetaBench. For generating a MusICA-MetaBench instance from ChoraleBricks, we use MusicXML for ground truth extraction, convert it to ABC notation⁵ for symbolic modality, and use reference mixes for audio, and sheet music PNGs for visual modalities.⁶ See 1 for a pipeline schema, showing item counts in ChoraleBricks benchmark generation.

5 Calibration of Benchmark Size↩︎

The full ChoraleBricks-derived benchmark would contain 12,666 items (4,222 \(\times\) 3 modalities), making exhaustive evaluation computationally impractical. Selecting an appropriate benchmark size requires balancing statistical power against evaluation cost. Because MusICA-MetaBench targets real-world model selection rather than fine-grained comparison, users need not detect arbitrarily small accuracy differences—price, speed, and openness also inform choice. We therefore frame the problem as: for a user-defined minimum relevant effect size \(E\) (e.g., \(E = 5\%\)), what is the smallest benchmark size that reliably detects a difference of \(E\) percentage points?

5.1 Calibration Methodology↩︎

For each candidate size \(s \in \{\fpeval{3 * 5 * 1}, \fpeval{3 * 5 * 5}, \fpeval{3 * 5 * 10}, \fpeval{3 * 5 * 20}, \fpeval{3 * 5 * 50}, \fpeval{3 * 5 * 100}\}\) (1, 5, 10, 20, 50, or 100 questions per category-modality combination), we draw 10 random subsets without replacement using distinct random seeds.⁷ Evaluating several models on these subsets yields 10 accuracy measurements per model per size. Normality verified by Shapiro-Wilk test [30], vast majority of cases passing.

To assess detectability, we shift the accuracy vector of the hypothesised superior model so that the mean gap equals exactly \(E\) % while preserving the original variance structure, then apply a paired \(t\)-test to the shifted values, and repeat this every ordered model pair across \(E \in \{1, 2, \dots, 20, 25, 30\}\) pp. For each \(E\), we identify the smallest \(s\) at which all pairwise tests are significant (\(\alpha = 0.05\), Bonferroni-corrected).

5.2 Calibration Results↩︎

We used 6 MLLMs for (see 2 and 3.3). 2 shows the minimum detectable effect size per benchmark size. A size of \(s = \fpeval{3 * 5 * 20}\) suffices for a 6% effect size. Validation on observed model differences confirms this: all accuracy gaps of at least 6% are significant at \(s = \fpeval{3 * 5 * 20}\), and in practice even gaps of at least 3.5% are.

2 reports accuracy standard deviation across 10 subsampled instances per model-size combination. Standard deviation decreases as benchmark size grows, while mean accuracy remains stable, supporting the hypothesis that a practically small benchmark yields a reliable estimate of full-benchmark performance.

6 Is Perception Required?↩︎

To validate our distractor generation, we test whether answering benchmark questions requires perception of the musical content; that the benchmark does not suffer from the “not really listening” problem [5], extended to “not looking at the image” and “not reading the symbolic score” in our multimodal setup). We compare MLLM performance against two ablation setups:

• no-input (text-only): the musical file is omitted entirely; compatible with both MLLMs and text-only LLMs (cf.[5]).

• noise-input: Gaussian noise replaces audio/image, and a structurally valid but lexically nonsensical ABC score replaces the symbolic modality.

In 3 we observe a clear performance drop when switching from the normal setup to no-input or noise-input. Both baseline setups are above random chance, however, which indicates we could do better in the choice of distractors: e.g., using the methodology from [5].

7 Benchmark Test-Drive↩︎

We demonstrate MusICA-MetaBench in practice on the ChoraleBricks and ChoraleSynth datasets.

7.1 Model Comparison Results on ChoraleBricks↩︎

We set the minimum effect size of interest to 6 % and consider three modalities: audio recording, sheet image (PNG), and ABC notation. Based on 5.2, this requires \(s = \fpeval{3 * 5 * 20}\) (300 questions total, 100 per modality). We generate one benchmark instance using a fresh random seed and evaluate 8 MLLMs for overall (3), and per-modality accuracy (4).⁸

Gemini 3.1 Pro (released in preview on Feb 19th, 2026) is the clear winner. Its margin of improvement suggests that music perception skills previously out of reach for MLLMs [4] may soon become attainable — reinforcing the need for flexible, actionable benchmarking as we may be entering a period of time where some MLLMs will become competitive for some music perception applications, and decision-making between specialised models and MLLMs may grow more complex.

Gemini 3.1 Pro is, however, astronomically expensive and slow compared to all other models. If speed matters, Gemini 3.1 FL is the likely choice, though the benchmark should be re-run at a larger size before concluding: the accuracy gap between Gemini 3.1 FL and the next fast competitor, GPT-4o, is not safely significant at \(s = \fpeval{3 * 5 * 20}\) (see 2).

Cross-modal results (4) show that most performance variation across models stems from the symbolic modality. In audio, all models perform similarly except Gemini 3.1 Pro and GPT-5, the latter failing to reach random-chance performance. Image modality scores are closely clustered across models. Averaged across models, \(\text{accuracy}(\text{symbolic}) > \text{accuracy}(\text{image}) > \text{accuracy}(\text{audio})\).

7.2 Generalisation to Another Dataset: ChoralSynth↩︎

To verify MusICA-MetaBench generalises beyond ChoraleBricks, we apply it to ChoralSynth [21], a synthetic multitrack choral dataset (20 pieces). Ground truth and distractors are extracted from the provided MusicXML and converted to ABC notation for the symbolic modality. We mix the audio tracks to a single recording per piece;⁹ render PDF score from MusicXML via MuseScore and concatenate into a single PNG per piece.

Repeating the 10-run size calibration (2 models: Gemini 2.0 FL and Gemini 3.1 FL), standard deviation decreases with benchmark size as on ChoraleBricks, but is slightly higher at fixed sizes, which we attribute to greater item variance arising from the larger number of distinct pieces in ChoralSynth.

On a single benchmark instance (\(s = \fpeval{3 * 5 * 20}\)) with the same 8 MLLMs, the normal setup achieves 28% mean accuracy versus 17% for no-input and 20.1% for noise-input (both near the random baseline of 20%), confirming perception is required.

Full results are in 4. Gemini 3.1 Pro is again the top model, but overall performance is substantially lower than on ChoraleBricks. Performance drops are most pronounced in the audio and image modalities. For images, the likely cause is that concatenated scores sometimes span multiple pages, making them impractical to read. For audio, voice-specific labels (e.g., “Bassus Ch1-F4”) may be impossible to identify from a mixed recording.

7.3 Guidelines for Custom Datasets and Formats↩︎

MusICA-MetaBench supports any classical tonal repertoire consisting of one or more monophonic voices (choral, strings, winds, etc.). For such datasets, the ontology mapping the system’s chords, intervals, and rhythms, is generated automatically from music21.

MusicXML is required for ground truth extraction. Other symbolic formats (MIDI, ABC notation, LilyPond, Humdrum, etc.) may also work but have not been verified and may require adjustments. Non-symbolic modalities must be piece-aligned with the MusicXML source (same piece, same key, etc.). Modalities can vary in format and quality (e.g., rendered, printed scan, or handwritten scan of a score) and can be compared as submodalities, provided all are aligned to the same MusicXML file. Missing modalities may be synthesised (e.g., rendering a score image or converting to ABC notation). New question templates can be added, but require implementing the corresponding ground truth extraction method.¹⁰

8 Conclusions and Future Work↩︎

MusICA-MetaBench is a programmatic pipeline that generates QA benchmarks from user-provided musical data, targeting musical perception across modalities (audio, sheet images, symbolic formats) and scalable to optimal size. It offers an alternative to large static benchmarks, avoiding data leakage and generality constraints. We provide proof-of-concept evaluations on ChoraleBricks and ChoralSynth, noting that the new Gemini 3.1 Pro MLLM achieves a significant leap in musical capability on both. The pipeline is configurable and extensible to new datasets and formats. Total LLM experiment costs were 730 USD, dominated by leading-edge models.

Limitations. Distractor difficulty may be slightly overestimated in the no-input condition: the prompt instructs models to analyze a provided musical work, causing some to refuse when no file is present; a prompt such as “try to guess the correct answer” would be more appropriate.¹¹ Individual skill scores are not yet mapped to grading schemes such as ABRSM. Extending the pipeline to new datasets may require adjustment of ground-truth extraction methods, as validation was limited to ChoraleBricks and ChoralSynth.

Future work includes harder distractor selection [5], open-ended question formats, an extended question template set, musically multimodal questions requiring simultaneous perception of audio and notation, questions with multimodal answer options (e.g., multiple audio clips), and extension to polyphonic instruments (piano) and beyond tonal music.

As the musical capabilities of MLLMs grow, assessing what they can and cannot do in individual use-cases is becoming an important part of music processing system design. We believe the paradigm of MusICA-MetaBench will be a practical way to obtain actionable insights on what “AI” really is (and isn’t) capable of.

9 Acknowledgments↩︎

This work was supported by the project “Human-centred AI for a Sustainable and Adaptive Society” (reg. no.: CZ.02.01.01/00/23_025/0008691), co-funded by the European Union, and partially by the SVV project number 260 821. The work described herein has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2023062).

10 AI Usage Statement↩︎

Large language models were used in three capacities in this work. First, as coding assistants during development of the MusICA-MetaBench pipeline, including code drafting, debugging, and refactoring. Second, as writing assistants for drafting and rewriting portions of this manuscript. All AI-assisted outputs were critically reviewed, edited, and verified by the authors, who take full responsibility for the content of this paper. Third, and most centrally to this work, (M)LLMs served as the subjects of evaluation: the models assessed by MusICA-MetaBench are themselves large language models, evaluated on their music perception capabilities as described in the experimental sections. The pipeline itself (both benchmark generation and response evaluation) is programmatic and LLM-independent.

11 Ethics Statement↩︎

Data and copyright. MusICA-MetaBench is demonstrated on the ChoraleBricks and ChoralSynth datasets, both of which are publicly available research datasets distributed under open licences. A key motivation of our framework is precisely to avoid the copyright concerns inherent in large static benchmarks: by enabling researchers to generate benchmarks from their own data, MusICA-MetaBench shifts responsibility for data licensing to the end user, who evaluates models on material they are already authorised to use.

Bias and generalisability. Our question templates are grounded in Western tonal music theory and pedagogy (e.g., Kodály, ABRSM, GCSE frameworks). The framework therefore reflects a culturally specific analytical tradition and may not generalise to non-Western musical systems, microtonal music, or oral traditions. Users should be aware of this scope when drawing conclusions about model capabilities.

Intended and unintended use. MusICA-MetaBench is intended as a diagnostic tool for MIR researchers assessing MLLM readiness for specific tasks, not as a definitive ranking of model quality. Misuse of benchmark results to make broad claims about model superiority, or to guide high-stakes deployment decisions without additional validation, is discouraged.

Environmental cost. Total API costs for the reported experiments were approximately 730 USD, dominated by frontier models. We encourage users to apply the benchmark-size calibration procedure described in this paper to minimise unnecessary inference compute.

Human subjects. No human subjects were involved in this research. No personally identifiable information was collected or processed.

References↩︎