1 Introduction↩︎

Audio recognition systems such as Shazam [1] are highly effective for identifying exact recordings, but recognizing the underlying tune across different performances is a different problem. This distinction is especially important in jazz: a standard such as Autumn Leaves can appear in many keys, different tempo, different arrangements, and in many improvisational contexts. In this paper, we study whether modern audio representations support this form of cross-performance tune recognition. The task is difficult for several reasons. First, jazz performances often devote long stretches to improvisation, where cropped local windows may not contain the head melody. Second, different standards can share common harmonic progressions or similar melodic fragments. Third, recordings by the same group can be acoustically and stylistically similar across performances of different standards, which causes problems for nearest-neighbor retrieval methods.

Our work makes three contributions. First, we construct a curated filtered benchmark subset from the Jazz Trio Database [2] for standard recognition across performances. Second, we compare spectrogram-based training, supervised probing of frozen embeddings, and embedding-based retrieval under the same evaluation protocol. Third, we analyze retrieval-based classification and demonstrate that nearest neighbors often retrieve performer identity rather than tune identity, and explore a lightweight supervised contrastive projection to reduce performer-biased retrieval.

2 Related Work↩︎

Fingerprinting systems for audio such as Shazam [1] are very effective for exact recording identification from short and noisy excerpts, however they are designed to match a given signal to a recording already present in the database rather than recognizing a new performance of the same underlying composition. With this in mind, jazz standard recognition is therefore closer to cover or version identification [3], [4], which is concerned with retrieval of different renditions of the same work using features that are invariant to changes in key, tempo, timbre, and arrangement. Recent version-identification systems have increasingly adopted embedding-based retrieval formulations to improve scalability while preserving work-level invariance [4].

Recent developments in music foundation models further motivate our approach. Earlier work showed that pretrained audio representations can transfer well across downstream MIR tasks with shallow heads or no fine-tuning, including codified-audio language-model features from Jukebox-based MIR [5] and broader comparative studies of supervised and unsupervised music embeddings [6]. MERT [7] scales masked self-supervised pretraining with acoustic and musical teachers and reports strong transfer across various music-understanding tasks, while MuQ uses Mel quantization targets and shows broad downstream gains with improvements with larger-scale pretraining. A closely related reference in approach is [8], who evaluate multiple audio foundation models through probing and lightweight fine-tuning across culturally diverse tagging datasets. Our setting is complementary because it probes jazz tune identity across performances rather than tag prediction across cross-cultural musical traditions.

This work complements previous approaches in the literature by using jazz standards as a deliberately hard retrieval target where unlike fingerprinting the goal is not exact recording recognition, and unlike generic cover-song benchmarks many query windows may not contain an explicit statement of the melody. Methodologically, it is closest to recent evaluation-first studies of music foundation models that compare probing and lightweight adaptation, however our target is cross-performance jazz standard identification under heavy improvisation.

3 Dataset, Evaluation and Experiment Setup↩︎

3.0.0.1 Dataset construction.

Given our focus on jazz performances, we use the Jazz Trio Database [2] and curate a subset of the data intended for standard-level recognition. The JTD dataset has 1294 performances from various standards, however the distribution of the data is long-tailed: more than 85% of standards from the dataset have fewer than 3 performances from distinct bandleader/pianist groups. From this observation, we construct a filtered dataset based on the following approach: we first canonicalize standard titles to merge spelling and naming variants. Afterwards, we collect all standards with at least 4 performances from distinct groups. This threshold is chosen to support our leave-one-performance-per-standard evaluation so that in each fold one performance of every standard is held out for testing, while the remaining performances provide training and validation examples. Finally, for each standard, we keep at most one performance from any given group, reducing leakage from repeated performances by the same ensemble. The resulting training subset is not perfectly class-balanced, but it greatly reduces the long-tailed structure of the original dataset and ensures multiple performances and a balanced test setup. Figure 1 shows counts of various standards within the subset. With these adjustments, the evaluation dataset contains 16 standards with 79 performances, with more metadata described in Table 1. For each selected recording, we assign a canonical standard label and segment the audio into fixed-length windows. We use the fixed standards list for all reported experiments.

3.0.0.2 Windowing strategy.

Each recording is converted to 24 kHz mono audio and split into 10-second windows with 5-second hop. Each window inherits the standard label of its parent performance. This creates many training examples per performance, but these windows are highly correlated within a recording. Therefore, we report performance-level metrics in addition to window-level metrics.

3.0.0.3 Splits and evaluation protocol.

Given the small number of recordings in the final subset, we use a leave-one-performance-out evaluation protocol. For each evaluation fold, one performance for each standard is held out as the test set, while remaining performances form the training set. Windows from the same held-out performance do not appear in the training set. When training supervised models, one additional random performance from the training side is reserved for validation and hyperparameter tuning. For the final reported fold results, hyperparameters were fixed and models were trained on the full training partition before being evaluated on the held-out performances.

3.0.0.4 Evaluation Metrics.

We report three metrics based on classification and retrieval. Window accuracy measures Top-1 classification of individual 10-second windows. Performance Top-1 accuracy aggregates predictions over all windows from a held-out performance and evaluates the Top-1 standard prediction, corresponding to the strict standard identification setting. Performance Top-5 reports whether the true standard appears among the five highest-scoring standards after performance-level aggregation, which better reflects a retrieval-based use case for jazz standard recognition. We include window accuracy mainly as a diagnostic metric, since many individual windows may be noisy, or may not contain sufficient melodic or harmonic evidence for a given standard.

4 Methods↩︎

4.0.0.1 From-scratch Harmonic CNN.

As a supervised baseline trained directly on the subset, we train a Harmonic CNN-style [9] model on harmonic time-frequency representations of short audio windows. The model predicts a standard label for each window. At test time, we average window-level class probabilities over each held-out performance and choose the highest-scoring standard.

4.0.0.2 Frozen pretrained embeddings.

For this task, we extract frozen audio embeddings using MERT [7] and MuQ [10]. MERT-v1-95M¹ is a pretrained musical understanding model with 95M parameters, while MuQ² is a similar larger model with 300M parameters with an open-weight version trained on the Million Song Dataset [11]. For each window, hidden states are mean-pooled over time and concatenated across selected layers. We then train lightweight classifiers on top of these frozen embeddings. Linear probing tests whether standard identity is linearly separable in the representation, while a small MLP classification head is used to test whether a nonlinear classifier improves the performance of the probe.

4.0.0.3 kNN-based retrieval.

Given that performances from the same standard should follow similar melodies and underlying harmonies in theory, we also evaluate nearest-neighbor retrieval. Here, we build a reference set from embeddings from 10-second windows from the training split with MERT and MuQ encoders and L2-normalize each embedding. For each query window, we retrieve the top-\(k\) nearest reference windows and aggregate their standard labels using temperature-scaled weighted voting with cosine similarity. We obtain performance-level scores by averaging window-level scores across all query windows.

5 Results↩︎

We compare the results of model training on the filtered dataset in Table 2. In particular, from the results we observe the following interesting properties:

5.0.0.1 From-scratch training overfits on the curated subset.

The Harmonic CNN [9] baseline trains well but its held-out performance remains close to chance-level under both Top-1 and Top-5 accuracy metrics, with worse-than-random performance on Top-1 accuracy. This suggests that the model learns performance-specific or recording-specific structure rather than a standard-invariant representation. This behavior is expected given the small number of independent performances in the dataset and the relative difficulty of the standard identification problem. The HCNN also shows low window-level accuracy across the test set, which supports these observations.

5.0.0.2 Pretrained embeddings consistently improve Top-\(k\) recognition.

Linear and MLP probes on MERT and MuQ embeddings improve over the from-scratch HCNN baseline and over random prediction odds, especially in the performance-level Top-5 accuracy metric. This indicates that audio foundation models indeed learn useful general-purpose representations for improved standard recognition, even when Top-1 recognition remains difficult. Although we assumed that the relative high dimensionality of the embeddings may be noisy, in our preliminary experiments dimensionality reduction with PCA before probing did not consistently improve these results.

5.0.0.3 Retrieval is promising but naive retrieval is limited by performer identity.

Nearest-neighbor retrieval eliminates the need for fine-tuning on the dataset toward a Shazam-like system because new standards can be added by inserting embeddings into the reference database. However, retrieval errors show a particular tendency: nearest neighbors are often drawn from the same trio or performer but correspond to different standards. This suggests that the embedding space of pretrained models preserves performer or recording similarity in addition to jazz standard identity. We revisit this idea in Section 6.

6 Ablations↩︎

6.0.0.1 Effect of window length.

With the relatively long duration of solos in performances from the dataset and low window accuracies from methods in Table 2, we investigate whether using a longer window length helps overall recognition for pretrained models. Albeit minor and model-specific, results from Table 3 show that increasing the window length most notably increases the Top-1 accuracy of the MuQ + MLP probe approach to a value of \(0.125\), while being otherwise uninformative for other configurations.

6.0.0.2 Retrieval failures and improvements.

On investigation of the window-based retrieval approach from pretrained embeddings, we notice a particular trend: during retrieval, query windows from a given performance are often matched to windows from the same trio or performer but a different standard. For example, query sections of Autumn Leaves from Ahmed Jamal might retrieve embeddings from Ahmed Jamal’s Someday My Prince Will Come, due to similarity in playing style, volume, or recording composition. Although these properties do represent a degree of similarity, this is not our target standard-focused similarity measure.

As a lightweight adaptation towards this problem, we experiment with the following pipeline. We keep the pretrained audio encoder frozen and train only a lightweight projection model on top of its window-level embeddings. Given a 10-second audio window, we first extract a fixed embedding using MERT or MuQ. This embedding is passed through a two-layer projection MLP to obtain a normalized representation \(z_i\), and an MLP classification head maps \(z_i\) to logits over the set of jazz standards. The model is trained with a combined objective: \[\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda \mathcal{L}_{\mathrm{SupCon}},\] where \(\mathcal{L}_{\mathrm{CE}}\) denotes standard cross-entropy between the MLP head and the target standard label and \(\mathcal{L}_{\mathrm{SupCon}}\) represents a supervised contrastive objective [12]. For a mini-batch of projected embeddings \(\{z_i\}_{i=1}^N\), the supervised contrastive loss for anchor \(i\) is \[\begin{align} \mathcal{L}_{\mathrm{SupCon},i} &= -\frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(s_{ip}/\tau)}{\sum_{a \in A(i)} \exp(s_{ia}/\tau)}, \\ s_{ij} &= \operatorname{sim}(\mathbf{z}_i,\mathbf{z}_j), \end{align} \label{eq:supcon}\tag{1}\]

where \(P(i)\) is the set of positive examples for anchor \(i\), \(A(i)=\{1,\dots,N\}\setminus{i}\), \(\mathrm{sim}(\cdot, \cdot)\) is cosine similarity, and \(\tau\) is a temperature parameter. In this setup, positives are windows with the same standard label but from a different performance, so that the projection is encouraged to learn cross-performance standard identity rather than recording similarity. After training, we discard the MLP head and use the learned projected embeddings for retrieval. Figure 2 shows the proposed pipeline.

We track the effectiveness of this method through the “Same Group Frequency” feature: while this ratio is high for vanilla kNN-based matching, we observe that this value drops significantly for kNN-based retrieval trained with the auxiliary supervised contrastive loss, and the overall Top-1 and Top-5 accuracy of the models appear to improve as well, as highlighted in Table 4. We set \(\lambda=0.2\) based on preliminary validation experiments and keep it fixed across all folds to avoid any fold-specific tuning.

7 Discussion and Future Work↩︎

Our experiments suggest that jazz standard recognition is a challenging stress test for music representations. The difficulty is not merely the number of classes. In our curated dataset, many recordings appear to omit the head or begin in sections where the standard melody is not clearly present, which makes the task much harder than melody recognition. Consequently, a randomly sampled 10-second window may be considered difficult even for a human listener to identify. This makes window-level labels noisy: a solo passage labeled as Stella by Starlight may contain little direct evidence of that standard, and we cannot accurately predict what is a “good” target accuracy for a trained model.

7.0.0.1 Future work.

The gap between Top-1 and Top-5 performance suggests that pretrained models often retrieve or classify within a plausible neighborhood but do not reliably rank the correct standard in first place. This motivates two future directions for this area of research: First, parameter efficient fine-tuning of musical foundation models instead of working with frozen embeddings is a very promising candidate for improving recognition. Second, further explicit analysis of the underlying melody and harmony with a given audio sample may further provide useful features towards better generalization of such recognition models.

7.0.0.2 Limitations.

Our methodology and evaluation use local audio windows rather than a full symbolic or harmonic representation of a performance. As a result it only has a limited access to long-range musical context, which is important as many standards share similar local windows but possess a characteristic global flow that is important for identification. Conversely, many standards that share similar progressions (e.g., a 12-bar blues) are identifiable mainly through small but distinctive excerpts which can be rare relative to the duration of an entire performance. In this context, models might benefit from longer-context modeling, melody extraction and harmony identification for better results. Lastly, our curated dataset is intentionally small and controlled, so the results should be interpreted as an exploratory benchmark rather than a definitive ranking of music foundation models.

8 Conclusion↩︎

In this study, we present an exploratory study of jazz standard recognition across performances using a curated subset of the Jazz Trio Database. Our results indicate that a from-scratch HCNN network overfits to training performances, while probing based on embeddings from audio encoding models such as MERT and MuQ provide stronger results due to their pretraining. Our retrieval analysis suggests that the group identity influences embedding-based similarity and retrieval. Ultimately, our results indicate that jazz standard recognition is a realistic but difficult setting for evaluating whether music representations encode tune identity rather than only acoustic or performance similarity.

References↩︎