1 Motivation & Related Works↩︎

In Music Information Retrieval (MIR), Automatic Music Transcription (AMT) involves converting music audio into symbolic music data, such as MIDI. While quantitative metrics, such as precision, recall, and F1-score, are used to assess the performance of deep learning-based AMT models [1], they do not fully reveal the nature of errors, including which specific notes are incorrect or whether errors occur consistently in certain sections. Therefore, to thoroughly understand a model’s behavior, it is essential to visually and aurally examine the resulting MIDI data.

Current research on AMT primarily presents qualitative results through static or non-interactive media. For instance, some studies display input audio and resulting MIDI as a single video [2], while others provide the model’s output for a Ground Truth as a piano roll image with an individual MIDI player [3], or allow playback of a MIDI output for a Ground Truth audio [4]. However, these approaches have limitations. They make it difficult for users to compare outputs from multiple models for the same input simultaneously or to perform interactive analysis by seeking specific positions within the audio.

WaveRoll is an interactive JavaScript library that reconfigures the piano-roll style of music data visualization for multi-MIDI comparison on the web. It is proposed as a solution based on existing concepts: web-based MIDI piano-roll players (from html-midi-player [5]) and multi-track control (from trackswitch.js [6]). This feature allows users to toggle, play, stop, and seek multiple MIDI files simultaneously on a single screen, adopting a style commonly found in Digital Audio Workstations (DAWs).

Users can concurrently load MIDI results generated by different models for the same audio input, providing an immediate means to identify discrepancies in pitch, timing, onset, and sustain. WaveRoll is delivered as a <wave-roll> custom web component, facilitating easy analysis of model performance in local development environments using Node Package Manager(npm)¹ or via the library’s demo page². Furthermore, researchers can utilize WaveRoll to create web-based interactive demonstrations with minimal code, enabling readers to grasp their findings intuitively.

2 WaveRoll↩︎

WaveRoll introduces the Layered Piano Roll uniquely displays multiple MIDI files as distinct visual layers on a single, unified grid as illustrated in Figure 1.

2.1 Key Insight Features↩︎

WaveRoll is designed to help users gain comprehensive insights by integrating visual, auditory, and note-level analysis.

2.1.1 Interactive Visual Analysis↩︎

The library provides an interactive environment for visual analysis. Users can view multiple MIDI tracks on a layered canvas, controlling the visibility of each track’s piano roll and sustain data while using zoom and pan for detailed examination. To aid comparison, each track can be assigned a custom color. Furthermore, various highlighting modes can visually emphasize matching or differing notes between tracks, allowing for a more focused analysis of similarities and discrepancies in model outputs.

2.1.2 Detailed Auditory Comparison↩︎

The library integrates not only the visualization but also the playback of original audio waveforms alongside MIDI data. For in-depth auditory analysis, it offers Mute and Panning controls for individual tracks. Crucially, the A-B Looping feature enables users to repeat specific sections, allowing for a focused comparison of subtle rhythmic or harmonic variations that may be challenging to identify through visual inspection alone.

2.1.3 Visual Note Matching↩︎

WaveRoll facilitates qualitative judgment through note-level comparison. Based on the note evaluation method from mir_eval[7], the library offers a visual highlighting feature for matching notes between two tracks, configurable by a user-defined tolerance. This visualization is beneficial for analyzing model accuracy and identifying error patterns in Automatic Music Transcription.

2.2 Implementation↩︎

Using WaveRoll on a webpage is straightforward. It is available as an npm package, installed with the command npm install wave-roll. As illustrated in Figure 2, developers can integrate the component into their HTML and provide the MIDI files for comparison as a JSON array to the files attribute. It can also be embedded in Jupyter notebooks using IFrame.

3 Conclusion↩︎

WaveRoll is designed to enhance the qualitative evaluation of AMT research. It addresses current limitations by visualizing multiple MIDI files on a layered piano roll with synchronized playback. This functionality allows researchers to intuitively analyze model error patterns and helps readers more easily understand the results.

Future developments for WaveRoll include improved user experience and custom SoundFonts. While its primary focus is currently AMT evaluation, WaveRoll’s application can be extended to other research areas that require the comparison of multiple outputs, such as Symbolic Music Generation. As an open-source project, community contributions are highly encouraged.

4 Acknowledgement↩︎

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2024S1A5C3A03046168).

References↩︎