1 Introduction↩︎

Gregorian chant, the liturgical monody of the Latin church, was the first musical tradition to be notated on a staff, already in the 11th century [1]. It is also one of the largest homogeneous musical traditions, with up to 30.000 extant manuscripts [2], and it has been a major pillar of European musical identity and an essential element of its soundscape up until the late 20th century [3]. Despite significant loss of status since the 2nd Vatican Council and the subsequent liturgical reforms in the 1970s, it is still practised not only in Europe, but in all places with Catholic communities worldwide. Within musicology, chant scholarship is a large field, with its own multi-day conference.¹

But while the scale of chant presents major opportunities for computational musicology to have an impact [4]–[13], this is held back by the need for correspondingly large-scale datasets. Specifically for melody research, transcriptions are necessary (typically in the Volpiano [14], [15], GABC [4] or MEI [16], [17] encodings), and these are time-consuming and require highly specialised expertise in reading chant notations. While over 40,000 fully transcribed melodies are present in the Cantus network of databases [4], [13], these in fact mostly come from just several fully transcribed manuscripts: full-size Gregorian chant books contain 1000-4000 melodies each.

It is thus no surprise that optical music recognition (OMR) for Gregorian chant has seen multiple systematic efforts to tackle it [2], [18]–[24], with some success. However, the available datasets are split across many different encodings — sometimes obviously distinct, sometimes subtly so. Despite the principles of Gregorian staff notation being remarkably consistent across its multiple visual realisations (square notes vs. rhombic notes, subtle graphics vs. thick lines, etc.), it is thus not straightforward to pool these resources.

2 Related work↩︎

While the notation of Gregorian chant may seem peripheral to the justification of OMR in terms of user-facing applications, it has in fact seen a lot of OMR research. Besides didactic reasons, the justification comes from the needs of digital and computational humanities: with tens of thousands of extant manuscripts, containing some 1000-3000 chants each, computational methods are the only viable way of robustly understanding this major chapter of European cultural heritage — and these in turn require machine-readable data, where again the scale of chant precludes any path but automation, via OMR. At the same time, the staff notation of chant is relatively straightforward: at least from the 13th century onwards, chant melodies can be conceptualised simply as sequences of pitches, and their alignment to text (usually expressed in terms of syllable and word boundaries).

Chant was first substantially addressed with the interactive Gamera toolkit [25], based on kNN classification of neume shapes after staff removal. This gradually evolved into web-based tools Pixel.js [26], Neon [27] and Neon2 [28], and the Rodan framework for orchestrating jobs with interactive steps [29].

A different software environment, a monolithic application instead of the Rodan-orchestrated combined workflows, was developed by the Corpus Monodicum project: the OMMR4All software [19], [20]. However, it follows a similar workflow (staff detection, symbol detection, notation assembly, and final encoding for the MonodiKit environment [30]), though the text recognition component of the pipeline does use an image-to-sequence model [19].

Most recently, end-to-end recognition with a host of different image-to-sequence models was introduced as the aligned music notation and lyrics transcription (AMNLT) task by Fuentes-Martinez et al. [22]. This is the first substantial work on OMR for chant using end-to-end methods such as the Sheet Music Transformer [31], [32]. It introduces four distinct datasets for this purpose (GregoSynth, Einsiedeln, Salzinnes, Solesmes) with ground truth suitable for staff-wise recognition, and measures results on these datasets for 11 different models. We follow up on this work: we take the same datasets and evaluation procedure, harmonise the encodings to enable merging the datasets into one, and train shared models that substantially improve on this work, opening the doors towards a foundational model for diastematic Gregorian notation.

3 Datasets↩︎

The baseline work [22] has combined four datasets of diastematic (that is, staff) Gregorian notation. The dataset sizes are summarized in table ¿tbl:tab:datasets?. The datasets are available via HuggingFace (upon request).²

Each dataset has in principle the same ground truth setup: images of individual staffs, and their transcriptions, both of the melody and the text. Rather than following the older OMR pipelines based on object detection [21], [25], [33], these are structured for end-to-end recognition, which has become a more dominant paradigm recently [31], [34]–[36].

3.1 GregoSynth↩︎

The GregoSynth dataset, as the name suggests, is a dataset of synthetic Gregorian chants. It was generated from the Gregorian Chant Database³ using GregorioTex⁴. The transcriptions use GABC encoding with music aware tokenization (discussed in detail in Section 4). An example can be seen in figure 1.

3.2 Salzinnes↩︎

The Salzinnes dataset contains neumes from a 16th-century Cistercian antiphoner from the Abbey of Salzinnes, Namur, in the Diocese of Liège. It was originally transcribed in the Music Encoding Initiative (MEI) format; [22] converted the MEI format to Pseudo-GABC encoding (discussed in detail in section 4). An example can be seen in figure 2.

3.3 Einsiedeln↩︎

The Einsiedeln dataset contains neumes from a manuscript of the monastery of Einsiedeln, Switzerland. Its transcriptions are encoded the same as Salzinnes (section 3.2), i.e. originally in MEI and then converted to Pseudo-GABC. An example can be seen in figure 3.

3.4 Solesmes↩︎

The Solesmes dataset consists of Gregorian chants from the extensive collections of the Solesmes abbey that were central to the monks’ early efforts at a critical edition of Gregorian chant in the early 20th century and since. The transcriptions are encoded in the S-GABC format introduced by Thomae et al. [37]. An example can be seen in figure 4. This is the smallest and most diverse dataset, and therefore the most challenging of the four.

3.5 Duplicates cleaning↩︎

We found issues stemming from the fact that the underlying chants are present in multiple versions with only slight differences, which means that individual samples (staffs, which can also be relatively short) may contain the same (or extremely similar) ground truth string. This is a small issue in Salzinnes, Einsiedeln, and Solesmes, where the images corresponding to (near-)duplicate ground truth will still be different. In GregoSynth, however, the images are synthesised from the ground truth encoding, so they end up (near-)identical, to the pixel; this is a major issue especially when such duplicate sets end up distributed across both the training and test splits.

We computed similarity between ground truth pairs as Levenshtein distance between the strings using the StringZilla library.⁵ All datasets were first converted to the common encoding (see below), which removed most of the formatting symbols irrelevant to the content and with only slight implications for the visual appearance in the case of GregoSynth. We also removed all whitespace and music tags as these do not change the content.

We then need to also consider image similarity. For example, the Salzinnes dataset has several samples with just a differentia (lyrics “euoae”) and the same melody which is normal. If it has exactly the same images, it is a true duplicate. However, if the images are visually different, we consider them to be two distinct samples, and they do not need to be discarded. We use the structural similarity index (SSIM) [38]. The total definition of a duplicate is as follows: (1) the Levenshtein distance between samples is less than or equal to 3 and (2) the image similarity score (SSIM) is greater or equal 0.97.

This pipeline was run on each dataset separately. The resulting dataset sizes can be found in table ¿tbl:tab:dataset95cleaning?.

4 Joint encoding↩︎

The four datasets are given in three different encodings: GABC, S-GABC, and a MEI-derived “Pseudo-GABC”. So, despite being datasets pertaining to the same — or, compatible enough — type of notation in the images, the datasets cannot be combined, which is the opposite of helpful. To some extent, this even makes the prior results incomparable with each other: getting a lower error rate on Salzinnes than on Solesmes with the same model may not mean that Solesmes is harder to recognise, as the encoding of the Solesmes dataset may contain more symbols not entirely relevant to actually recovering text and melody (and aligning them, as framed in [22]) where errors are made. This is not to take issue with any particular encoding of chant: there are multiple good solutions to how to encode diastematic chant notation digitally. However, because the datasets contain images of the same tradition, with notation conforming to identical principles, there is no reason why the encodings couldn’t be mapped onto each other.

This is the key step that we perform: we define a shared GABC-derived encoding to which all the datasets can then be re-encoded without losing information about melody, texts, their alignment, and grouping into neumes. We then re-encode the datasets in this shared encoding. The point of departure is the proposed Systematic GABC [37].

4.1 GABC↩︎

The GABC encoding was derived from ABC notation by the Gregorio project.⁶ The other encodings are derived from GABC; hence we describe it in more detail.

The encoding combines lyrics and notes in a single sequence. Since both share the same character set, they are distinguished by parentheses: musical notation is enclosed, while lyrics remain outside. The GregoSynth dataset has additionally differentiated the music notation by prepending <m> to every character, but only to differentiate music and lyric tokens in training models.

Notes are encoded as a single letter from a to m for the four line staff which is used in Gregorian chants. There is no separating character between notes. The letter describes the note position within the staff, not the pitch (in this, it is analogous to the Volpiano encoding used in the Cantus network of databases [15], [39]). This puts a hard limit where we can place notes.

Each note can then have prefixes or suffixes that modify the visual appearance or the meaning of the note. Neither prefixes, nor suffixes have any separation character between them and the note.

The pitch of the notes are determined by a clef which is placed within the staff. There can be either a c clef or a f clef which symbolizes the position of a c or a f pitch. It can be modified to a flat clef which will symbolize c flat or f flat. The clef can be positioned only on a line which is indicated by a number from 1 to 4 in our case.

4.2 Systematic GABC (S-GABC)↩︎

The GABC encoding is intended for digital typesetting of modern Gregorian square notation used by the Solesmes editions of the 20th century. The encoding thus supports various formatting options for lyrics and notes, which are not used in other encodings that were designed with the opposite process (from image to structured encoding) in mind. This use-case leads to the most important issue GABC has for the purposes of OMR: there are multiple ways to encode the same (or effectively the same) music.

To address this issue, Thomae et al. [37] proposed Systematic GABC (S-GABC). This canonical encoding formalizes the GABC definition by context-free grammar which avoids any ambiguities of the original encoding. However, the proposed grammar is not complete, and some rules are not fully defined. Additionally, the ground truth of the Solesmes dataset, which is encoded in S-GABC, does not fully comply with the specification in [37].

Systematic GABC has the same encoding style as GABC, differing only in a few things. Hollow notes are not supported; instead, the letter r is used to encode uncertain reading of neumes. Completely illegible sections are marked with empty parentheses. Irrelevant options for formatting lyrics are not supported.

Like GregoSynth, the Solesmes dataset also has the music tag <m> for every music character.

4.3 Pseudo-GABC↩︎

Finally, there is the Pseudo-GABC encoding. This was introduced in [22] for data converted from the XML-based Music Encoding Initiative (MEI) neumes format to a linear structure similar to the GABC encoding. However, Pseudo-GABC uses absolute pitches as note symbols instead of “agnostic” position on the staff lines.

Pseudo-GABC is, unfortunately, not documented. However, it was possible to reverse-engineeer the meanings of the encoding symbols from the Salzinnes and Einsiedeln datasets, in combination with the documentation of MEI Neumes module.

Pseudo-GABC, like GABC, encloses all melody encoding in parentheses, and everything outside is lyrics. Notes are encoded as absolute pitches, with optional suffixes indicating neume shapes (e.g., the presence of a stem). The pitch is always a pitch in English format (c-b) and a number that represents the octave. Suffix is separated from the pitch or other suffix by dash (-). The clef is encoded as in GABC: a letter indicating a clef type, and a number denoting the line on which it is placed. However, the clef type is an uppercase letter instead of lowercase one. Accidentals are encoded as a symbol f for flat or n for neutral before a pitch, with a single space between them.

4.4 Defining a joint encoding and the conversion process↩︎

Based on a thorough comparison of the features in the GABC, S-GABC, and Psuedo-GABC encodings used in the four available datasets, we developed a common encoding. The main criterium was retaining all essential information about the melody and text as possible while keeping the encoding consistent across datasets. For example, neume spacing was removed, because Pseudo-GABC does not support it and it cannot be inferred unambiguously. A special marking for the porrectus neume with an oblique component was kept, however, because while GABC did not support it, it was possible to write an algorithm which added the oblique marking retroactively. The full list of supported symbols across the three input encodings and the common encoding is provided.⁷

The conversion to the common encoding⁸ is implemented as a set of Lark grammars for each of the GABC,⁹ S-GABC, and Pseudo-GABC encodings, and an output CFG into the shared format. The transcriptions are parsed using corresponding grammar into a parse tree. The parse tree is then transformed to a tree that represents the common encoding¹⁰. Due to malformed strings in the original datasets, not all samples were successfully re-encoded, though not more than 5 % of each dataset. The full report of mistakes is made available.¹¹

5 Recognition experiments and Results↩︎

To measure the potential of pooling datasets, we train a shared model. Out of the architectures used in [22], we choose the DAN model [40]: it performed well, and in preliminary experiments, it was faster to train than the Sheet Music Transformer. For the combined training, we randomly downsample GregoSynth in each epoch to the same number of training examples as second the largest dataset (that is, Salzinnes), so that the synthetic data don’t overwhelm the training process while taking advantage of its more varied musical content.

To train the shared model, we copy the setup of [22]. To see the the potential of the shared model as a foundational model, we then run fine-tuning experiments, where we use the same training procedure on each of the real datasets (Salzinnes, Einsiedeln, Solesmes) initialised with the weights of the shared model with learning rate of \(1e^{-5}\) instead of the original \(1e^{-4}\).

The minibatches in [22] are of size 1 due to uneven scales of input images. Noting this as a potential weakness in the training setup, we also train the shared model with 8-fold gradient accumulation instead.

We retain the train/validation/test splits for all four datasets as in [22]. The encodings changed between the baseline models and models trained in this work, so the comparison between baseline and our models is not quite direct; however, the vast majority of symbols participating in the evaluation metric are retained, so at least an approximate comparison is still meaningful. We report the same metrics as [22]: symbol error rates on the subset of symbols that comprise the melody (MER), the sung text (CER), entire combined syllables (SylER), the percentage of errors caused by melody-text alignment issues (ALER), and the combined rate of all errors in melody, text, and alignment together (AMLER). For applications in an automated manuscript cataloguing and retrieval (searh), CER and MER are most important; for use in workflows that produce inputs for human readers, the combined AMLER is most indicative of overall applicability. All results are reported in Table 1.

6 Discussion and Conclusions↩︎

What did harmonising the encodings and cleaning the datasets achieve?

The DAN-shared model does not match the best baseline across all datasets, but at least on GregoSynth and Salzinnes, it outperforms DAN-baseline. This is not entirely unexpected: the shared model has a somewhat harder job generalising across all four dataset (though it saw training samples from them all).

However, the real impact of harmonising the encodings is shown to be enabling the shared pretraining + fine-tuning pipeline. The fine-tuned DAN models match or exceed the performance of the best baseline results (except for Solsemes CER). This confirms the value of merging datasets when the underlying notation is mutually compatible, as already observed for white mensural notation [23].

But then the shared model with 8-fold gradient accumulation outperforms everything else. While a complete comparison would involve training the DAN baseline with the same gradient accumulation policy rather than a batch size of 1,¹² the shared model might be able to take advantage of gradients combined across samples from different datasets (GregoSynth was undersampled in every epoch), thus starting to take advantage of the greater diversity of notations in the combined datasets. Fine-tuning this model then improves the results further, though percentage-wise the impact is not as large as before: the improved training allows already the shared model to learn more from the data it had. The common encoding is very much based on the S-GABC proposal, which was a first step in this direction of a shared diastematic chant notation encoding; however, there are some differences from S-GABC as well: all lyric formatting was removed, since it was not used in datesets, uncertain readings were removed, and hollow notes were reintroduced using the same syntax as GABC. Horizontal episema position was reduced to only 0 and 1. We intentionally do not give a name for our encoding, because we believe it could simply be used together with the original S-GABC proposal as the next step towards a full implementation of what S-GABC could be, and we would welcome a discussion on the future.

In any case, the effort put into the harmonising encodings and cleaning the datasets has resulted in a new state of the art for end-to-end OMR that is packaged into a more universal model that can provide these results across the diverse visual domains of Gregorian notations.

References↩︎