Keywords: Gurbani, Sikh Kirtan, low-resource ASR, lyrics alignment, closed-vocabulary captioning, Punjabi, Gurmukhi, benchmark, sacred-text NLP.

1 Introduction↩︎

Sikh Kirtan - the devotional singing of verses from the Sri Guru Granth Sahib Ji (SGGS) - is the central spiritual practice of Sikhi. In Gurdwaras (Sikh houses of worship), listeners follow along with a projected display showing the canonical Gurmukhi text of the line currently being sung. Today the display is driven manually by a volunteer (sewadar) using tools such as Sikhi To The Max (STTM): the sewadar listens for the opening words of the next shabad, types the first letter of each word into a search box, picks the matching shabad from a candidate list, and then clicks line-by-line through the shabad’s verses as the singer advances (Figure 1).

This end-to-end process is also leveraged by Kirtan video editors to caption recordings. Personal listening applications also offer a way to search and navigate through Gurbani. But across all of these settings the task is not an open-vocabulary lyrics transcription problem. It is instead a closed-vocabulary line-identification problem with a hard cultural constraint: the displayed line must be a verbatim canonical line, because displaying misspelled Gurmukhi is considered religiously inappropriate. Free-form ASR - even very accurate ASR - is therefore not acceptable as the output layer. This is not a hypothetical concern: automatically generated captions already accompany online videos of Kirtan, and where they come from free-form transcription we have observed them rendering incorrect or misspelled Gurmukhi - showing scripture wrongly to every viewer. That failure mode motivates the canonical-output constraint we place at the center of both the task and the metric. Some component, somewhere in the system, must snap the output to the canonical text.

This paper makes four contributions:

1. The canonical-output constraint as a metric-layer property. We argue that for Gurbani captioning the religious-appropriateness constraint belongs not just in the system but in the metric: every prediction is checked for an exact match against the canonical Gurbani text, so the scorer rewards only verbatim canonical output or \(\varnothing\), never approximate Gurmukhi.

2. A frame-temporal display-accuracy metric tied to the deployment question - “is the right line on screen at time \(t\)?” - rather than the question - “did the system emit the right verses somewhere?” The metric uses a scored region (to exclude instrumental interludes and discussion from scoring), a 1 s collar at segment boundaries, and gap-tolerant scoring (frames between adjacent segments accept either neighboring line or \(\varnothing\)). Cold-start offsets are evaluated explicitly, not averaged away.

3. A small, hand-annotated benchmark (v1): Initial seed of 4 Kirtan recordings \(\times\) 3 cold-start offsets \(= 12\) evaluation cases, \({\sim}57\) minutes of scored audio, with a Python scorer, visualizer, and annotation tool.

4. A reference system demonstrating the benchmark is tractable: a fine-tuned 120M-parameter IndicConformer, a fuzzy line matcher, and a state machine that locks on shabad identity and advances through lines. The full pipeline is INT8 ONNX, runs at RTF \({\approx}0.05\) on a single Apple Silicon CPU core, and scores 57.9% overall (10/12 correct shabad locks) on the difficult variant (live audio, unknown shabad) across all 12 evaluation cases.

This paper offers a shared yardstick: a benchmark, a scorer, and a reference system. We are not alone in working toward live scripture captioning - several efforts in the Sikh tech community target related problems, and in the parallel Quranic tradition Yazin A.’s open-source offline-tarteel (now Tilawa) [1] independently arrived at a closely related ASR \(\rightarrow\) matcher \(\rightarrow\) state-machine pipeline. For the Sikh context, this work provides shared language to measure accuracy and a deployed reference system as a first baseline.

2 Task Formalization↩︎

2.0.0.1 Task.

Given a stream of Kirtan audio, at every time \(t\) output a prediction \[\hat{y}(t) \in \{(s, \ell) : s \in \mathcal{S},\;\ell \in \mathcal{L}_s\} \cup \{\varnothing\},\] where \(\mathcal{S}\) is the set of shabads in SGGS and \(\mathcal{L}_s\) is the ordered set of lines within shabad \(s\). We use the integer identifiers from BaniDB / Sikhi To The Max; the canonical Gurmukhi for any \((s, \ell)\) is recovered via API lookup as an alternative submission path (exact match still required).

2.0.0.2 Two natural axes.

Systems for this task differ along two binary axes: temporal access (offline = full audio available; live = causal, audio up to \(t\) only) and shabad knowledge (oracle = shabad identity given; blind = must be discovered from audio). The four variants are shown in Table 1.

This paper targets the live+blind variant, but the benchmark scores any variant under the same metric and submission format.

2.0.0.3 Cold-start variants.

Each recording is additionally evaluated from offsets 33% and 66% into the audio, by moving the scored region forward while feeding the same audio. This tests how quickly a system locks on when a listener joins mid-program.

3 Why Not WER?↩︎

WER and CER are natural metrics for transcription tasks; we deliberately do not use them, for two reasons.

3.0.0.1 Different layer.

Three layers of representation are relevant:

WER measures whether the system transcribed the audio correctly; it is the right metric at the transcription layer, where prior Gurbani ASR work [2] applies it, as do we when evaluating this system’s underlying ASR component. The captioning task is a different question: which canonical line is on the projector at time \(t\). A system can transcribe perfectly and still display the wrong line (wrong shabad, late by ten seconds), so the benchmark scores the displayed line over time rather than transcription fidelity.

3.0.0.2 Different cultural constraint.

In Kirtan, misspelled Gurmukhi is religiously inappropriate. A canonical-only metric forces systems to either output real Gurbani or output nothing - which matches what the product can ethically display.

4 The v1 Benchmark↩︎

4.0.0.1 Data.

Four publicly available Kirtan recordings on YouTube were selected as v1, chosen for clean audio and the simplest task structure (one shabad per recording, no Katha, Simran, or interludes). Each recording yields three evaluation cases: the full file, and cold-start variants at 33% and 66%. Recordings range from 4.9 to 10.9 minutes; this initial sample is \({\sim}57\) minutes of scored audio across 12 cases (Table 3).

4.0.0.2 Ground-truth format.

Each case is one JSON file containing the shabad_id, total duration, a scored region {start, end} (frames outside it are unscored - this excludes intros, outros, and the cold-start skip region), and a list of segments, each a (start, end, line_idx) triple. line_idx is 0-indexed within the shabad. The canonical Gurmukhi text of each line is included inline in the GT file so submissions using the text-key path (banidb_gurmukhi) can be scored without an API lookup.

4.0.0.3 Submission format.

A submission is one JSON file per GT case, listing predicted segments (start, end, line_idx). We ship a few different paths for producing this JSON - direct file write, a relay that impersonates the STTM display protocol for systems that already drive it, and a minimal client adapter - but all produce the same final format, which is what the scorer reads.

4.0.0.4 Scoring.

Time is discretized to 1-second frames; every frame within the scored region is evaluated. Each frame falls into one of three regions (Table 4).

The primary number reported is frame accuracy at collar \(= 1\) s. The collar credits systems that handle line transitions within \(1\) s of the true boundary; tolerable latency is itself a property of the deployment, so the collar is set tight enough to discriminate real-time-capable systems from sluggish ones, but not so tight that small timing differences dominate the score.

4.0.0.5 Baselines.

The empty baseline is non-trivially above 0% because inter-segment gaps accept \(\varnothing\) as correct; it functions as a sanity check on the scorer. The shifted_5s baseline approximates the latency profile of an online ASR pipeline that lags audio by a few seconds - the 1 s collar absorbs a small fraction; the rest of the cost shows up in segment interiors. Confidently emitting the wrong shabad scores worse than emitting nothing, by design.

4.0.0.6 Implementation.

The scorer (eval.py) is standard-library Python 3.10+. A companion script, visualize.py, renders an interactive HTML page with GT/prediction/diff strips, audio scrubbing, hover-tooltips showing canonical Gurmukhi, and a cold-start toggle.

5 Reference System↩︎

We describe a reference system targeting the hardest variant (live \(\times\) blind), to demonstrate the benchmark is tractable. The system runs as a three-stage pipeline.

5.0.0.1 Stage 1: ASR.

We fine-tuned the AI4Bharat 120M-parameter Punjabi IndicConformer (Hybrid CTC-RNNT) [3] on weakly-supervised Kirtan clips. Two pipelines feed the training set, both of which align a noisy transcript to the canonical SGGS text and keep only high-confidence lines: (i) starting from existing transcripts (subtitles, larger ASR outputs, or earlier model outputs), and (ii) generating fresh transcripts with Google Chirp, then identifying the shabad over a window of audio and snapping each transcript word to the closest canonical line within it. In both cases the canonical text serves as the cleanup target: noisy words near a canonical line word are corrected; lines below a confidence threshold are dropped. The fine-tuned model is exported to INT8 ONNX; on a single Apple Silicon CPU core, a 10-second window decodes in \({\sim}490\) ms (RTF \({\approx}0.05\)); on a commodity cloud vCPU we measured RTF \({\approx}0.08\). At inference time, once a shabad is locked (Stage 3 below), CTC decoding is hard-constrained to a trie of that shabad’s lexicon: the beam search can only emit token sequences that spell words from the locked shabad, which can improve line-level accuracy once the shabad is correctly locked. The unconstrained decoder is still used during the blind cold-start phase.

5.0.0.2 Stage 2: Matcher.

The matcher runs in two phases. Phase 1 (identification) scores a noisy Gurmukhi transcript \(T\) against the entire SGGS text by fuzzy string matching against every line. For each candidate shabad \(s\), we compute \[\textsf{score}(T, s) \;=\; \alpha\,\max_{\ell \in \mathcal{L}_s} \textsf{sim}(T, \ell) \;+\; \beta\,\min(g_s,\, 5),\] where \(\textsf{sim}(\cdot,\cdot)\in[0,100]\) is a fuzzy substring-similarity score (RapidFuzz partial_ratio), \(g_s = |\{\ell \in \mathcal{L}_s : \textsf{sim}(T,\ell) \ge \tau\}|\) is the number of the shabad’s lines that match \(T\) above a threshold \(\tau\), and \((\alpha, \beta) = (0.7, 6)\) are tuned weights. The first term rewards the single best-matching line; the second is a multi-line agreement bonus, capped at five lines so that a few strongly matching lines suffice. Shabads with very few lines receive no agreement bonus, since they cannot demonstrate agreement. Phase 2 (line tracking) runs only after a shabad is locked (Stage 3): it scores the transcript against the \({\sim}10{-}20\) lines of the locked shabad alone, using a bidirectional fuzzy word-overlap score (the F1 of line-word recall and window-word precision). Like the hard-CTC decoder, this scorer is restricted to the locked shabad and exploits the lock; until lock, only Phase 1 runs.

5.0.0.3 Stage 3: State machine.

A small state machine consumes matcher outputs on two cadences: an identification tick over a wider window (\({\sim}30\) s) when not yet locked, and a tracking tick over a shorter window (\({\sim}15\) s) once locked. In autolock mode (used for the benchmark), the lock decision fires when several gates pass together: a minimum amount of audio has elapsed, the top-ranked shabad has stayed top-ranked for several consecutive identification windows, and the relative score gap between top-1 and top-2 candidates exceeds a confidence threshold (with a more permissive threshold available later in the audio, so a system that has heard a lot but is only moderately confident can still commit). The thresholds are tuned hyperparameters. Once locked, a within-shabad pointer advances through lines biased to the matcher’s best line per chunk; a small hysteresis margin on line switching prevents per-window display flicker, requiring a competing line to beat the displayed one by a margin before it takes over. The pointer does not assume monotonic forward progress: Kirtan repeatedly returns to the rahao (the recurring refrain that alternates with verses), and a strictly forward tracker would treat legitimate rahao returns as backward jumps and suppress them. A recovery mechanism guards against bad locks: an auto-unlock sliding window tracks recent line-match scores and falls back to identification if they sustain below a dynamic threshold. An optional periodic sanity check can additionally re-run full identification and force unlock if the locked shabad falls out of the top-\(K\) candidates.

5.0.0.4 Deployment.

The full system has a live deployment (current demo link in the reference-system repository) that continuously captions Sikhnet Radio. It is also deployed as a desktop app in the GitHub repository, and as an on-device iOS app (a Swift port of the same engine, INT8 ONNX / CoreML), distributed via a TestFlight beta linked from the reference-system repository. The autonomous machinery described above - multi-gate autolock, hysteresis, auto-unlock - is what the benchmark exercises: the scorer requires the system to make committed decisions over time with no human in the loop. The deployment runs a different UX on top of the same underlying model: the interface displays top-\(k\) shabad candidates rather than autonomously committing, and the listener can confirm a candidate or reset the tracker. The benchmark numbers in Section 6 therefore reflect the system deciding alone, with no human in the loop; the assisted deployment, where a listener confirms candidates, should do at least as well. We view assisted-commit as the appropriate experience today - sewadars must retain control over the display, especially in group settings.

6 Results↩︎

We report frame accuracy on all 12 evaluation cases (4 recordings \(\times\) 3 cold-start offsets) on the hardest variant (live \(\times\) blind). Results are from kirtan-captioning v0.2.0 (commit 3b42245), model v4.int8.onnx, scored with eval.py ``-``-``collar 1 on macOS arm64 (CPUExecutionProvider). Predictions are released alongside this paper for direct re-scoring.

6.0.0.1 Headline.

Overall frame accuracy is 57.9% (1,982 of 3,425 scored frames) with 10 of 12 correct shabad locks. Both lock failures are _cold33 cases - dropped into the middle of a recording with no prior audio history, the matcher’s first 30 s of audio does not contain enough distinctive words to separate the target shabad from confusable candidates. This is the hardest variant in the suite.

6.0.0.2 Where the cost is paid.

Inspecting the per-case lock timestamps shows two distinct failure modes that the frame-temporal metric exposes but which utterance-level retrieval metrics would conflate (Figure 3):

1. Slow lock, then good tracking. zOtIpxMT9hU locks correctly but only at 188 s (more than half the recording is gone), so the score is dominated by the pre-lock red region rather than by tracking errors. The diff strip is red for the first \({\sim}3\) minutes and green thereafter.

2. Confidently wrong lock. IZOsmkdmmcg_cold33 locks to the wrong shabad (S3643 instead of S4377) and stays wrong; this scores worse than the empty baseline on this case because frames in segment interiors only accept the exact line_idx.

The shifted_5s baseline scores 85.5%, so there remains a substantial gap between the reference system and an oracle pipeline with only latency error - most of the gap comes from shabad identification (the two wrong locks plus the several slow locks) rather than within-shabad tracking, which is reasonably accurate once a lock is established.

7 Related Work↩︎

7.0.0.1 Parallel work on Quranic recitation.

Yazin A.’s open-source offline-tarteel (since renamed Tilawa) [1] is an on-device system that identifies the Quranic verse being recited from audio with a similar ASR \(\rightarrow\) fuzzy matcher \(\rightarrow\) state-machine pipeline. It targets a different scripture and reports verse-retrieval metrics. Separately, and despite the shared name, Tarteel Inc.[4] is an unrelated closed-source Quran-memorization product that tracks the user’s place within a chosen verse, an oracle-variant analogue at finer (word-level) granularity.

7.0.0.2 Direct prior Gurbani ASR.

[2] build a CNN speech-to-text system for Gurbani hymns and report WER; that measures transcription, whereas we measure which canonical line is displayed over time.

7.0.0.3 Generic Punjabi ASR.

A broader research community works on general-purpose Punjabi ASR without addressing Kirtan or the canonical-output constraint.

8 Limitations and Future Work↩︎

8.0.0.1 Limitations of v1.

• Scale. Four recordings is small; statistical claims about system comparisons would be noisy. We will extend to more recordings.

• Task simplicity. v1 recordings contain one shabad each, with no Katha (spoken commentary), Sehaj Paath (a complete reading of the SGGS at a relaxed pace), Simran (meditative repetition), or interludes. These are common in real Gurdwara programs and a future update will include them.

• SGGS only. Dasam Bani and other Sikh scripture are not yet included.

8.0.0.2 Limitations of the reference system.

• Sensitive to recitation speed (very slow or very fast hurts the matcher).

• No awareness of adjacent shabads on the same ang (page), so Akhand Path / Nitnem flows are not tracked smoothly. An update with support for this continuous recitation is in testing.

• Degrades on low-quality audio; opportunity to fine-tune with intentionally noisy data.

8.0.0.3 Future work.

STTM-instrumented capture (using deployed sessions as weak ground truth) is an attractive route to scale annotation; we ship a prototype recorder (sttm_recorder.py) that could serve as the data-collection endpoint. A related direction is to measure and tune the training-data pipeline of Section 5 as a labeling loop in its own right - how often does retrieval pick the wrong shabad, what threshold settings trade label quality for label volume, and how does that trade-off affect downstream model quality - which could be a separate data-centric paper.

The reference system has many directions worth exploring: alternative ASR models specialised for Kirtan (recent public examples include [5]), alternative matcher designs, and replacing parts of the hand-tuned state machine with learned components. A complementary direction is normalising the implementation of Yazin’s offline-tarteel/Tilawa [1] against ours - both pipelines decompose into ASR, matcher, and state-machine layers, so an apples-to-apples comparison would let the best ideas from each tradition feed back into the other. The system can also serve as a baseline reference for other Gurbani applications to include related functionality. Lastly, we hope to leverage this work to remove inaccuracies in current automated video captioning platforms that treat Gurbani no different than generic speech.

9 Ethics and Cultural Considerations↩︎

The choice to enforce a canonical-only output is not technical. Misspelled Gurmukhi in a religious display context is not a neutral error; it is read as disrespect to the Guru Granth Sahib Ji, revered by Sikhs as a living Guru. We therefore designed both the metric and the system so that emitting nothing is preferable to emitting the wrong text, and we argue against any deployment of free-form ASR output as the display layer in this context.

Our current deployment runs the reference system in beta with a confirm-style interface - the system shows candidates and a listener taps to commit - rather than autonomously driving the display.

The pipeline is built around a small on-device model that can run on consumer hardware. Users do not need to send their audio to any external servers for processing; the iOS app runs the model entirely on-device.

10 Availability↩︎

• Benchmark: https://github.com/karanbirsingh/live-gurbani-captioning-benchmark-v1 (code MIT, annotations CC BY 4.0)

• Reference system: https://github.com/karanbirsingh/kirtan-captioning (CC BY-NC-SA 4.0)

• iOS app (on-device, INT8 ONNX / CoreML): TestFlight beta linked from the reference-system repository README

• Live deployment: current demo link in the reference-system repository README

• Visualizations: https://karanbirsingh.github.io/live-gurbani-captioning-benchmark-v1

10.0.0.1 Reproducibility.

The scorer (eval.py) is standard-library Python 3.10+. The exact prediction JSONs that produced Table 6 are committed under benchmark/results/ in the reference-system repository, alongside the model checksum and a single-command reproduction recipe.

Acknowledgments↩︎

Thanks to friends from within and outside the Sikh community who served as a sounding board throughout this work, to the maintainers of BaniDB and Sikhi To The Max for open canonical-text APIs, and to the reciters whose recordings comprise v1.

References↩︎