Index Terms: speech recognition, code-switching, contextual biasing, technical terms, Mandarin

1 Introduction↩︎

Lecture ASR powers subtitles, search, and note-taking for students. In AI/ML lectures, bilingual technical terms anchor the content. Terms such as RAG, SWE-bench, AI Agent, token, and embedding name the concepts the speaker explains. A student who reads “RIG” instead of “RAG” loses the point of the segment.

These terms occupy a small fraction of the characters. A transcript can reach low CER while still misrecognizing the key term. The Mandarin around the term is fluent, so CER stays low. The usable content, however, is wrong. CER does not capture this failure.

Lecture terms are bursty. Once the speaker introduces AI Agent, the term recurs across many later segments. The lecture title, the previous transcript, and the repeated terms all point to the same vocabulary. Segment-only decoding discards this signal. It recognizes each segment in isolation, so it cannot use the strong context that the lecture provides.

We address this gap by building a term-rich benchmark, measuring term recognition directly, and supplying a self-built glossary as decoding context.

Our contributions are as follows.

• We release a term-rich Mandarin AI/ML lecture ASR benchmark. It contains 8,888 technical-term occurrences over a 5.01-hour code-switched test set.

• We define term-centric evaluation. It reports term recall, term precision, term F1, and term error rate, so it separates technical-term accuracy from overall CER.

• We propose a two-pass, reference-free glossary prompt. It extracts a lecture glossary from first-pass hypotheses, and it improves term recall across five ASR backbones while holding or lowering CER on all of them. A hybrid that adds a small external glossary reaches 62.05% term recall on Breeze-ASR-25, closing part of the remaining headroom.

2 Related Work↩︎

Code-switching Mandarin-English ASR. Mandarin-English code-switching is a long-standing challenge for ASR. Public corpora such as SEAME [1], the ASRU 2019 challenge set [2], and ASCEND [3] established Mandarin-English benchmarks, but they target conversational rather than lecture speech. Recent work fine-tunes large models on Mandarin with English terms [4], and distills language-specific knowledge for code-switching [5]. These methods improve the acoustic model. We instead keep the model frozen, and we add context at decoding time.

Contextual biasing. Contextual ASR injects external knowledge into recognition. Deep context learns to bias toward a phrase list [6]. Trie-based deep biasing and shallow fusion bias streaming recognizers toward a catalog [7]. Audio-language models couple speech encoders with LLMs [8], [9], and LLM-based recognizers consume free-form context [10]. Qwen3-ASR treats context tokens as background knowledge for biasing [11]. These methods assume a context source is available. We build the context from the model’s own first-pass output, so the method needs no external list.

Prompting and retrieval. Whisper accepts an initial prompt that conditions decoding [12]. Two-pass ASR refines first-pass output, either by rescoring streamed hypotheses with a second decoder [13], [14] or by correcting the first-pass N-best list with an LLM [15]. Retrieval-augmented generation supplies retrieved evidence to a generator [16]. Our two-pass glossary is an in-domain analogue: the first pass retrieves over the lecture itself, and the second pass conditions on the retrieved terms. Unlike rescoring or correction, we feed the terms back as context rather than editing the hypotheses, and we derive that list from the model’s own lecture-level output, so the method is reference-free.

3 Benchmark↩︎

We build the benchmark from a publicly available Mandarin-English AI/ML lecture series¹. The lectures are spontaneous speech, with English technical terms code-switched into Mandarin. We segment the lectures and pair each segment with its reference transcript.

We then build a term-rich test set. A rule-based recognizer extracts technical terms from the reference transcripts. It targets four kinds of terms: acronyms such as AI and LLM; model and tool names such as Llama, Whisper, and OpenClaw; code-like strings such as SWE-bench and log1.txt; and multi-word terms such as AI Agent and Context Engineering. We canonicalize each term with NFKC normalization, lowercasing, and light stemming, so surface variants map to one key. A segment enters the term-rich set when it contains at least one term.

Table 1 summarizes the benchmark. The full set has 29,043 segments and 16.92 hours. The term-rich test set has 7,443 segments and 5.01 hours, with 8,888 term occurrences and 1,030 unique terms over 15 lectures. Terms are bursty. Fig. 1 shows the term frequencies. The term token occurs 784 times, AI Agent 292 times, and the lecture-specific name OpenClaw 135 times. This burstiness is the signal that motivates lecture-level context. The distribution is heavy-tailed: acronyms and short model names lead the head, while multi-word terms and code-like strings fill the tail. A method must handle both the frequent acronyms and the rare composite names.

4 Term-Centric Evaluation↩︎

CER measures character accuracy over the whole segment. It does not tell us whether the recognizer captured the technical terms. We therefore add four term-centric metrics. We compute them over canonical term keys.

We define term recall as the fraction of reference term occurrences whose canonical key appears in the hypothesis of the same segment. We define term precision as the fraction of predicted term keys that appear in the reference. We define term F1 as their harmonic mean. We define term error rate as the edit distance between the reference and hypothesis term sequences, normalized by the reference term count.

We keep CER as the overall accuracy measure. We normalize all Mandarin text with OpenCC² (s2twp) and apply NFKC and lowercasing before scoring. A usable transcript needs both high recall and high precision, and CER captures neither. The two metrics can move in opposite directions: a recognizer that copies a long context into its output raises recall while collapsing precision, as Table 3 shows. We therefore report recall and precision together, and treat term recall as the headline metric because it counts how many technical terms the recognizer recovered.

Table 2 reports five recognizers under segment-only decoding. Breeze-ASR-25 has the best CER and the best term error rate. Yet it recovers only 52.50% of the terms. Whisper-large-v3-turbo has slightly higher term recall but much lower term precision. Every model leaves a large share of technical terms unrecognized. A low CER does not imply usable technical transcripts.

5 Method↩︎

We propose a two-pass, reference-free decoding method. We call it asr-glossary. The idea is simple. A lecture repeats its terms, so the model’s own first-pass output already contains those terms. We collect them, and we feed them back as context.

Algorithm 2 states the procedure. The first pass runs segment-only ASR over the lecture. We extract canonical terms from every first-pass hypothesis, and we rank them by frequency across the lecture. We keep the top \(k\) surface forms as the first-pass glossary. The second pass re-decodes each segment with this glossary as context.

We set \(k=30\). A short glossary keeps the prompt compact, so it avoids over-conditioning. Frequency ranking captures the bursty repeats, so the glossary holds the terms that matter for this lecture. A sweep over \(k\) (Sec. 6.1) confirms that \(k=30\) gives the lowest CER, and a selection ablation confirms that lecture-level frequency, not the mere presence of a term, is the signal that matters. We use the surface form, so the prompt reads like natural text.

Conditioning interface. The two model families consume the glossary differently. For Whisper-family models, including Breeze, we pass the glossary as an initial prompt through prompt tokens. We apply it to the first decoding window, we cap it at 160 tokens, and we decode greedily. For Qwen3-ASR, we pass the glossary through the native context field as a system message. The instruction tells the model to use the context for technical-term disambiguation and not to copy unspoken text. We call this last constraint the guard sentence. Qwen3-ASR is trained to treat such context as background knowledge [11], so the design matches the model.

Context variants. We compare the first-pass glossary against five alternative context sources and an oracle upper bound.

• Title: the lecture title only.

• Prev-ASR: the previous five first-pass segments.

• Combined: joins the lecture title, prev-ASR segments, and first-pass glossary.

• External list: a fixed, course-level ML/LLM term list, ranked by lecture-title match.

• Hybrid: fills the first 15 prompt slots from the external list and the remaining slots from the first-pass glossary.

• Oracle: prompts with terms from the reference transcript. Not deployable; reported as an upper bound only.

All variants except the oracle are reference-free.

6 Experiments↩︎

Setup. We decode five backbones with context, and we run the glossary ablations on Breeze-ASR-25. We rescore each segment-only baseline on the 5.01-hour term-rich subset, so its CER is comparable to the contextual rows. These subset CERs are higher than the full-set CERs in Table 2, because the term-rich subset is harder. We decode greedily unless stated otherwise. We test significance with a paired bootstrap over segments, using 2,000 resamples. All term-recall gains below are significant at \(p<0.001\) [17].

Main result. Table 3 reports the contextual results. The first-pass glossary raises term recall for all five backbones. The gain ranges from +6.95 on whisper-large-v3-turbo to +9.45 on Qwen3-ASR-0.6B across the four competitive backbones, and it reaches +17.59 on the weak Breeze-ASR-26. The glossary also holds or lowers CER on all five models. On Breeze-ASR-25 it cuts CER from 11.37% to 9.79% while lifting recall from 52.50% to 60.13%.

Combined context vs.first-pass glossary. The combined context (title + prev-ASR + glossary) helps some models and hurts others. It gives the best recall on Qwen3-ASR-1.7B and on the weak Breeze-ASR-26, but it raises CER on whisper-large-v3-turbo to 15.89% and on Qwen3-ASR-0.6B to 25.39%. The added length is the cause. A small model cannot manage long context, so the extra text corrupts decoding. The raw prev-ASR context is the worst case: on Qwen3-ASR-0.6B it alone drives CER to 34.01%, because the model copies whole mis-recognized sentences. The first-pass glossary avoids this failure. It is short, and it improves recall on every model. The glossary is therefore the robust first choice, and the combined context is a per-model option.

Closing the headroom. The oracle sits well above every deployable setting, at 68.88% recall on Breeze-ASR-25, 64.41% on Qwen3-ASR-1.7B, and 62.84% on Qwen3-ASR-0.6B. The gap is mostly a coverage problem. Of the terms only the oracle recovers, 55% to 65% never appear in any first-pass hypothesis. These are lecture-specific names: OpenClaw is heard as “open cloud”, ACON as “Aiken”, and AppWorld as “app world”. A first-pass glossary cannot recover them, so we add a small external list of 15 course-level terms per lecture, which still uses no reference transcript. Table 4 compares the sources on Breeze-ASR-25. The external list alone matches the first-pass glossary. The hybrid of the two reaches 62.05% recall and 9.40% CER, a significant +1.92 recall gain over the first-pass glossary. Relative to the segment-only baseline, the hybrid improves all four term metrics at once: it raises recall from 52.50% to 62.05% and precision from 80.55% to 82.73%, lifts F1 from 63.57% to 70.91%, and lowers CER from 11.37% to 9.40%, each significant at \(p<0.001\). The gain is therefore not a recall-for-precision trade. It is the strongest deployable setting we test.

6.1 Analysis↩︎

Glossary size. We sweep the glossary size on Breeze-ASR-25 over \(k \in \{10,20,30,50,100,\text{all}\}\). Recall rises with \(k\) and saturates near \(k{=}100\), since each lecture yields at most 106 first-pass candidates. CER is lowest at \(k{=}30\) (9.79%), and term F1 peaks at \(k{=}50\) (69.50%). Beyond \(k{=}50\), the recall gain shrinks while CER rises and precision drops. We keep \(k{=}30\) as a compact operating point that gives the lowest CER and near-peak F1.

Term-selection criterion. We fix \(k{=}30\) and change how the 30 terms are chosen. Frequency ranking gives the best recall (60.13%), F1, and CER. First-occurrence (57.27%) and random (58.58%) selection are significantly worse. Downweighting generic terms stays close on recall (59.90%) but still lowers F1 and precision. The bursty repeat count, not the mere presence of a term, is what makes the glossary work.

Where it helps. The recall gain spans all four term kinds (Table 5). It is concentrated on acronyms and model names for Breeze-ASR-25, and on acronyms and multi-word terms for the small Qwen3-ASR-0.6B, so a weak baseline has the most term headroom to recover. Every per-category gain is significant under paired bootstrap. For Qwen3, the guard sentence protects precision: removing it lowers precision on Qwen3-ASR-0.6B by 1.39 points and raises its CER.

Robustness checks. Under beam search (\(b{=}5\)) the recall gain holds at +7.56 and CER is non-worse, so it is not a greedy-decoding effect. The first-pass glossaries are clean: weighted purity is 95.56% on Breeze-ASR-25, 92.19% on Qwen3-ASR-1.7B, and 90.75% on Qwen3-ASR-0.6B, so the reference-free list is mostly real lecture terms. A third decoding pass that rebuilds the glossary from the second-pass output raises recall by a further +0.90 to 61.03%, but it does not improve F1, CER, or term error rate, because precision falls as the glossary grows. One refinement pass is therefore enough. The glossary also rescues near-miss forms: “aia” becomes AI agent, “lnn” becomes llm, “pump” becomes prompt, and “SW1 bench” becomes SWE-bench.

7 Conclusion↩︎

We studied ASR for Mandarin technical lectures, where CER hides technical-term failures. We measured term recognition directly with term recall, precision, F1, and term error rate, and we proposed a two-pass, reference-free glossary prompt built from first-pass hypotheses. It raises term recall on all five backbones while holding or lowering CER, and the lecture-level frequency of the terms, not their mere presence, drives the gain. On Breeze-ASR-25 a hybrid with a small external list reaches 62.05% term recall and 9.40% CER.

The method reframes contextual biasing as retrieval over the recording itself: the first pass retrieves, and the second pass conditions on the retrieved terms with no external catalog. Because term burstiness drives the gain, it should carry to other long-form code-switched audio such as meetings and seminars. The benchmark and the term-centric metrics also transfer: any domain with specialized vocabulary and low-CER models would benefit from evaluation that separates term accuracy from overall character accuracy.

Several limitations remain. The benchmark covers one instructor and one domain (AI/ML). Multi-speaker and cross-domain evaluation, for example on medical or legal lectures, would test whether the burstiness assumption generalizes. The term extractor is rule-based, so it misses terms that do not match pre-defined patterns. A learned extractor could improve both benchmark coverage and first-pass glossary quality. The largest open gap is coverage: 55% to 65% of oracle-only terms never appear in any first-pass hypothesis, because the model has never seen the surface form. Closing this gap may require retrieval from course-level materials such as slides and syllabi, which are typically available in educational settings.

References↩︎

---

1. Dataset, code, and term metadata will be released upon publication.↩︎

2. https://github.com/BYVoid/OpenCC↩︎