1 Introduction↩︎

Speech translation (ST) converts spoken language directly into text in another language. Two paradigms dominate: end-to-end ST (E2E ST), which maps audio directly to translated text, and cascaded ST, which combines an Automatic Speech Recognition (ASR) system with a Neural Machine Translation (NMT) model. Despite progress in E2E ST, recent benchmarks show that cascaded systems often match or outperform them [1]–[4].

However, a key limitation of cascaded ST is error propagation: ASR errors are passed to the NMT component, degrading translation quality. This effect is amplified by a training–inference mismatch, as NMT models trained on clean text must handle noisy ASR outputs at test time [5]–[9]. On the Vietnamese–English subset of FLEURS [10], this mismatch results in 6.79 to 10.64 BLEU drops when translating ASR outputs instead of clean transcripts (Section 5.2).

A common approach to improving NMT robustness is to augment training data with noisy or corrupted input text [5], [6]. However, [7] show that robustness gains depend on the match between injected noise and real error distributions, raising a question: what kinds of errors should we inject?

To answer this question, we conduct, to our knowledge, the first systematic analysis of ASR error patterns in Vietnamese speech translation. We categorize substitution errors by their phonetic causes, and quantify each category’s impact on translation quality using a linear mixed-effects model. Our analysis confirms that the majority of ASR substitution errors arise from systematic phonetic confusions rather than random noise and that phonetic ASR errors meaningfully impact NMT performance.

This finding directly motivates our approach: since ASR errors are predominantly phonetic, synthetic augmentation should also be phonetic. We propose Phonetically-Informed Data Augmentation (PiDA) using phonetic embeddings [11], [12], which generates corruptions by substituting syllables with phonetically similar alternatives identified through embedding similarity. To our knowledge, this is the first work to use explicit phonetic embeddings to generate synthetic ASR-style training data for machine translation on any language. We provide qualitative examples of phonetic corruption and recovery in 1.

We focus on Vietnamese as a case study for tonal, isolating languages. Our contributions are: (1) the first phonetic categorization and impact analysis of ASR errors for Vietnamese ST, revealing that most errors are systematic phonetic confusions; (2) phonetically-informed data augmentation using phonetic embeddings motivated by this analysis; (3) demonstration that fine-tuning on a mix of clean & PiDA-corrupted text improves ST (up to +2.04 BLEU) while improving MT performance. Our method avoids the text translation degradation caused by fine-tuning on real ASR outputs, and requires no audio data. Being text only, PiDA has the potential to be utilized in domains with available NMT data but limited speech resources.

2 Related Work↩︎

Vietnamese Speech Translation. Vietnamese ST remains underrepresented in the literature, with only two data sources providing (audio, transcripts, translation) triplets: the Vietnamese split of multilingual datasets MultiMed-ST [4] (16 hours, 9.1k samples) and FLEURS [10] (12 hours, 4.2k samples). This lack of high-quality datasets motivates our use of cascaded ST where ASR [13]–[15] and NMT [16]–[18] modules can be tuned separately with more abundant resources.

We leverage FLEURS for our ASR error quantification and data augmentation experiments. We also evaluated MultiMed-ST but found substantial audio-transcript misalignments (67 out of 100 manually verified samples contained an average of 2.39 missing and 1.57 erroneously present words per sample) that inflate observed error rates, making the error analysis and speech translation benchmarking on this dataset unreliable.

ASR Error Analysis for Machine Translation. [19] applied linear mixed-effects models to quantify the impact of ASR errors on statistical MT, decomposing errors into substitutions, insertions, and deletions. They found substitutions most detrimental to translation quality, though treated them as one single category. We extend the regression analysis of [19] by decomposing substitutions according to their phonetic cause.

Data Augmentation for Robust Spoken Language Tasks. [8] fine-tune the NMT component on paired clean and noisy transcripts to improve cascaded ST. [5] show that NMT robustness improves through synthetic noise augmentation when noise type and amount are carefully calibrated. [7] use adversarial training to improve robustness. Recently, MEDSAGE [9] extracts error statistics from real ASR outputs and uses a large language model (LLM) to generate ASR-like corrupted text matching these profiles. While effective for English dialogue summarization, MEDSAGE relies on an LLM’s implicit notion of phonetic plausibility, which may inadequately model true acoustic confusability for phonologically distant languages. We instead use phonetic embeddings to identify phonetically similar syllables.

Phonetic Word Embeddings. Phonetic representations range from rule-based articulatory features (PanPhon [20]) to learned phoneme embeddings such as Phoneme2Vec [21], with benchmarks like PWESuite [11] and PSET [12] evaluating their ability to capture phonetic similarity. More recently, XPhoneBERT [22], a multilingual phoneme-level Transformer pretrained on 330M phoneme sequences across nearly 100 languages (including Vietnamese), introduced contextualized phoneme representations. Originally, Phoneme2Vec incorporated phoneme embeddings as an additional input channel in classifier models for ASR-robust sentence classification, whereas XPhoneBERT replaced the phoneme encoder in VITS [23] to provide contextualized phoneme representations for Text-to-Speech. We instead adopt the pretrained XPhoneBERT embeddings to model phonetic similarity for data augmentation.

3 ASR Error Analysis↩︎

3.1 Error Extraction and Alignment↩︎

We use PhoWhisper-large (PhoWhisper-large) [24] and wav2vec2-base (wav2vec2-base-vietnamese-250h) [25], Vietnamese ASR models based on Whisper [26] and Wav2vec 2.0 [27], to transcribe the Vietnamese training split of FLEURS Vietnamese-English (3k samples). ASR outputs are word-aligned with reference transcripts using minimum edit distance, from which we extract substitution, insertion, and deletion errors.

3.2 Phonetic Categorization of Substitution Errors↩︎

For each substitution error, we classify it into one of six categories based on the phonetic relationship between the reference word and the ASR hypothesis:

• Vowel confusion: The substitution involves changes in vowel quality, vowel diacritics, or vowel length while the consonant structure remains similar. Example: sau\(_\text{after}\) \(\rightarrow\) sao\(_\text{star}\)

• Consonant confusion: The substitution involves consonant changes while vowels remain similar. Example: sử\(_\text{history}\) \(\rightarrow\) xử\(_\text{execute}\).

• Tonal confusion: The reference and hypothesis words are segmentally identical but differ in tone marking. Example: ma\(_\text{ghost}\) \(\rightarrow\) mà\(_\text{but}\).

• OOV (Out-of-vocabulary): The reference contains a foreign term, typically English vocabulary, that the ASR system transcribes as Vietnamese syllables approximating the foreign phonetics. Example: feet \(\rightarrow\) phít.

• NA (Not applicable): Substitutions where the reference and hypothesis are phonetically unrelated, possibly due to alignment errors or severe ASR failures.

• No error: Orthographic variants representing the same word, mostly due to typography in the reference transcription (e.g., hoà\(_\text{draw}\) \(\rightarrow\) hòa or tỉ\(_\text{billion}\) \(\rightarrow\) tỷ). These are not semantic errors.

Classification is performed using Gemini 2.5 Flash [28] with a prompt containing definitions, decision criteria, and examples for each category. As validation, we manually labeled the top 100 substitution errors and compared against LLM labels, achieving agreement in 94 cases and yielding a Cohen’s \(\kappa\) = \(0.95\). Disagreements primarily involved ambiguous cases where multiple phonetic differences co-occurred. In these cases, the LLM classifier typically assigns the dominant phonetic cause.

3.3 Impact Quantification via Mixed-Effects Modelling↩︎

Following [19], we use a Linear Mixed-Effects Model (LMM) to quantify how each error type impacts translation quality while controlling for differences between ASR systems. We use VinAI-Translate (vinai-translate-vi2en-v2) [29] (a fine-tuned mBART [30], leading model in recent Vietnamese NMT benchmarks [4], [16]) to translate the ground-truth text of FLEURS and the ASR outputs of wav2vec2-base and PhoWhisper-large.

We define the dependent variable in our LMM as: \[\Delta\text{TER} = \text{TER}_{\text{ST}} - \text{TER}_{\text{MT}}\] where \(\text{TER}_{\text{ST}}\) is Translation Edit Rate using ASR output as NMT input and \(\text{TER}_{\text{MT}}\) uses reference transcripts. This difference isolates translation degradation specifically attributable to ASR errors. We use TER rather than BLEU following [19] for its favorable statistical properties in linear modeling.

The mixed-effects model takes the form: \[\Delta\text{TER} = \beta_0 + \sum_i \beta_i \cdot x_i + \gamma \cdot \text{ASR}_{\text{sys}} + u_j + \epsilon\] where \(x_i\) represents the normalized frequency (count divided by sentence length) of each error type, \(\text{ASR}_{\text{sys}}\) is a fixed effect controlling for differences between the two ASR systems, and \(u_j \sim \mathcal{N}(0, \sigma^2_u)\) is a random intercept per sentence \(j\), accounting for variation in translation difficulty across utterances.

3.4 Results↩︎

1 shows the error distribution across both ASR systems. Among within-vocabulary errors, phonetic confusions are most prevalent: vowel and consonant confusions together account for 3,050 errors in wav2vec2 and 1,013 in PhoWhisper, outnumbering tonal confusions and phonetically unrelated substitutions.

2 presents mixed-effects coefficients quantifying translation impact¹. Insertions and deletions show the largest effects, consistent with prior work [19]. Among substitution types, phonetic confusions have the most dominant impacts on ST degradation. Vowel confusions have the strongest effect (97.66), followed by tonal (79.12) and consonant confusions (71.84), while phonetically unrelated errors (NA) are insignificant.

4 Phonetically-Informed Data Augmentation↩︎

We propose the Phonetically-Informed Data Augmentation (PiDA) pipeline, consisting of six components across two phases:

4.1 Precomputation Phase↩︎

1. Syllable Inventory Construction. We extract the Vietnamese syllable inventory using the wordfreq library [31], which provides frequency statistics derived from large web corpora². We retrieve the top 50,000 most frequent Vietnamese words and filter to entries matching valid Vietnamese orthographic patterns, yielding approximately 9,400 unique syllables.

2. Phoneme Conversion. Each syllable is converted to International Phonetic Alphabet (IPA) using CharsiuG2P [32], the grapheme-to-phoneme system used by XPhoneBERT.

3. Embedding Extraction. We pass each phoneme sequence through XPhoneBERT (xphonebert-base) [22] and mean-pool the output hidden states across all phoneme positions (excluding the [CLS] and [SEP] tokens) to obtain a 768-dimensional vector per syllable.

4. Similarity Index Construction. We L2-normalize all syllable embeddings and build a FAISS index [33] using inner product search for efficient approximate nearest-neighbor retrieval. For each syllable, we precompute its top-50 phonetically similar neighbors and their cosine similarity scores.

4.2 Augmentation Phase↩︎

5. Error Annotation. We follow the error annotation procedure of [9]: Given an input sentence to corrupt, we annotate individual words with error markers based on ASR error statistics computed from the training set. For each word \(w_i\), we sample from a Bernoulli distribution with probability \(p = \text{WER}_{\text{train}}\) (the observed word error rate on the training set). If selected for corruption, we sample an operation type (deletion or substitution) according to the observed proportions of these error types in real ASR errors³. We mark substitutions with braces \(\{w_i\}\) and deletions by omitting the word entirely.

6. Corruption. We then process the annotated text to generate the corrupted output. For deletion markers, the word is simply removed. For substitution markers, we perform phonetic corruption with the following procedure.

For each syllable \(s_i\) marked for substitution, we sample a replacement from its top-\(k\) (\(k=5\)) precomputed phonetic neighbors using temperature-scaled softmax sampling:

\[P(s_j \mid s_i) = \frac{\exp\bigl(\text{sim}(s_i, s_j) / \tau\bigr)}{\sum_{s_k \in \mathcal{N}_k(s_i)} \exp\bigl(\text{sim}(s_i, s_k) / \tau\bigr)}\]

where \(\text{sim}(s_i, s_j)\) is the cosine similarity between syllable embeddings, \(\mathcal{N}_k(s_i)\) denotes the top-\(k\) neighbors of \(s_i\), and \(\tau\) is a temperature parameter controlling the sharpness of the distribution (lower values favor more phonetically similar substitutions).

5 Experiments↩︎

5.1 Experimental Setup↩︎

Data. We use PiDA to corrupt the training split of FLEURS, then evaluate the fine-tuned models on 0.9k test samples.

Models. We use PhoWhisper-large and wav2vec2-base as the ASR systems, and VinAI-Translate as the NMT.

Training protocol. We fine-tune VinAI-Translate with: 3 epochs, batch size 8, learning rate \(3\times10^{-5}\), maximum sequence length 256 tokens, gradient accumulation over 8 steps (effective batch size 64), 300 warmup steps, weight decay 0.01, AdamW optimizer with early stopping based on validation loss.

Baselines. We compare against: (1) Clean pairs: baseline fine-tuning on reference Vietnamese-English pairs only; (2) Random frequency-based substitutions [5]: corrupting reference transcripts by sampling error operations (deletion, substitution) according to WER statistics from real ASR outputs, with substitutions drawn from frequency-weighted Vietnamese syllable inventory (the same as PiDA’s); (3) Real noisy pairs [8]: using actual ASR outputs paired with English translations; (4) MEDSAGE [9]: similarly sampling error operations from ASR error profiles, but using LLM-generated insertions and substitutions via Gemini 2.5 Flash instead of random vocabulary draws⁴. In all conditions, when clean and noisy data are mixed (denoted as clean & {method name}), the ratio is always 1:1 (3k clean & 3k noisy), following [8].

Evaluation. We report BLEU [38] and COMET [39] (Unbabel/wmt22-comet-da) on clean reference transcripts (MT) and on PhoWhisper-large and wav2vec2-base ASR outputs (ST). We abbreviate the metrics as Metric\(_\texttt{Setting}^\texttt{Model}\). For example, BLEU\(_\text{ST}^\text{PW}\) refers to the BLEU score for speech translation using the PhoWhisper-large ASR outputs.

5.2 Results↩︎

3 presents our main results across training conditions.

Baseline and ASR error propagation. Fine-tuning on clean reference FLEURS pairs improves speech translation quality, but a substantial MT-ST gap remains (6.79 BLEU for PhoWhisper-large and 10.64 BLEU for wav2vec2-base), indicating that domain adaptation alone does not address robustness to ASR errors, particularly for weaker ASR front-end.

Random substitutions & Real noisy pairs. Frequency-based random corruption yields ST gains (27.45–27.46 BLEU\(_\text{ST}^\text{PW}\), \(p < 0.05\)) comparable to mixing clean with real noisy text (27.43 BLEU\(_\text{ST}^\text{PW}\), \(p < 0.05\)) without degrading BLEU\(_{\text{MT}}\). Fine-tuning with only real noisy pairs increases both BLEU\(_\text{ST}^\text{PW}\) (28.06) and BLEU\(_\text{ST}^\text{w2v}\) (23.65), both \(p < 0.05\). However, noisy-only training significantly degrades BLEU\(_{\text{MT}}\) by 1.04 (\(p < 0.05\)).

MEDSAGE. LLM-generated corruptions yield no significant BLEU improvement over the clean baseline for either ASR system, though COMET scores are similar to other methods.

PiDA (ours). Mixing clean text with PiDA achieves the strongest BLEU\(_\text{ST}^\text{PW}\) (+2.04 over baseline, +0.84 over random substitutions, \(p < 0.05\)) and significant gains in BLEU\(_\text{ST}^\text{w2v}\) (+0.78, \(p < 0.05\)), while preserving both BLEU_MT and COMET_MT. Real-noisy-only training achieves slightly lower BLEU\(_\text{ST}^\text{PW}\) and slightly better BLEU\(_\text{ST}^\text{w2v}\), but degrades BLEU_MT. Mixing clean data & PiDA is the only method that improves speech translation across both ASR systems while also improving clean-text performance. Without clean pairs, PiDA still preserves MT quality and delivers significant ST gains. COMET scores across all augmentation methods are comparable.

Overall, PiDA consistently provides robustness gains comparable to real noisy data while eliminating the robustness-accuracy tradeoff, and also requiring neither audio data nor external LLMs for text augmentation.

Sampling hyperparameter ablation. We analyze the sensitivity of PiDA to the sampling hyperparameters \(k\) and \(\tau\) in 4, showing that performance is stable across a wide range of values. The best BLEU\(_\text{ST}^\text{{PW}}\) and BLEU\(_\text{ST}^\text{{w2v}}\) are achieved with \(k{=}5\) and \(\tau{=}0.5\), which we used in our main experiments.

6 Conclusion & Future Work↩︎

We presented the first systematic categorization of ASR errors for Vietnamese ST, showing that most substitution errors stem from structured phonetic confusions, and that these errors substantially affect downstream NMT performance.

This motivated Phonetically-Informed Data Augmentation using XPhoneBERT embeddings (PiDA), which generates synthetic errors by substituting phonetically similar syllables. Across ASR systems, PiDA achieves consistent gains in ST, scoring the highest BLEU\(_\text{ST}\) when translating PhoWhisper-large outputs and significant improvements for wav2vec2-base, while retaining or improving text translation quality on both BLEU and COMET. In contrast, real-ASR augmentation yields comparable ST gains but degrades MT performance. Being text-only, PiDA has the potential to extend to ST domains where NMT parallel data exists but domain-specific speech resources are limited.

Limitations We only evaluate on one dataset due to the lack of high-quality ST datasets for Vietnamese. OOV errors involve cross-lingual phonetic mapping not addressed by our within-vocabulary method. Future work will extend to OOV handling and validate on additional Vietnamese ST benchmarks.

7 Acknowledgments↩︎

The research results are a part of the outputs of the Cross-College project Robust Vietnamese–English Clinical and Educational Medical Translation, a collaboration between the College of Engineering & Computer Science (CECS) and the College of Health Sciences (CHS), VinUniversity. This research was funded under Project ID VUNI.2324.CC06.

Giang Son Nguyen, Tung X. Nguyen, Hieu Minh Truong and Dung D. Le are partly supported by the Center for AI Research, VinUniversity. Nhu Vo is supported by the Vingroup Scholarship.

Giang Son Nguyen thanks the AUMOVIO-NTU Corporate Lab for lending GPU resources for the experiments in this paper.

8 Generative AI Use Disclosure↩︎

The authors used generative AI tools solely to assist with minor language editing and readability improvements. No generative AI tools were used to produce any scientific content, experimental results, data analysis, or conclusions.

References↩︎