SamaVaani: Auditing and Debiasing
Multilingual Clinical ASR for Indian Languages
June 25, 2026
Automatic Speech Recognition (ASR) is increasingly used to document clinical encounters, yet its reliability in multilingual and demographically diverse Indian healthcare context remains largely unknown. In this study, we first conduct the systematic audit of ASR performance on real-world psychiatric interview data spanning Kannada, Hindi and Indian English, comparing eight state-of-the-art models including IndicWhisper, WhisperLargeV3, Sarvam, GoogleS2T, Gemma3n, OmniLingual, Vaani, and Gemini. Our results reveal substantial variability across models and languages, with some systems performing competitively in Indian English but failing in regional speech. We further fine-tune two of the best performing open-source models, i.e., Gemma3n and OmniLingual, using various methods. With this, we uncover systematic performance gaps tied to speaker role and gender, raising concerns about equitable deployment in clinical settings, which are further mitigated by fairness-aware fine-tuning. To this end, we propose SamaVaani, a unified debiasing technique that simultaneously improves ASR performance and improves fairness across demographic groups.
The field of psychiatry is highly dependent on language, with a detailed psychiatric interview the primary diagnostic tool [1]
rather than laboratory or radiological investigations. Subsequently, verbatim transcripts of these interviews are widely used for clinical diagnosis, academic training, qualitative research, and, more recently, for the development of AI systems in
psychiatric tasks [2]. However, producing accurate transcripts remains a major bottleneck: manual transcription is strenuous, time-intensive
(5-8 hours for every hour of audio) and prone to human error [3]. ASR systems provide a scalable alternative albeit transcription
errors [4] in psychiatric settings can significantly change clinical interpretation [5], [6]. Modern ASR systems, including proprietary platforms
(GoogleS2T [7], Microsoft Azure [8], Amazon Transcribe [9]) and open-source models like Whisper [10] have improved transcription quality. Although these systems work well for standard English
(American and British) and in controlled environments, performance deteriorates substantially for non-standard English, conversational and accented English [11], code-mixing and code-switching, which is common in multilingual contexts like India [12], [13]. In addition, Indian English and Indian regional languages remain underrepresented in global
training corpora, leading to lower accuracy and greater variability [14], [15].
These difficulties are amplified with clinically distinctive speech patterns in psychiatric interviews. For instance, low tone, hesitations, and long pauses are common in depression [16]–[18]; fast and loud speech in mania [19]–[21]; stammering and repeating in anxiety [22]–[24]; and disordered agrammatical speech containing
neologisms (made-up words to which the patient attaches special meaning) in schizophrenia [25]–[27]. Beyond this, recordings are often made under acoustically difficult conditions (in wards with noise from ceiling fans, hospital instruments, or ambulance sirens) [28]. Another big concern is the fairness and accuracy of the transcription. These problems may be magnified in psychiatric interviews, as doctors and patients are very different in terms of
education, socio-economic background, conversational role, and speaking style [13], [29]. While recent advances in multilingual ASR from open-source models such as Whisper [10], IndicWhisper [30], OmniLingual [31], Vaani [32] and commercial models like Sarvam [33] have improved support for multilingual and regional speech in India, existing evaluations rely mainly on general-purpose datasets and do not capture the complexities of real-world psychiatric
interviews. Our work addresses this gap by systematically analyzing and further improving the performance of ASR on multilingual psychiatric interactions.
Our contributions and findings: In this work, we present the first systematic audit of ASR systems on real-world multilingual psychiatric interviews in the Indian context. Leveraging a novel dataset spanning Kannada, Hindi, and Indian
English, we evaluate eight state-of-the-art ASR models across languages, speaker roles, and demographic groups, and introduce a comprehensive fairness analysis grounded in WER and fine-grained error patterns. We find substantial variability in performance
across models and languages, with consistently higher error rates for low-resource languages such as Kannada, as well as systematic disparities across speaker roles and gender. Therefore, we propose SamaVaani, a simple yet
effective fairness-aware fine-tuning framework combining contrastive learning and CTC alignment, which significantly improves both transcription accuracy (up to \(\sim 50\%\) WER reduction) and fairness across demographic
groups. Together, our study highlights critical limitations of current ASR systems in clinical settings and provides actionable pathways toward more equitable and robust multilingual ASR deployment in healthcare.
Research on ASR in Indian multilingual psychiatric settings spans three interconnected areas: (i) ASR for psychiatric interviews, (ii) Bias and speaker-level disparities in ASR, and (iii) ASR for Indian languages and accents. We highlight the
challenges and gaps in each area that motivate the current study.
ASR for psychiatric interviews: A growing set of studies has examined the use of ASR to transcribe or analyze psychiatric interviews. [5] and [34] evaluated ASR in schizophrenia patients, comparing them with healthy controls using interviews
conducted in Dutch and German respectively. Several studies [35], [36] in Spanish and French psychiatric interviews in patients with psychosis has found high WER performance. Lastly, [37] studied ASR in psychiatric interviews in American English in patients with depression vs. controls with past history of depression, and found no difference in WER in either group.
Bias and speaker-level disparities in ASR: Prior works has identified gender differences in YouTube ASR [29], higher
error rates for African-American speakers in commercial ASR systems [13], and accent-based inequity among non-native German speakers in
clinical contexts [34]. Biases related to age, gender, accents and low-resource languages also affect the performance of ASR [38]. However, these studies are not designed for multilingual psychiatric interviews, where conversational roles are inherently unequal with
clinicians typically producing longer, more structured speech and patients providing shorter and more uncertain responses.
ASR for Indian languages and accents: Recent work has focused on ASR challenges for Indian English and regional Indian languages. Svarah had significantly higher WER for English with Indian accent than for LibriSpeech [39]. [15] observed substantial
disparities in gender, region, and speech rate by analyzing 8,740 hours of NPTEL’s Indian lecture in English. Considering Indic languages the large datasets IndicSUPERB and IndicVoices further highlight the linguistic,
morphological, and prosodic diversity of Indian languages which pose challenges to ASR [40], [41]. However, they do not contain clinical conversations and analysis of error types. In this regard, the Eka Medical ASR Evaluation dataset [42] offers valuable Indian accents and drug vocabulary with over 3,900 recordings but is limited to brief, static conversations. On the other hand, the
DISPLACE-M dataset contains 55 hours of annotated conversational speech in the healthcare domain. However, this dataset lacks interviews of psychiatric cases.
| Characteristics | Overall | English | Hindi | Kannada | \(p\)-value\(^\#\) |
| N = 202\(^*\) | N = 54\(^*\) | N = 78\(^*\) | N = 70\(^*\) | ||
| Duration | 30.9 | 36.5 | 25.3 | 27.2 | \(<\)0.001 |
| (minutes) | (6.1, 45.6) | (28.7, 51.1) | (4.5, 36.5) | (3.6, 45.2) | |
| Total words | 4756.5 | 5636.5 | 3877.5 | 3637.0 | \(<\)0.001 |
| (1042, 6111.5) | (4331, 6216.8) | (845.8, 7135) | (526, 4873) | ||
| Unique words | 1152.0 | 1269.5 | 885.5 | 1450.5 | \(<\)0.001 |
| (407.5, 1412.2) | (1028.8, 1404.2) | (316.2, 1185.5) | (248.8, 1759.8) | ||
| Moving average | 0.64 | 0.61 | 0.64 | 0.69 | \(<\)0.001 |
| Type-token ratio | (0.61, 0.67) | (0.59, 0.62) | (0.62, 0.66) | (0.67, 0.71) | |
| (window = 100) |
The data for this study comes from a tertiary teaching hospital dedicated to the treatment of psychiatric and neurological conditions. The hospital provides free inpatient and outpatient treatment for economically disadvantaged patients, and thus, a majority of beneficiaries are from such a background. We collected 202 audio recordings of patient and doctor/therapist interactions. These recordings were collected using Android mobile phones in mp3 format. While an attempt was made to make recordings in a quiet environment, no special arrangements were made for this. Therefore, the data represent a real-world setting where recordings are made in busy wards and outpatient department rooms. The language, duration and lexical diversity of the dataset are summarised in Table 1 and the speaker profiles are detailed in Table 2.
| Characteristics | Overall | English | Hindi | Kannada | \(p\)-value\(^\#\) |
| N = 202\(^*\) | N = 54\(^*\) | N = 78\(^*\) | N = 70\(^*\) | ||
| Patient’s gender | |||||
| F | 51 (25.2%) | 2 (3.7%) | 18 (23.1%) | 31 (44.3%) | \(<\)0.001 |
| M | 151 (74.8%) | 52 (96.3%) | 60 (76.9%) | 39 (55.7%) | \(<\)0.001 |
| Doctor’s gender | |||||
| F | 104 (51.5%) | 54 (100%) | 30 (38.5%) | 20 (28.6%) | \(<\)0.001 |
| M | 98 (48.5%) | 0 (0%) | 48 (61.5%) | 50 (71.4%) | \(<\)0.001 |
| Patient’s education level | |||||
| \(<\)Graduate | 132 (65.3%) | 18 (33.3%) | 58 (74.4%) | 56 (80.0%) | \(<\)0.001 |
| \(>=\)Graduate | 70 (34.7%) | 36 (66.7%) | 20 (25.6%) | 14 (20.0%) | \(<\)0.001 |
This dataset contains speech from 130 unique speakers, including 7 doctors/therapists and 123 patients. All conversations are between two individuals, making 202 unique doctor-patient/therapist-patient pairs.
Preprocessing: All recordings were listened to by two psychiatrists to ensure they were intelligible. It was ensured that the recordings did not contain the name of the patient or any numerical identifier like phone number, etc. However,
we did not exclude segments that contained names of places, dates, etc., as we wish to evaluate if such named entities (Section 7) lead to more errors in ASR.
Annotation: As part of earlier research, transcripts in native languages were available for 103 of these recordings. For the remaining transcripts (99), two psychiatrists transcribed the recordings. The annotation guidelines for
transcribing speech into text are given in the Appendix 13 with examples for each of the three languages.
In this section, we first note the base ASR models that we use to perform the audit. We also discuss different fine-tuning approaches to improve the WER and the fairness of the base models. Finally, we introduce the debiasing algorithm used to develop the SamaVaani framework.
Base models: We evaluate a total of eight ASR models, as mentioned earlier. These include IndicWhisper [30], WhisperLargeV3 [43], Sarvam [33], GoogleS2T [7], Gemma3n [44], OmniLingual [31], Vaani [32], and Gemini [45]. Of these, GoogleS2T, Sarvam, and Gemini are proprietary models inferenced through APIs, while others are open-source. Recall that we have the audio files in an interview format where each of them has exactly two speakers, i.e., the patient and the doctor. We generate transcripts for each of these long-form audio files. Couple of these ASR models (Sarvam’s Saarika-2.5 and Gemini) can generate transcripts with long-form audio as input while the other models (IndicWhisper, WhisperLargeV3, Vaani, Gemma3n and GoogleS2T) can only transcribe audio in 30-second chunks.
Fine-tuned models: One of the straightforward ways to improve the overall WER and fairness across demographic groups is fine-tuning the base models. For this we choose two of the best performing open-source ASR models – Gemma3n1 and OmniLingual that have the best scores across groups (see Section 6).
We perform two types of fine-tuning as follows.
FT\(^\mathrm{\small Std.}\): This refers to the standard LoRA fine-tuning using part of our dataset for training.
FT\(^\mathrm{\small PS}\): Here we double the dataset by augmenting the pitch of the audio files. The hypothesis is that this augmentation of synthetic data would result in stronger fine-tuning, and therefore better WER and
fairness. For our experiments, we have used PitchShift2 to augment our original audio by randomly selecting semitones in the range of [\(-5, +5\)].
While straightforward fine-tuning can potentially improve the WER and the fairness of the models they do not specifically target the issue of debiasing. Recall that the transformer based ASR models follow the modern encoder decoder paradigm where the
encoder learns the audio representation and the decoder follows the next word prediction task to generate transcription. Our approach integrates a contrastive learning and a CTC (connectionist temporal classification) [46] head in the decoding stage while fine-tuning. We only train the LoRA adapters to adapt for Indic languages, freezing the transformer layers as shown in Figure 1. The entire architecture and the associated loss functions are discussed below.
Contrastive learning: There are several studies that shows the effectiveness of improving fairness when using a contrastive learning approach [47]–[50]. As in the case of FT\(^\mathrm{\small PS}\) setup, here also we have used the PitchShift algorithm to augment our original audio by randomly selecting
semitones in the range of [\(-5, +5\)]. Thus, in a batch of \(N\) audio samples, we have one original sample and its corresponding pitch-shifted sample constituting a positive pair, and the
rest of the \(N-1\) audio samples are naturally considered as negative pairs. This forces the model to learn the same utterance regardless of the pitch (a key point of distinction between demographic groups). Further, this
doubles the training data and makes the model robust to the high phonetic variance found in languages like Kannada. For an anchor representation \(z_a\) and its pitch-shifted positive pair \(z_p\), with \(N-1\) remaining negative original samples, the contrastive loss for each sample is formulated as: \[\mathcal{L}_{CL} = -\log
\frac{e^\frac{\text{sim(}z_a, z_p\text{)}}{\tau}}{e^\frac{\text{sim(}z_a, z_p\text{)}}{\tau} \text{ + } \Sigma_{k=1}^{N-1} e^{\frac{\text{sim(}z_a, z_{n,k}\text{)}}{\tau}}}\] where \(z_{n,k}\) are the other \(N-1\) audio samples in the batch which are considered as negative pairs and \(\tau\) is the temperature that dictates the sharpness of the probability distribution, ensuring the model is heavily
penalized for any phonetic overlap between the anchor and the negative samples. The contrastive loss implemented is a variation of the NT-Xent (normalized temperature-scaled cross entropy) loss [51] which is the standard objective for self-supervised frameworks like SimCLR [52]. Here, we only
create the pitch augmentation for the anchor file.
CTC head: In this stage, the logits from last hidden state (\(H\)) of transformer layers is projected to the vocabulary space (\(V\)). By projecting the last hidden states
through the CTC head, the model receives a secondary signal that rewards correct character-level sequencing. This head is initialized from scratch and is fully trainable (\(W_{CTC}\) = \(H\times
V\)) with CTC loss function represented as, \[\mathcal{L}_{CTC} = -\log P\left(y \mid x\right)\] where \(x\) = (\(x_0, x_1, \dots, x_T\)) denotes the
input sequence of audio features, \(T\) is the time steps, \(y\) = (\(y_0, y_1, ... , y_m\)) is the target label sequence text (transcription), and \(P\left(y \mid x\right)\) is the total probability of correct transcription over various alignment paths. The CTC loss enables alignment-free training by allowing the model to learn mappings between input frames and output
sequences without explicit frame-level labels.
Overall loss function: The weighted sum of the standard cross entropy (CE) loss, the contrastive loss (CL) and the CTC loss constitutes the final loss as follows. \[\mathcal{L}_{total} = \mathcal{\alpha \times L}_{CE} \text{ + } \mathcal{\beta \times L}_{CL} \text{ + } \mathcal{\gamma \times L}_{CTC} \label{eqn:total95loss}\tag{1}\] where \(\alpha,
\beta\), and \(\gamma\) are the weights for each of the loss components. These weights are optimized using Optuna3 over 20 trials. The best values of \(\alpha, \beta\), and \(\gamma\) are selected based on the WER score on the validation set.
LoRA adaptation: We train all the attention and MLP modules of the transformer based on the above loss function in eq. ( 1 ). This allows the model to repurpose the internal representations for Indic
languages with less compute. In addition, it contributes toward the causal cross entropy loss for next word prediction in the speech to text task. Finally, it ensures that the model retains the ability to generate contextually and grammatically correct
sentences in text from the speech in a multilingual setting.
Performance metric: The main indicator used to assess the accuracy of ASR transcription is the word error rate (WER). Since WER is the most widely used metric for assessing ASRs and has been utilized by several researchers in the literature, we
have chosen it as the evaluation metric. The WER is mathematically defined as the ratio of the sum of substitutions (\(S\)), deletions (\(D\)), and insertions (\(I\)) to the total number of words (N) in the reference transcript. \[\mathcal{WER \%} = \frac{S \text{ + } D \text{ + } I}{N} \times 100\] First, we normalize both the ground truth and the
generated transcripts by preprocessing the text. This preprocessing includes lowercasing, removing punctuation, and standardizing numbers to reduce superficial mismatches. Second, we use the JiWER Python library for the calculation of \(\mathcal{WER}\).
Fairness metric: The fairness score (\(\mathcal{FS}\)) combines a couple of components, namely, the \(\mathcal{WER}\) gap and the average \(\mathcal{WER}\) among groups (say two groups – \(\mathcal{G}_1\) and \(\mathcal{G}_2\)). \[\mathcal{FS} = - \delta \times
\mathcal{WER}_{avg} - \theta \times \mathcal{WER}_{gap}; \delta, \theta>=0\] where \(\mathcal{WER}_{avg}\) and \(\mathcal{WER}_{gap}\) are the average \(\mathcal{WER}\) of the two groups (\(\mathcal{G}_1\), \(\mathcal{G}_2\)) and the absolute difference in \(\mathcal{WER}\)
between the groups (\(\mathcal{G}_1\), \(\mathcal{G}_2\)) respectively. A lower \(\mathcal{WER}_{gap}\) suggests better fairness across groups. In addition,
a higher value of \(\mathcal{FS}\) (ranging from -\(\infty\) to 0) indicates overall balanced performance across groups. In this study, we set \(\delta =
\theta\) = 0.5 to assign equal importance to both units.
We employed Gemma3n and OmniLingual multimodal transformer-based models as the backbone for all of our experiments. We implement LoRA by injecting trainable rank 8 decomposition matrices into the query (\({q}_{proj}\)), key (\({k}_{proj}\)), value (\({v}_{proj}\)), output (\({o}_{proj}\)) and MLP projection layers (\({gate}_{proj}\), \({up}_{proj}\), and \({down}_{proj}\)). The experimental configuration and the different hyperparameters are noted in Appendix 11.
| Models | WER (%) | (\(S, D, I\)) % |
|---|---|---|
| 34.33 | (8.94 ,16.40, 36.87) | |
| 39.03 | (14.47, 6.76, 39.15) | |
| 54.37 | (27.87, 17.68, 39.03) | |
| 74.60 | (36.13, 40.22, 9.71) | |
| 85.55 | (10.09, 75.60, 0.13) | |
| 94.90 | (11.34, 84.01, 0.005) | |
| 14.15 | (5.28, 18.58, 4.94) | |
| 18.52 | (11.32, 4.35, 8.74) | |
| 35.01 | (22.71, 16.86, 7.53) | |
| 46.76 | (9.48, 19.81, 21.76) | |
| 71.68 | (26.23, 39.17, 9.21) | |
| 98.55 | (22.22, 76.51, 0.0014) | |
| - | - | |
| 70.3 | (23.44, 45.58, 4.27) | |
| 97.05 | (29.03, 68.21, 0.02) | |
| - | - | |
| 44.42 | (21.54, 15.60, 6.29) | |
| 77.21 | (5.56, 24.92, 1.73) | |
| 40.22 | (14.63, 17.64, 19.54) | |
| 48.14 | (19.88, 6.45, 26.48) | |
| 90.90 | (48.57, 16.67, 30.47) | |
| 58.64 | (23.40, 31.20, 4.00) | |
| 43.55 | (23.76, 12.20, 6.53) | |
| 75.35 | (41.08, 32.19, 2.06) |
This section is divided into two major parts. In the first part we audit the performance of the eight ASR models across three languages – Hindi, Kannada and Indian English. Next, we present the performance of the different fine-tuning methods as well as
the debiasing algorithm SamaVaani proposed by us.
Audit outcomes: Table 3 reports the WER and its constituent parts, substitution (\(S\)), deletion (\(D\)), and insertion (\(I\)) in percentages, to compare ASR models in all the three languages. As we observe from the table, Gemini achieves the best WER scores (English: 14.15%, Hindi: 18.52%, Kannada: 35.01%),
demonstrating better multilingual and generalization abilities. Among the three languages, Kannada poses relatively higher challenge to the ASR models possibly due to the lack of enough pre-training data. Further, models like WhisperLargeV3, GoogleS2T, and Gemma3n show a very high WER for Kannada because their architectures and training corpora are not deeply optimised for rich Indian
phonetic diversity and retroflex phonemes in Dravidian languages.
Fine-tuning and debiasing results: For fine-tuning, we choose Gemma3n and OmniLingual as both have the lowest \(\mathcal{WER}\) averaged over
three languages. We split the data into train, development (dev) and test folds with language as a stratification variable. Train set had 83.16 hours of audio (English 30.34, Hindi 27.56, Kannada 25.26), dev set had 9.64 hours (English 3.52, Hindi 3.02,
Kannada 3.10) and test set had 10.22 hours (English 3.80, Hindi 2.96 and Kannada 3.46 hours).
| Metric | Base | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall WER (\(\downarrow\)) | 70.47 | 65.85 | 47.62 | 47.41 | 41.14 | 45.83 | 39.08 | 40.7 | 37.92 | 38.12 | 35.19 | 35.29 |
| English WER (\(\downarrow\)) | 57.92 | 47.02 | 25.60 | 26.02 | 24.13 | 24.13 | 23.17 | 24.00 | 22.22 | 22.44 | 20.43 | 20.54 |
| Hindi WER (\(\downarrow\)) | 58.33 | 63.44 | 44.34 | 43.95 | 38.80 | 42.31 | 37.25 | 38.23 | 36.36 | 36.00 | 33.08 | 32.96 |
| Kannada WER (\(\downarrow\)) | 84.48 | 90.36 | 75.00 | 75.00 | 61.11 | 72.36 | 58.65 | 60.94 | 57.82 | 56.41 | 52.84 | 52.88 |
| Male (M) (\(\downarrow\)) | 69.23 | 76.55 | 53.33 | 53.19 | 44.23 | 50.00 | 41.77 | 42.31 | 40.71 | 39.76 | 37.25 | 37.25 |
| Female (F) (\(\downarrow\)) | 71.43 | 74.62 | 48.53 | 48.37 | 42.04 | 42.86 | 39.41 | 42.31 | 38.33 | 39.08 | 34.83 | 35.48 |
| \(\mathcal{FS}\): M vs F (\(\uparrow\)) | -36.27 | -43.76 | -27.87 | -27.80 | -22.66 | -26.79 | -21.48 | -22.46 | -20.95 | -20.05 | -19.23 | -19.07 |
| Patient (P) (\(\downarrow\)) | 76.47 | 84.44 | 53.85 | 53.85 | 47.62 | 49.04 | 45.45 | 47.49 | 44.71 | 43.40 | 41.67 | 41.18 |
| Doctor (D) (\(\downarrow\)) | 63.16 | 66.92 | 50.00 | 50.00 | 41.54 | 46.18 | 39.24 | 41.46 | 37.60 | 38.03 | 34.40 | 34.69 |
| \(\mathcal{FS}\): P vs D (\(\uparrow\)) | -41.56 | -51.60 | -27.88 | -27.88 | -25.33 | -26.48 | -24.28 | -25.25 | -24.13 | -23.04 | -22.65 | -22.21 |
| \(>\)=Graduate (\(\downarrow\)) | 63.64 | 75.23 | 28.45 | 28.28 | 26.66 | 28.45 | 25.16 | 27.27 | 25.16 | 26.43 | 22.28 | 22.42 |
| \(<\)Graduate (\(\downarrow\)) | 80.00 | 83.44 | 70.43 | 70.47 | 61.53 | 64.80 | 59.17 | 62.17 | 58.82 | 57.14 | 53.64 | 53.39 |
| \(\mathcal{FS}\): \(>\)=G vs \(<\)G (\(\uparrow\)) | -44.09 | -48.07 | -45.71 | -39.81 | -39.49 | -38.90 | -38.09 | -36.81 | -37.83 | -36.25 | -34.66 | -34.44 |
Main results: We present the results from the different fine-tuned models in Table 4. The results are organized under different categories including overall \(\mathcal{WER}\), language-wise \(\mathcal{WER}\) and various demographic-wise \(\mathcal{WER}\). Further we also present the fairness score (\(\mathcal{FS}\)) for each demographic group. The key observations are as follows.
We observe that SamaVaani results in a reduction of 50% in overall \(\mathcal{WER}\) when compared to the Base pre-trained model. The overall \(\mathcal{WER}\) of SamaVaani is also substantially better than standard fine-tuning setups FT\(^\mathrm{\small Std.}\) and FT\(^\mathrm{\small PS}\).
Across all the three languages SamaVaani is remarkably better than the Base model as well as FT\(^\mathrm{\small Std.}\) and FT\(^\mathrm{\small PS}\). Among the three languages, even SamaVaani struggles the most with Kannada, like all the other models.
For all the demographic groups SamaVaani is not only better in terms of the group-wise \(\mathcal{WER}\) but also in terms of \(\mathcal{FS}\) when compared to Base, FT\(^\mathrm{\small Std.}\) and FT\(^\mathrm{\small PS}\).
Ablation study: A natural question in the design of SamaVaani regards the necessity of both the CL and CTC heads. In order to check whether any one of them is as good as the combination, we present in columns 4 and 5 of Table 4 the results from two additional fine-tuning setups – (i) FT\(^\mathrm{\small CL}\): an ablation of SamaVaani where only the contrastive loss is used and (ii) FT\(^\mathrm{\small CTC}\): an ablation of SamaVaani where only the CTC loss is used. We observe that though these models outperform the standard fine-tuning setups FT\(^\mathrm{\small Std.}\) and FT\(^\mathrm{\small PS}\) in terms of both \(\mathcal{WER}\) and \(\mathcal{FS}\), they are not as good as SamaVaani. This quantitatively justifies the benefit of combining the two loss terms. In the next section, we present some qualitative insights into the advantages of each of these components.
| Type | Transcript |
|---|---|
| Ground truth | friends, relatives. Yeah, if he values \(\dots\) if he values that conflicted person a very high level in in his own mind \(\dots\) it is uh, good decision that to worry \(\dots\) for him \(\dots\) that he didn’t come. If he is a normal person who had a conflict, he won’t mind that. He won’t mind that. |
| Base | friends, celebrities. Yeah, if if if if if \(\dots\) (repeated 210 times more) |
| friends, celebrities. Yeah, if if if if if \(\dots\) (repeated 212 times more) | |
| friends, celebrities. Yeah, if if if if if \(\dots\) (repeated 212 times more) | |
| friends, celebrities. Yeah, if if if if you value if if you value that conflicted person a very high level in in his own mind, it is a good decision that to worry for him that he didn’t come. If he is a normal person who had a conflict, he won’t mind that. He won’t mind that. | |
| friends cigarettes. Yeah, if you value if if you value that conflicted person a very high level in in his own mind, it is a good decision that to worry for him that he didn’t come. If he is a normal person who had a conflict, he won’t mind that. He won’t mind that. |
In this section, we discuss representative qualitative advantages of the CL and CTC loss terms (for Gemma3n4) and finally discuss some error
cases. We posit that while the CTC head provides essential character alignment from speech to text, the contrastive learning objective acts as a phonetic regularizer.
Role of contrastive learning: Recall that we use PitchShift to obtain a pitch-shifted variant of the anchor audio. With this, the objective of contrastive learning is to recognize the semantic equivalence between the
original audio and its pitch augmented variant to ignore acoustic noise and focus on phonetic content. By restricting the augmentation to a single anchor-positive pair (\(z_i, z_i^+\)) within a pool of \(N-1\) negative samples (\(z_k\)), the framework creates an asymmetric learning signal that is highly effective for phonetically dense languages. We set the temperature (\(\tau\)) at 0.05 to sharpen the distribution, forcing the model to be very certain about the representations of samples in the latent space. \(\mathcal{FS}\) improve consistently on all
demographic attributes by 13-41% compared to the base model and 15-22% compared to the standard fine-tuned model.
Role of CTC head: Standard LLMs use auto regressive decoding, which are prone to hallucinations with same words repeating sequentially, for example, if if if if... (get caught in a
word repeating indefinitely). The CTC heads enforces monotonic alignment. While the generative head ensures the sentence makes sense, the CTC head acts as a “sanity check” to ensure every word/token corresponds to an actual speech in the file. This balance
significantly reduces instances of “word-skipping” or adding “filler” that wasn’t in the original audio. In the raw audio signal, a single phoneme (like ‘s’ in “speech”) lasts for many frames. Without a special mechanism, a model might predict the letter
‘s’ many times in a row. CTC solves this using a unique decoding rule involving blank tokens (\(\phi\)). The CTC decoding algorithm follows two simple but powerful rules to convert a long sequence of frame-by-frame
predictions into a clean word as follows – (i) collapse identical consecutive tokens: if the model predicts ‘aaaa-bbbb-cccc’ due to slow speech, CTC collapses them into ‘abc’; (ii) blank as a separator: to actually output two of the same
letter (like the ‘ll’ in ‘hello’), the model must predict a blank token between them (e.g., h-e-l-\(\phi\) -l-o).
Error analysis: Table 5 shows qualitative examples as to how auto-regressive models can get stuck in a loop where they start repeating the same word sequentially. Fine-tuning with only contrastive learning
also fails to get out of the repeating loop. On the other hand, incorporating a CTC head on top LoRA is able to break out of this loop and generates better text. Furthermore, SamaVaani generates the best transcripts compared
to ground truth. Not only it can get out of the indefinite loop, it also removes the same repeating words with a few inconsistencies. Lastly, the error in the transcripts generated by SamaVaani is essentially divided into the
following three categories.
Inverse text normalization: The generation should appear in text as spoken word-by-word. For instance, “So, no matter what time I sleep, 8:00 o’clock is when I have to wake up.” Here, the time should appear in words ‘eight’ as it is spoken and not as it is represented.
Named entity errors: There are several instances where the model fails to correct text for named entities, especially for organization and drug names. For example, the organization name ‘NIMHANS’ in ground truth is being substituted by terms like ‘neeman’s’ or ‘2 months’. Similarily, the drug name ‘benzodiazipines’ is being transcribed as ‘benzodiazepam pains’, suggesting a lack of acoustic robustness, where the model struggles to align the correct token sequences.
Deletion errors: There are a few missing phonemes while transcribing speech to text. Forcing monotonic alignment with the CTC head prevents the model from skipping any word/token. It forces the model to attempt a phonetic transcription of every sound.
In this study, we highlighted the disparities in ASR models, particularly in a clinical psychiatric interview setting between a patient and a doctor. We comprehensively audited eight state-of-the-art models across three linguistically diverse languages and reported multiple disparities. Next, we propose SamaVaani that leads to simultaneous improvement of the overall \(\mathcal{WER}\) as well as the demographic fairness. The key uniqueness lies in combining the two loss functions based on contrastive learning and CTC that serve complementary roles.
While our approach is better than the other fine-tuning setups, there is still a lot of scope for improvement, especially for languages like Kannada. The values suggest that it is important to build customised ASR models for a clinical psychiatric interview setting, pre-trained from scratch.
While this study poses a solid foundation towards multilingual ASR in the psychiatric interview setting, there are a couple of limitations. First, the experimental setup depends on LoRA fine-tuning with rank of 8 (low). This setup was chosen due to our hardware constraints. With a high-end infrastructure, a full supervised fine-tuning may further improve the transcription accuracy and fairness. And second, our evaluation is limited only to Indian English, Hindi and Kannada psychiatric interviews. India contains substantial linguistic diversity, and ASR behaviour may differ considerably across other Indic languages, dialects, and code-mixed settings.
This study was approved by the Institute Ethics Committee. The speech data were collected from a tertiary teaching hospital with a specialised addiction treatment centre offering 24-hour emergency services dedicated to the treatment of psychiatric and neurological conditions, along with inpatient and outpatient services. Written informed consent was obtained from patients to audio-record psychiatric interviews. Although deidentified, the data cannot be made public as it contains highly sensitive personal health information, which can compromise patient privacy and confidentiality.
We implement LoRA by injecting trainable rank 8 decomposition matrices into the query (\({q}_{proj}\)), key (\({k}_{proj}\)), value (\({v}_{proj}\)), output (\({o}_{proj}\)) and MLP projection layers (\({gate}_{proj}\), \({up}_{proj}\), and \({down}_{proj}\)) on the base ASR models. The experimental configuration and the different hyperparameters are noted in Table [tbl:tab:hyperp].
| Configuration | Value |
|---|---|
| LoRA configuration | |
| Rank (\(r\)) | 8 |
| Scaling factor (\(\alpha_{\text{LoRA}}\)) | 16 |
| Dropout | 0.09 |
| Target modules | |
| \(o_{\text{proj}}, gate_{\text{proj}}\), | |
| \(up_{\text{proj}}, down_{\text{proj}}\) | |
| Base model quantization | 4-bit |
| Optimization | |
| Optimizer | AdamW (8-bit) |
| Base learning rate | \(5 \times 10^{-5}\) |
| Learning rate schedule | Cosine |
| Learning rate warmup period | 10% (for 3 epochs) |
| Max sequence length | 1024 tokens |
| Temperature (\(\tau\)) | 0.05 |
| Training infrastructure | |
| Per-device batch size | 1 |
| Gradient accumulation steps | 8 |
| Effective global batch size | 32 |
| GPUs | |
| (48 GB each) | |
| Validation & early stopping | |
| Validation metric | Median WER |
| Early stopping patience | 3 evaluation steps |
The loss coefficients (\(\alpha, \beta\), \(\gamma\)) for each of the three components in the total loss function for both the fine-tuned models, Gemma3n and OmniLingual are optimized through Optuna5 over 20 trials. The exact coefficient values for Gemma3n and OmniLingual are (0.4135, 0.2186, 0.3679) and (0.4385, 0.2480, 0.3135) respectively.
In this section, we analyze the acoustic features of the dataset used in this stdy. We evaluate several key features, which indicates how good the voice quality of the data is. Performance evaluations indicate that males, patients, and Kannada speakers are structurally disadvantaged subgroups which are disproportionately affected by higher Word Error Rates (\(\mathcal{WER}\)). Importantly, the performance gap stems from acoustic differences, not data size. Although males represent 65% of the dataset, their speech leads to worse WERs because of a naturally lower fundamental frequency (\(r = 0.84\)) and a lower voice quality, indicated by lower Harmonics-to-Noise Ratio (HNR) and higher amplitude instability (shimmer). Likewise, the speech of patients is a reflection of clinical realities such as psychomotor symptoms or emotional distress, which manifest acoustically as lower pitch and degraded voice quality. We depict the differences in pitch and voice quality of the speech data among gender and speaker role respectively from Figures 2 to 5. In addition, the illustrate the speech intelligibility analysis in Figure 6.
In this section, we detail the exact transcription instructions given to annotators with examples for each language.
Transcription instructions to annotators:
Your task is to carefully listen to the provided audio file and create a .txt or .srt file.
About the recordings:
\(\bullet\) These are audio-recorded clinical interviews between doctors and patients.
\(\bullet\) Accuracy of transcription is very important for clinical and research purposes.
\(\bullet\) Please follow the following guidance material to ensure accurate transcriptions.
Carefully read the instructions below on how to do the transcriptions and their examples for each of the three language, namely, English, Hindi, and Kannada (in order).
Speaker diarisation:
Clearly distinguish between the speakers. Use the following labels at the beginning of each new turn of speech.
\(\bullet\) For English \(\to\) [Doctor:] [Patient:]
\(\bullet\) For Hindi \(\to\) [डॉक्टर:] [मरीज़:]
\(\bullet\) For Kannada \(\to\) [ವೈದ್ಯ:] [ರೋಗಿ:]
Use of punctuation:
Allowed punctuations are full-stop, question mark, comma, ellipsis, em-dash and exclamation marks.
| Punctuation :================== Full Stop (.) | Guidance :================================================= Use for the end of a sentence. |
| Comma (,) | Use for short pauses and separating multiple items or phrases. |
| Ellipsis (...) | Use for long pauses. |
| Em-dash (–) | Use for interruptions or cut-offs. English example, Doctor: So what brings you h– Hindi example, चिकित्सक: तुम यहाँ क्यों – Kannada example, ಡಾಕ್ಟರ್: ನೀವು ಯಾಕೆ ಇಲ್ಲಿಗೆ — |
| Question Mark (?) | Use for direct questions. |
| Exclamation (!) | Use a single mark to indicate strong emotions. English example, It just feels so bad I can’t even tell you! Hindi example, मैं आपको बता नहीं सकता कि मैं कितना परेशान हूँ! Kannada example, ನಂಗೆ ಎಷ್ಟು ಬೇಜಾರ್ ಆಗುತ್ತೆ ಅಂದ್ರೆ ಹೇಳೋಕ್ಕೆ ಆಗಲ್ಲ! |
Strict verbatim transcription:
Preserve all dysfluencies like pauses, filler words, stammers etc. Do not correct if there are grammatical errors in the speech itself. See table below.
| Guidance :================== Capture filler words and phrases | English examples :================================= Um, Yeah, Hmm, Mmmm, etc | Hindi examples :================================= अच्छा, हाँ, हम्म, हा, आ, etc | Kannada examples :================================= ಉಮ್, ಅಹ್, ಹಮ್ಮ್, ಹಾ, ಆ, etc |
| Include all incomplete phrases as they are said | I... I yesterday... no, the day before, I had gone there. | मैं...मैं कल...नहीं, परसों वहाँ गया था। | ನಾನು... ನಾನು ನಿನ್ನೆ... ಇಲ್ಲ, ಮೊನ್ನೆ ಅಲ್ಲಿಗೆ ಹೋಗಿದ್ದೆ |
| Include stutters and stammers | I-I-I get scared. | मैं-मैं-मैं डर जाता हूं| | ನ-ನ-ನನಗೆ ಭಯ ಆಗುತ್ತೆ |
| Use ellipsis (three dots) for long pauses or incomplete sentences | I don’t know… I feel very scared. | मुझे नहीं पता... मुझे बहुत डर लग रहा है| | ನನಗೆ ಗೊತ್ತಿಲ್ಲ... ತುಂಬಾ ಭಯ ಆಗುತ್ತೆ |
| Transcribe as spoken, do not change informal to formal style (applies for languages like Kannada where spoken/colloquial and written styles are very different). Do not correct grammar or polish the output in any manner. | I am afraid \(\to\) I am feeling scared | मुझे डर है \(\to\) मुझे डर लग रहा है | ನಂಗೆ ಭಯ ಆಗ್ತಿದೆ \(\to\) ನನಗೆ ಭಯ ಆಗುತ್ತಿದೆ |
| There can be frequent language mixing in these audios. You can expect English, Hindi, Telugu, Tamil and Malayalam. Write these words in their native script as of speech. | main samajh gaya, gottaytu | मैं समझ गया, गोथायतू | ಠೀಕ್ ಹೈ, ನಂಗೆ ಅರ್ಥ ಆಯಿತು [theek hai, nange artha aytu] |
| If there are numbers, type them out in words and not as numerals. | Incorrect: I took 10 tablets that day. Correct: I took ten tablets that day. |
Incorrect: मैंने 10 गोलियाँ लीं / मैंने १० गोलियाँ लीं। Correct: मैंने दस गोलियाँ ले लीं |
Incorrect: ನಾನು ಅವತ್ತು 10 ಮಾತ್ರೆಗಳು ತಗೊಂಡೆ / ನಾನು ಅವತ್ತು ೧೦ ಮಾತ್ರೆಗಳು ತಗೊಂಡೆ Correct: ನಾನು ಅವತ್ತು ಹತ್ತು ಮಾತ್ರೆಗಳು ತಗೊಂಡೆ |
Non-speech occurrences:
Five relevant non-speech sounds must be noted. Use square brackets [\(\dots\)] for these annotations. For example, when the discussion is about depression, sounds of the speaker crying becomes important. See the indicators
below. Do not use any other indicators.
| Type | English examples | Hindi examples | Kannada examples |
|---|---|---|---|
| Three Special tokens are allowed for emotional expressions. They must be in square brackets and are the only ones allowed. | [laughs], [cries], and [shouts] | [हंसने की आवाज़], [रोने की आवाज़] and [चिल्लाना] | [ನಗುವಿನ ಸದ್ದು], [ಅಳುವಿನ ಸದ್ದು] and [ಕೂಗುತ್ತಾ] |
| One special token is allowed for unclear speech | [unclear] | [अस्पष्ट] | [ಅಸ್ಪಷ್ಟ] |
| One special token is allowed for other noises [noise]. Use this for all non-human background sounds like phone ringing or buzzing, ambulance siren, car honks, furniture scraping, other mechanical sounds. | [00:03:16] Patient: I had come then, but [noise] could not meet him. | [00:03:16] मरीज़: मैं तब आया था, लेकिन [noise] मिला नहीं | [00:03:16] ರೋಗಿ: ಆವಾಗ್ಲೇ ಬಂದಿದ್ದೆ, ಆದ್ರೆ [noise] ಸಿಗ್ಲಿಲ್ಲ. |
Timestamping and General Formatting:
Use uniform font size and line spacing. The document must be in .txt or .srt format. The transcript must be timestamped in [HH:MM:SS] format (hours: minutes: seconds). See the example below:
| English examples :===================== Doctor: Please come in, have a seat, and tell me, how are you? [00:00:18] Patient: Hello, Doctor. I haven’t been feeling well for the past one or two weeks and I feel sad all the time. [00:01:05] Patient: Yes, Doctor. I used to enjoy talking to my friends and gardening, but now I feel like just sitting alone all the time. |
Hindi examples :===================== [00:00:05] डॉक्टर: आइए, बैठिए और बताइए कि आप कैसे हैं? [00:00:18] मरीज़: नमस्कार डॉक्टर साहब. पिछले एक-दो सप्ताह से मेरी तबीयत ठीक नहीं है और हमेशा उदास रहता हूं. [00:01:05] मरीज़: हाँ डॉक्टर, पहले मुझे अपने दोस्तों से बात करना और बागवानी करना अच्छा लगता था, लेकिन अब हर समय अकेले बैठे रहने का मन करता है. |
Kannada examples :====================== [00:00:05] ವೈದ್ಯರು: ಬನ್ನಿ, ಕೂತ್ಕೊಳ್ಳಿ. ಹೇಳಿ, ಈವಾಗ ಹೇಗಿದ್ದೀರ? [00:00:18] ರೋಗಿ: ನಮಸ್ಕಾರ ಡಾಕ್ಟರ್. ಒಂದು ಎರಡು ವಾರದಿಂದ ಏನೋ ಸರಿ ಇಲ್ಲ. ಯಾವಾಗಲೂ ಬೇಜಾರು... ಒಂಥರಾ ಅನ್ಸುತ್ತೆ. [00:01:05] ರೋಗಿ: ಹೌದು ಡಾಕ್ಟರ್. ಫ್ರೆಂಡ್ಸ್ ಜೊತೆ ಮಾತಾಡೋದು, ಗಿಡ ನೋಡಿಕೊಳ್ಳೋದು ಅಂದ್ರೆ ಇಷ್ಟ. ಆದ್ರೆ ಈಗ ಏನೂ ಮಾಡೋಕೆ ಇಷ್ಟ ಆಗಲ್ಲ. ಸುಮ್ಮನೆ ಕೂತಿರ್ತೀನಿ ಅಷ್ಟೇ. |
A speaker’s entire turn must be kept in one line. Do not press ENTER (start a new line) in the middle of their speech even if it contains multiple sentences. Use line breaks between separate speakers. See example below.
P5cm P5cm P5cm English examples & Hindi examples & Kannada examples
Correct
Doctor: When was the last time you slept well?
Patient: I can’t remember. It’s been many months.
Incorrect
Doctor: When was the last time you slept well?
Patient: I can’t remember.
It’s been many months. & Correct
चिकित्सक: पिछली बार आपको अच्छी नींद कब आई थी?
मरीज़: मुझे याद नहीं. लगभग एक महीना हो गया.
Incorrect
चिकित्सक: पिछली बार आपको कब अच्छी नींद आई थी?
मरीज़: मुझे याद नहीं.
लगभग एक महीना हो गया. & Correct
ವೈದ್ಯರು: ನೀವು ಕೊನೆಯದಾಗಿ ಯಾವಾಗ ಚೆನ್ನಾಗಿ ನಿದ್ದೆ ಮಾಡಿದ್ದೀರಿ?
ರೋಗಿ: ನನಗೆ ನೆನಪಿಲ್ಲ. ಸುಮಾರು ಒಂದು ತಿಂಗಳಾಯ್ತು.
Incorrect
ವೈದ್ಯರು: ನೀವು ಕೊನೆಯದಾಗಿ ಯಾವಾಗ ಚೆನ್ನಾಗಿ ನಿದ್ದೆ ಮಾಡಿದ್ದೀರಿ?
ರೋಗಿ: ನನಗೆ ನೆನಪಿಲ್ಲ.
ಸುಮಾರು ಒಂದು ತಿಂಗಳಾಯ್ತು.
https://docs.pytorch.org/audio/main/generated/torchaudio.transforms.PitchShift.html↩︎
Similar observations hold for OmniLingual.↩︎