1 Introduction↩︎

Spoken Question Answering (SQA) extends question answering to spoken inputs, requiring models to understand both the linguistic content and the acoustic characteristics of speech [1]. Therefore, SQA must cope with disfluencies, informal phrasing, speaker and channel variability, and background noise. Early systems mainly used a cascade approach, where an automatic speech recognition (ASR) module transcribed the audio before a text QA model produced the answer. Although effective in clean conditions, this design suffers from error propagation when the ASR output is noisy, and it discards prosodic cues such as prominence and phrasing that can help localize answer spans [2].

Recent speech-LLM architectures address these issues by coupling powerful speech encoders with instruction-tuned language models via a lightweight connector or projector: the projector maps acoustic embeddings into the token/embedding space of a frozen LLM so that only a small number of parameters need updating. This parameter-efficient connector idea (small adapters + LoRA) has been shown to yield strong performance while keeping compute and data requirements modest (e.g., Speech, Language, Audio and Music-ASR (SLAM-ASR) and related LLM-ASR studies);  [3]–[6] illustrate the framework. Follow-up work has explored robustness, scaling, and multilingual transfer of these projector-based systems; see evaluations and references in recent SLAM/SLM/SLAM-ASR analyses [7]. The projector + LLM approach is attractive for low-resource scenarios because it reuses large pretrained components and pushes most task adaptation into a compact layer that can be trained with limited hardware and data; several articles explicitly study data volume, multilingual transfer, and adapter-based solutions for low-resource speech tasks [8].

This raises a practical question: Can we leverage TTS to build useful SQA training data without recording new speakers, and if so, how sensitive is the resulting system to TTS quality, voice diversity, and corpus size? Previous work that automatically generates SQA data or converts text QA corpora to speech shows that this is a feasible and useful strategy: frameworks for automatic SQA generation [9] and experiments with synthetic speech have been proposed and evaluated, with explicit analyzes of the choice of TTS and ASR noise effects. See the automatic generation framework and follow-ups by Menevşe et al. [10], [11] and empirical dataset studies such as HeySQuAD and LibriSQA [12], [13].

Early SQA systems mainly relied on ASR followed by text-based question answering, making performance strongly dependent on transcription quality. More recent work has moved toward direct modeling of spoken inputs. End-to-end SQA and spoken conversational QA have been studied for multi-turn dialogue and speech-based answer generation [1], [14], [15]. Generative systems such as GSQA reduce reliance on explicit ASR transcripts by mapping spoken questions to spoken or textual answers, while Spectron adapts pretrained LLMs to speech input and output using spectrogram-based representations [16], [17]. These studies show that direct SQA is feasible, but they usually require substantial task-specific training and do not focus on low-resource languages. A related line of work connects speech encoders to LLMs through lightweight projectors. LLM-based ASR and SLAM-style systems map acoustic embeddings into the LLM embedding space and can achieve competitive performance while keeping the speech encoder and LLM frozen or lightly tuned [4], [8]. Instruction-following speech LLMs extend this approach to tasks such as spoken summarization, classification, and dialogue, including benchmarks with closed-ended and open-ended speech instructions [18]. Most of this work, however, targets transcription or general speech instruction following rather than low-resource SQA. Synthetic speech has also been widely used for data augmentation in ASR, speech translation, and related low-resource speech tasks. Prior studies show that TTS-generated audio can improve ASR by pairing available text with synthesized speech, and that TTS or voice conversion can help construct multi-speaker datasets when recording real speakers is impractical [19]–[21]. In SQA, augmentation of spoken questions can improve robustness to noise and accent variation [2]. However, most augmentation work is designed for ASR, retrieval, or translation, rather than for constructing spoken question-answer pairs to train projector-based SQA models. TTS-based augmentation also supports reproducible resource creation: even when raw speech cannot be released, the pipeline, quality checks, and design choices can be shared to guide similar low-resource SQA datasets.

Important empirical findings from previous work can be clustered as follows.

Synthetic speech quality and diversity. Synthetic speech can produce useful SQA training data, but model gains depend strongly on the realism and diversity of the TTS voices and on whether the subsequent evaluation uses real human speech [11], [12].

ASR and pipeline robustness. ASR/transcription errors substantially degrade retrieval and QA performance in cascade pipelines; end-to-end or ASR-free approaches (speech retrieval, direct speech→answer models, or projector-based speech-LLMs) are more robust when ASR is poor [2], [22].

Projector-based speech→LLM systems. Small linear projectors and LoRA-style adapters achieve strong transfer when trained on paired speech→text instruction data, but robustness to domain shift and noisy speech still requires either (a) larger/more diverse synthetic corpora, (b) pretraining the projector on high-resource languages, or (c) adapter mixtures / Mixture of Experts (MoE) techniques for multilinguality [4].

Supervised projector training. In our setup, the speech\(\rightarrow\)LLM connector (linear projector) is trained with supervision on paired spoken-question/text-answer data; we do not use data distillation or self-supervised objectives [3], [4].

In this paper, we explore this setup for Luxembourgish SQA. We start from existing text-based QA resources and an English SQA corpus, and construct a set of Luxembourgish spoken questions by combining machine translation with multiple TTS systems. We train a parameter-efficient SLAM-style model that connects a frozen Whisper large v3 turbo encoder to multilingual LLM backends via a linear projector and LoRA adapters. The model is trained to map spoken questions directly to textual answers, so that it operates in a speech instruction following regime rather than pure transcription.

1.1 Positioning of our work↩︎

Compared to end-to-end SQA models such as GSQA or Spectron, our approach keeps both the speech encoder and a multilingual LLM frozen and trains only a compact projector together with LoRA adapters, providing a parameter-efficient pathway from speech to textual answers [16]. Unlike LLM-based ASR frameworks that optimize the projector for transcription, we directly supervise it with QA pairs so that the model learns to map spoken questions in a low-resource language to answers in text form. At the data level, we follow the spirit of TTS-based augmentation for low-resource speech, but we apply TTS to generate spoken questions from large text corpora and pair them with textual answers, rather than synthesizing generic read speech or ASR-style utterances [19].

The main contribution of this paper is therefore methodological. We present a complete workflow for constructing and exploiting TTS-augmented SQA data in a low-resource language, including translation choices, TTS selection, corpus filtering, and evaluation. Luxembourgish serves as a concrete case study, but the design and error analysis are meant to travel to other settings. We emphasize what works, what does not work, and where synthetic data falls short of human recordings, so that future data collection efforts can better balance manual and synthetic speech.

2 Methodology↩︎

2.1 Design considerations↩︎

We distill empirical findings from previous SQA and speech LLM work to guide the Luxembourgish pipeline and model setup (see Figure 1). The main considerations are:

1. TTS quality and naturalness. Previous work shows that synthetic speech can provide useful training data for SQA and related tasks, but gains depend strongly on the realism of generated audio and the presence of artefacts [11], [12]. When TTS voices are more natural and closer to human prosody, models transfer better to real spoken queries. This motivates the use of high-quality TTS systems and quality control of the generated audio.

2. Voice diversity. Analysis of synthetic corpora such as HeySQuAD and LibriSQA indicates that using multiple voices (different speakers, genders, or timbres) reduces overfitting to a single synthetic voice and improves robustness to unseen speakers [12], [23]. We therefore employ several TTS engines and, where possible, multiple voices per engine, so that the synthetic Luxembourgish questions cover a range of speaking styles.

3. Corpus size and transfer. Projector-based systems benefit from large paired speech-text corpora, and previous work on SLAM-style architectures shows that reusing strong multilingual encoders and LLMs can improve sample efficiency for low-resource targets [4], [8]. Rather than pretraining a new encoder, we adopt a frozen Whisper encoder and frozen EuroLLM/Apertus backends and vary the amount of synthetic and human-spoken data used to train the projector and LoRA adapters.

4. Pipeline robustness. Studies on SQA and speech retrieval confirm that cascade pipelines suffer when ASR quality is low, as transcription errors propagate into the QA model [2], [22]. Direct speech-to-response architectures, where a projector feeds a frozen LLM, are more robust under such conditions. This evidence motivated our choice of a SLAM-style speech instruction setup instead of a cascade ASR plus text QA pipeline.

5. Generalizability of tasks. SQA is representative of a broader class of speech-centric applications such as spoken dialogue systems, voice assistants, and audio-based reasoning; consequently, the insights gained from analyzing SQA can generalize to a wide range of other multimodal speech understanding tasks [1] with LMMs.

These considerations shape our TTS choices, the use of multiple voices and engines, the corpus scaling experiments on synthetic versus human speech, and the decision to adopt a projector-based speech-to-answer model.

2.2 TTS models↩︎

We generate synthetic data using these models:

Overall, we evaluate five TTS-based synthetic data generation strategies: MMS-TTS, two Qwen3-TTS variants (Qwen-CV and Qwen-VD), and two OmniVoice variants (Omni-C and Omni-VD) as listed in Table ¿tbl:tab:tts95quality?.

2.3 Data pipeline and corpora↩︎

We derive all SQA data used in this work from existing text-based QA resources. For English, we rely on SQuAD v1.1 and its spoken extension HeySQuAD [12]. SQuAD consists of questions created from Wikipedia articles, where each answer is a text span extracted directly from the corresponding passage; it is considered a standardized benchmark for extractive question answering. HeySQuAD extends the SQuAD benchmark to SQA by introducing speech as the input modality; as a result, it is useful for developing and benchmarking voice-based question-answering systems operating under realistic acoustic conditions. For Luxembourgish, we create synthetic speech by translating English questions into Luxembourgish using TranslateGemma 12B Instruct model, then generating spoken questions with several neural TTS engines. Table 1 summarizes the resulting corpora and their roles.

2.75pt

SQuAD-LB-TTS is created by translating English questions into Luxembourgish and synthesizing spoken Luxembourgish questions with the TTS engines listed above, while keeping the textual answers unchanged. For evaluation, we additionally use LLAMA-LB-Test, a 300-question set ¹ whose Luxembourgish questions were human-translated and recorded by two native speakers, yielding two real-speech evaluation conditions, denoted Speaker 1 and Speaker 2 in the results. The proprietary Luxembourgish QA set contains internal LB and EN question-answer pairs and is used as an in-domain development resource ².

2.4 TTS system quality assessment↩︎

To select and validate the TTS systems used for Luxembourgish question synthesis, we conducted an automatic audio quality assessment using two no-reference toolkits: NISQA-TTS [29], which estimates naturalness MOS for synthesized speech, and DNSMOS [30], which reports overall quality (OVRL), speech signal quality (SIG) and background noise (BAK) on a 1–5 scale. We evaluated five TTS systems on 48,848 synthesized Luxembourgish utterances from the SQuAD Gemma training set. Results are reported in Table ¿tbl:tab:tts95quality?. Omni-C achieves the highest overall DNSMOS score (3.38), while Omni-VD obtains the best P.808 MOS (3.94) and is the only system for which NISQA-TTS naturalness was computed (3.94). Omni-VD obtains the highest P.808 MOS (3.94), followed closely by MMS-TTS (3.93), while Qwen-VD reaches 3.79.

13pt

All systems achieve strong background noise scores (BAK \(\geq\) 4.04), indicating clean synthesis without significant background artefacts. Qwen-CustomVoice (CV) scores lowest overall (DNSMOS-OVRL: 3.28), which is consistent with its lower downstream QA performance observed in Section 3.

2.5 Model architecture and training↩︎

The model architecture follows the SLAM-ASR recipe [4] and, instead of speech recognition, it is trained to answer the spoken question. We adopt Whisper-large-v3-turbo as the audio encoder and two multilingual backend LLMs, EuroLLM 9B Instruct [31] and Apertus 8B Instruct [32]. The speech encoder is kept frozen. Its output is a sequence of frame-level embeddings, which we downsample by a factor of \(k = 5\) to reduce the length mismatch between speech and text representations. A linear projector maps the downsampled speech embeddings to the token embedding space of the chosen EuroLLM backend. The LLM is equipped with a LoRA adapter so that only the projector and LoRA parameters are updated during training.

To study the effect of synthetic training data, we trained a family of SQA models that differ in the speech data and initialization used for projector training. We use E to denote the EuroLLM-9B-Instruct backbone and A to denote the Apertus-8B-Instruct backbone. Single-source configurations are named by the synthetic speech corpus used for projector training: MMS-TTS, Omni-C, Omni-VD, Qwen-CV, or Qwen-VD. The 4TTS mix combines the OmniVoice and Qwen3-TTS variants into a larger multi-speaker corpus of approximately 230k synthesized Luxembourgish questions, compared with about 48k questions per single-source corpus. E scratch denotes EuroLLM training without synthetic-speech initialization and ASR init. denotes initialization from an ASR-adapted checkpoint.

For each training example, the projected speech embeddings are concatenated with a short text prompt (for example, “Answer the spoken question”) and fed to the LLM, which generates the answer tokens autoregressively. The model is optimized using cross-entropy loss over the answer tokens. The models were trained using the Adam optimizer with an initial learning rate of \(10^{-4}\), a linear warmup of 1,000 steps, and a subsequent scheduled learning rate decay. Training was carried out for 5 epochs with a batch size of 16. LoRA is applied with rank \(r = 16\) and scaling factor \(\alpha = 16\). At inference time, only the spoken question and prompt are provided, and the model returns a textual answer.

2.6 Experimental conditions and metrics↩︎

We structure our experiments around three research questions:

Because the model generates answers autoregressively in a morphologically rich, low-resource language where predictions paraphrase the gold answer or code-switch between Luxembourgish, German, French, and English, exact-match and Token-overlap metrics are too brittle to be informative. We evaluate with two semantics-aware metrics, reported as percentages. Semantic Similarity (SemSim) embeds the gold answer and prediction with a multilingual sentence encoder (paraphrase-multilingual-MiniLM-L12-v2) and computes the maximum cosine similarity between the gold answer and any prediction span; this rewards predictions containing the correct answer even within longer or repetitive output. LLM Judge Mean uses two instruction-tuned judges, GPT-4o and Gemini-2.5-Flash, run at temperature 0. Each judge assigns a score from \(\{0, 0.5, 1\}\) to every (question, gold answer, prediction) triple under a fixed partial-credit rubric that accepts translations, aliases, and spelling or ASR variants of the gold answer. We average the two judge scores per item and report the mean of these item-level averages over the test set. The two metrics are independent by construction, one embedding-based and one LLM-based, so their agreement on model ranking serves as a robustness check.

3 Experimental results↩︎

We report results on the LLAMA1-Test-Set using both text-only baselines and spoken Luxembourgish input. Figure 2 summarizes the main comparison in terms of Semantic Similarity and LLM Judge Mean. These results address three questions 2.3: the usefulness of synthetic data and ASR-adapted initialization for low-resource SQA, the effect of TTS source and voice diversity across evaluation speakers, and the extent to which semantics-aware metrics and automatic TTS quality estimates explain downstream SQA performance.

3.1 Overall QA performance↩︎

The text-only baselines provide an upper bound once the acoustic channel is removed. EuroLLM-9B (base) reaches 67.3 SemSim and 79.0 LLM Judge Mean on English text input, while the non-LB-adapted text baselines score lower in SemSim: 51.6 for EuroLLM-9B and 51.3 for Apertus-8B. This shows that strong text-side generation alone is not sufficient for SQA.

Luxembourgish audio evaluation is substantially harder without speaker-specific synthetic training. Generic lB-adapted models reach 45.7 SemSim and 62.5 LLM Judge Mean for Apertus-8B, and 38.8 SemSim and 32.8 LLM Judge Mean for EuroLLM-9B, showing that generic Luxembourgish adaptation does not close the speech-text gap.

3.2 Effect of TTS choice and speaker-specific training↩︎

The strongest results are obtained by models trained on speaker-specific synthetic Luxembourgish speech. For Speaker 1, the best system is EuroLLM-9B (4tts), with 72.6 SemSim and 60.1 LLM Judge Mean. For Speaker 2, the best system is EuroLLM-9B (4tts-from-asr), with 74.8 SemSim and 61.8 LLM Judge Mean. These scores are not only far above the generic _lb models, but also close to or above the semantic similarity of the text baseline. This indicates that carefully constructed synthetic training data can compensate for the lack of real Luxembourgish QA recordings. The comparison between A + 4TTS mix and A + 4TTS, ASR init. Further shows that ASR initialization helps: using an ASR-adapted checkpoint trained with both original LuxASR and synthesized LuxASR-TTS data raises the result on both speakers.

Across both speakers, EuroLLM is stronger than Apertus in higher-performing configurations. Qwen-VD, Omni-VD, and 4tts yield the most competitive systems, while Qwen-CV, MMS-TTS, and Omni-C tend to lag. The gap is especially clear in the LLM Judge Mean, where weaker TTS configurations often fall to the low 20s or 30s despite moderate semantic similarity. This suggests that some systems preserve enough lexical content for partial semantic matching but still produce responses judged less appropriate or less well-formed.

The Speaker 1 and Speaker 2 rankings are also broadly stable. High-performing configurations remain near the top for both speakers, while weaker configurations remain weak across both test conditions. This stability strengthens the conclusion that the main driver is the training data configuration rather than speaker-specific randomness in the test set.

3.3 Relation between TTS quality and downstream QA↩︎

Table ¿tbl:tab:tts95quality? shows that the synthesized speech is clean, with uniformly strong DNSMOS background scores and relatively narrow ranges in no-reference quality estimates. However, downstream QA performance does not follow these perceptual metrics monotonically. For example, Omni-C achieves the highest DNSMOS overall score, yet its downstream QA results are below those of the strongest 4tts and voice-design configurations. Similarly, MMS-TTS receives strong perceptual scores but remains among the weaker setups in QA.

This mismatch indicates that good TTS quality is necessary but not sufficient for effective SQA training. What matters for downstream performance is not only naturalness but also whether the synthetic speech preserves the lexical, phonetic, and prosodic cues that help the speech encoder and projector recover the question content reliably. In our setting, speaker-aware and multi-source synthetic datasets appear more useful than choosing one TTS system based only on MOS-like scores.

4 Discussion↩︎

Our results support three conclusions. First, TTS-based augmentation is a viable strategy for low-resource SQA. This is consistent with previous SQA work showing that SQA data can be constructed from text-based QA resources using TTS and ASR components, and with HeySQuAD, which extends SQuAD with human-spoken and machine-generated questions [11], [12]. It also aligns with the work on data scarcity in SQA, where automatically generated question-answer pairs are used to improve QA performance under limited annotated data conditions [10]. In our Luxembourgish setting, generic audio models remain weak, but speaker-specific synthetic training substantially improves performance on LLAMA-LB-Test. This is an important practical finding because it shows that a usable SQA pipeline can be built without collecting a large human-recorded Luxembourgish QA corpus.

Second, the choice of synthetic training data matters more than the perceptual quality of TTS. Previous work on TTS model selection for synthetic data generation has shown that computable TTS quality metrics such as NISQA MOS and intelligibility do not necessarily predict downstream ASR performance [33]. Our results show a similar pattern for SQA: the best downstream models are obtained from 4tts and voice-design-based setups, rather than from systems with the strongest DNSMOS or MOS estimates. This suggests that downstream SQA should evaluate TTS not only as speech synthesis but also as a task-specific data generation mechanism.

Third, the results highlight a persistent gap between text and speech. Previous work has shown that SQA and spoken retrieval are affected by ASR errors, acoustic variability, and speech-text mismatch [2], [22]. In our experiments, even when semantic similarity is high for the best speaker-specific systems, LLM Judge Mean remains below the text-only baseline. This implies that the spoken pipeline still loses information relative to text input, likely through a combination of translation noise, pronunciation mismatches, synthetic-speaker bias, and imperfect speech-to-LLM alignment. The current approach is therefore effective, but it does not remove the need for better Luxembourgish speech coverage, more diverse speakers, and stronger evaluation on naturally recorded audio.

Taken together, these findings position LuxSQA as a data-centric recipe rather than a model comparison alone. The main contribution is that it identifies which types of synthetic speech are most useful for low-resource SQA and shows that carefully designed TTS augmentation can produce meaningful gains when real SQA data are scarce.

4.1 Limitations and Future Work↩︎

Although experiments show that synthetic data can support SQA in a low-resource language, several aspects require controlled evaluation. First, we compare different training configurations, but do not isolate the effect of data scale. Future work should measure how SQA performance changes with synthetic dataset size, TTS engine, and language coverage.

Second, the current datasets rely on a limited number of synthetic speakers. Prior SQA datasets such as HeySQuAD show that human-spoken and machine-generated questions differ in noise profile and evaluation behavior [12]. A systematic speaker-diversity study should therefore vary the number of voices, speaker characteristics, and speaking styles to test robustness to unseen speakers.

Third, our models map spoken questions directly to textual answers without explicit access to supporting context. This design isolates speech understanding, but many real-world QA systems require retrieval or reasoning over external evidence. Future work should examine context-aware SQA, where the model jointly processes the spoken question and associated documents or facts, following the broader motivation of open-domain SQA and spoken passage retrieval [2], [22].

In addition, LLM Judge Mean inherits the known self-preference, verbosity, and reproducibility limitations of LLM-as-judge evaluation [34], [35]; we reduce these with two independent judges at temperature 0 and a fixed rubric, and corroborate results with SemSim, but human evaluation remains future work.

Finally, Luxembourgish is one low-resource case study. Additional experiments on languages with different phonological and morphological properties are needed to test whether the proposed pipeline generalizes. Evaluation of naturally recorded speech from multiple speakers and acoustic conditions would further clarify robustness and transferability.

5 Conclusions↩︎

This paper presented LuxSQA, a workflow for constructing Luxembourgish SQA data from text-based QA resources using machine translation and multilingual TTS. Using a SLAM-style architecture with a frozen Whisper encoder, a learned projector, and LoRA-adapted multilingual LLMs, we showed that synthetic Luxembourgish speech can improve spoken QA performance when real QA recordings are scarce. The strongest results came from multi-source and speaker-aware TTS training configurations, while no-reference TTS quality scores alone did not reliably predict downstream QA performance. These findings suggest that TTS augmentation is useful for low-resource SQA, but that synthetic data should be evaluated as task-specific training data, not only as natural-sounding speech.

Generative AI Disclosure↩︎

Generative AI (Grammarly) tools were used to support English language editing and stylistic rephrasing. The scientific content, analyses, and conclusions remain the responsibility of the authors.

References↩︎

---

1. https://github.com/google-research-datasets/LLAMA1-Test-Set↩︎

2. https://anonymous.4open.science/r/LuxSQA-samples-18A4/↩︎