1 Introduction↩︎

Speech large language models (SpeechLLMs) extend the instruction-following (IF) capabilities of large language models to spoken inputs, where a speech encoder is connected to an LLM decoder through a modality adapter [1], [2]. This architecture supports multiple speech-to-text tasks with natural-language instructions, including automatic speech recognition (ASR), speech translation (ST), spoken question answering (SQA), and speech summarization (SSUM). Compared with task-specific systems, instruction-following SpeechLLMs [3], [4] provide a flexible interface for multilingual and multi-task speech processing, while allowing spoken inputs to benefit from the generation abilities of pretrained LLMs.

The International Conference on Spoken Language Translation (IWSLT) 2026 Instruction Following shared task¹ evaluates speech-based instruction-following systems in short-form and long-form settings. The short track focuses on short-form speech and covers ASR, ST, and SQA across languages. The long track extends this setting to longer audio, requiring systems to perform ASR, ST, SQA, SSUM, and audio chaptering (ACHAP). Together, the two tracks provide a benchmark for assessing instruction-following ability and robustness to long-form speech.

Despite recent progress in SpeechLLMs, long-form speech processing remains challenging. Long-form inputs introduce high computational cost, long acoustic token sequences, discourse-level context modeling, and hallucination risks. In IWSLT 2025, KIT [5] was the only participant in the long track [6]. For SpeechLLMs, longer speech sequences place greater pressure on the LLM context and may lead to unstable generation. It therefore remains unclear how effectively short-form SpeechLLMs can be extended to long-form settings, how much short-form ability is retained, and which segmentation strategies are most effective for long-form inference.

Under the constrained setting, this study examines three hypotheses: (1) long-form extension can improve long-form performance while preserving competitive short-form capability; (2) speech segmentation substantially affects long-form instruction following; and (3) hallucination-aware evaluation is needed for reliable comparison of long-form systems. The main contributions are:

• Practical and effective SpeechLLM solutions for short-form and long-form instruction following under constrained settings.

• Empirical comparisons across three speech segmentation strategies: fixed-window, CRDNN-based, and hybrid segmentation.

• Detailed evaluation analysis of hallucination effects and short-form capability retention after long-form extension.

2 Model Architecture↩︎

In the constrained setting, SpeechLLMs are required to be built upon two pretrained models, SeamlessM4T-v2-large² and Qwen3-4B-Instruct³. As illustrated in Figure 1 (a), our SpeechLLMs comprise a speech encoder, a modality adapter, and an LLM decoder. The model architectures are identical for both the short track and long track, where the finetuned short-form SpeechLLM can be extended to long-form directly.

2.1 Speech Encoder↩︎

The speech encoder is adopted from SeamlessM4T-v2-large and serves as the acoustic front end of the model, transforming raw speech into intermediate speech representations for downstream processing [7]. The feature extractor operates on 16 kHz audio and converts raw waveforms into 80-dimensional log-Mel filterbank features. With a 10 ms hop size, these features are extracted at 100 Hz. A stride of 2 is then applied in the frontend, such that two consecutive frames are combined into a 160-dimensional representation before feature projection, reducing the temporal resolution to 50 Hz. These acoustic features are provided to the model together with an attention mask indicating valid and padded positions. In this way, the feature extractor forms the interface between speech input and the encoder stack.

The speech encoder is built upon the extracted acoustic features and comprises a feature projection module followed by a Conformer stack [8]. The feature projection maps the 160-dimensional frontend features to the model hidden size of 1024, and the Conformer encoder then processes them into high-level speech representations while preserving the temporal resolution of 50 Hz. In the released large configuration, the encoder comprises 24 Conformer layers, each with 16 attention heads and a feed-forward network of intermediate dimension 4096. Relative positional representations [9] and a depthwise convolution with kernel size 31 are incorporated in each Conformer block, enabling the encoder to capture both long-range contextual dependencies and local acoustic structure effectively.

2.2 Modality Adapter↩︎

The modality adapter comprises the adapter inherited from SeamlessM4T-v2-large for temporal compression and a linear adapter for projection into the hidden space of the LLM decoder.

Given 50 Hz speech representations from the speech encoder, an intermediate feed-forward module first transforms the encoder outputs at the 1024-dimensional hidden size before passing them to the inherited adapter for temporal compression. In this adapter, the residual and self-attention branches each apply a one-dimensional convolution with kernel size 8, stride 8, and padding 4, followed by a gated linear unit. This design reduces the sequence length by approximately a factor of 8, yielding an effective frame rate of about 6.25 Hz while preserving the 1024-dimensional hidden size.

The adapter layer incorporates self-attention and a feed-forward network with an intermediate dimension of 4096, allowing the compressed representations to be refined after subsampling. Following the inherited adapter, a linear adapter is introduced to project the compressed speech representations from 1024 to 2560. This projection aligns the speech representations with the decoder input space, so that the resulting speech tokens have the same dimensionality as the text tokens.

2.3 LLM Decoder↩︎

The LLM decoder is inherited from Qwen3-4B-Instruct and leverages its strong instruction-following ability for general reasoning and token generation. To incorporate speech modality, the prepending fusion mechanism [10] is employed, whereby speech representations are prepended to the text embeddings of the instruction. In addition, parameter-efficient fine-tuning methods such as LoRA [11] are commonly used to adapt the LLM to downstream tasks without updating the full set of model parameters [1], [12].

3 Short Track↩︎

3.1 Corpora↩︎

In the constrained setting, training is conducted using CoVoST2 [13], EuroParlST [14], GigaST [15], and LibriSQA [16]. For data augmentation, synthetic data are generated with SeamlessM4T-v2-large and Qwen3-4B-Instruct. Validation is primarily performed on the MCIF [17] short-track datasets.

The training corpora, summarized in Table ¿tbl:tab:short-corpora?, comprise 5,056,973 samples with a total duration of 8,436.63 hours and an average duration of 6.01 seconds. Because these corpora differ substantially in annotation format, language coverage, and task supervision, dataset-specific processing is applied to construct a unified multilingual instruction-following training set for ASR, ST, and SQA. Specifically, synthetic translations are generated with SeamlessM4T-v2-large when target-language annotations are missing, low-quality samples are filtered using COMET [18], and synthetic question-answer pairs are generated with Qwen3-4B-Instruct when question-answering supervision is unavailable or limited. A small proportion of unanswerable examples is also introduced to improve robustness in SQA. The processing applied to each dataset is described below, and validation is primarily performed on the MCIF [17] short-track datasets, which provide a unified benchmark for multiple short-form tasks and languages.

CoVoST2 The original dataset provides English transcriptions together with translations from English into German and Chinese. Since Italian translations are not included, they are generated from the English transcriptions using SeamlessM4T-v2-large, after which low-quality translations are removed using COMET [18] scores below 0.85⁴. To increase the amount of training data for spoken question answering, synthetic question-answer pairs are generated with Qwen3-4B-Instruct. In addition, to explicitly model unanswerable cases, five percent of the pairs in each language are randomly sampled, their questions are replaced with random questions drawn from the remaining samples, and their answers are set to “Not answerable’’ in the corresponding language.

EuroParlST The original dataset provides English transcriptions together with translations from English into German and Italian. Two training subsets are available, train and train-noisy, but only the train subsets are used. No synthetic translations from English to Chinese are generated.

GigaST The original dataset provides English transcriptions together with translations from English into German and Chinese. Five training subsets are available, XS, S, M, L, XL, but only the M subsets are used to maintain a training data scale comparable to that of the other datasets. Since the English transcriptions are provided entirely in uppercase, text normalization is applied with Qwen3-4B-Instruct to convert them into natural sentence form. Low-quality translations are removed based on COMET scores lower than 0.85.

LibriSQA The original dataset provides English transcriptions together with question-answer pairs. To create synthetic translations, the English transcriptions are first translated into German, Italian, and Chinese using SeamlessM4T-v2-large, after which low-quality translations are removed based on COMET thresholds of 0.8, 0.8, and 0.75, respectively. Synthetic question-answer pairs are then generated using Qwen3-4B-Instruct. Following the same procedure as for CoVoST2, five percent of the pairs are randomly selected, and their answers are replaced with “Not answerable” in the corresponding language.

3.2 Training Strategy↩︎

The training pipeline is illustrated in Figure 1 (b). The SpeechLLM contains 4.73B parameters in total and includes a newly introduced two-layer linear module with an intermediate dimension of 3584. Parameter-efficient adaptation is performed by training 112.2M parameters, with LoRA applied to the query, key, and output projection modules using rank 8 and alpha 16. The model is optimized for two epochs with AdamW, using a total batch size of 128 and gradient clipping set to 1.0. Separate learning rates are used for the base model and LoRA parameters, set to \(1e{-4}\) and \(3e{-4}\), respectively, together with a cosine learning-rate scheduler and a warmup phase covering 3% of the total training steps. To improve robustness, data augmentation is applied, including speed perturbation with factors [0.9, 1.0, 1.1] and SpecAugment with two time masks and two frequency masks, where the maximum mask widths are 50 and 10, respectively. The model is trained on four NVIDIA A100 64GB GPUs for approximately two days, The final checkpoint is used for evaluation and submission.

3.3 System Evaluation↩︎

Validation is performed on the MCIF [17] short-track datasets covering multiple tasks and languages. The Short-form Instruction-Following Score (SIFS) is computed by summing the averaged task-level scores, as shown in Equation 1 . Here, \(\mathcal{T}\) includes ASR, ST, and SQA; \(\mathcal{L}_t\) denotes the language pairs with valid results for task \(t\); and \(m_{\ell,t}\) denotes the task-specific score, with ASR measured by \(1-\mathrm{WER}\).

\[\label{eq:sifs} \mathrm{SIFS} = \sum_{t \in \mathcal{T}} \frac{1}{|\mathcal{L}_t|} \sum_{\ell \in \mathcal{L}_t} m_{\ell,t}.\tag{1}\]

As shown in Table ¿tbl:tab:mcif-short? and Figure 2 (a), the primary system achieves competitive short-form performance, with a SIFS score of 2.0708, comprising an ASR accuracy of 0.8877, an average ST COMET score of 0.7550, and an average SQA BERTScore of 0.4281.

3.4 System Submission↩︎

As shown in Table ¿tbl:tab:mcif-short?, the primary short-form system achieves the best overall performance, with a SIFS score of 2.0708. The contrastive system is based on the long-form model and obtains a lower SIFS score of 2.0291, but remains competitive on SQA. This system is discussed further in Section 4.3.

Table 2 shows the IWSLT 2026 short-form results [19]. The primary submission outperforms the contrastive submission on most tasks, with comparable results on EN-EN and EN-IT. Overall, the primary submission is more balanced across all tasks, whereas the contrastive submission is less robust to unseen tasks.

4 Long Track↩︎

4.1 Corpora↩︎

In the constrained setting, the training corpora comprise LibriSQA for ASR, ST, and SQA, Nutshell [20] for SSUM and YTSeg [21] for ACHAP. For data augmentation, SeamlessM4T-v2-large and Qwen3-4B-Instruct are used to process the synthetic data. Validation is performed on the MCIF [17] long-track datasets.

The training corpora, summarized in Table ¿tbl:tab:long-corpora?, comprise 825,256 samples with a total duration of 9,496.36 hours and an average duration of 41.43 seconds. Because Nutshell provides only abstracts without transcriptions, and YTSeg contains highly noisy transcriptions, neither dataset provides reliable supervision for long-form speech recognition and translation. Initial zero-shot experiments further indicate that a SpeechLLM trained only on short-form data generalizes poorly to long-form speech, showing substantial performance degradation when required to process extended acoustic and linguistic context. This limitation is particularly important because speech recognition and translation constitute the foundation for higher-level long-form speech understanding tasks, such as speech summarization and audio chaptering. To address this gap, long-form training data for speech recognition, translation, and question answering are constructed from LibriSQA through artificial concatenation [22]. The processing applied to each dataset is described below. Validation is performed on the MCIF [17] long-track, which provide a unified benchmark for the long-form tasks and languages.

LibriSQA The original dataset provides exclusively short-form samples, but these can be reorganized into long-form examples by leveraging the chapter structure. For artificial concatenation, utterance-level audio segments are first grouped by chapter and then concatenated in sentence order, with a 0.5-second silence inserted between consecutive utterances to preserve natural boundaries. The transcriptions are processed in the same way: Chinese translations are concatenated directly, whereas spaces are inserted between consecutive sentences in English, German, and Italian to maintain sentence boundaries. Due to low-quality translations are removed during preprocessing, the resulting concatenated chapters in German, Italian, and Chinese are sometimes incomplete and therefore do not fully correspond to their English counterparts. Since question-answer pairs are provided at the utterance level, only one pair is randomly selected for each concatenated sample; otherwise, the total training duration would exceed 10,000 hours and become computationally prohibitive. The maximum duration at chapter level is limited to 10 minutes to reduce computational cost.

Nutshell The original dataset provides English abstracts from scientific talks, and translations into German, Italian, and Chinese are generated using SeamlessM4T-v2-large. Low-quality translations at abstract level are removed using COMET scores below 0.75, 0.75, and 0.8, respectively. The maximum duration at abstract level is limited to 15 minutes to reduce computational cost.

YTSeg The original dataset provides English transcriptions and chapter annotations for YouTube videos. Chapter titles and timestamps are automatically generated using Whisper and then merged into multiple target formats. The chapter titles in the target_text_ref field are extracted and processed to insert newlines and Markdown formatting. Because the videos often contain noisy acoustic content, such as music and background noise, VAD is applied to estimate voiced speech duration, and samples with speech ratios below 0.7 are removed. Translations into German, Italian, and Chinese are generated using SeamlessM4T-v2-large. No COMET-based filtering is applied to preserve complete chapter information. The maximum duration at chapter level is limited to 10 minutes, shorter than that used for Nutshell because chaptering requires substantially longer target texts.

4.2 Training Strategy↩︎

The training pipeline is illustrated in Figure 1 (c). The model follows the same architecture as the short-track system and is optimized using the same hyperparameters. Speed perturbation is disabled, while SpecAugment is kept unchanged. The gradient accumulation step is set to 8, and the fixed batch-size strategy is replaced with a dynamic bucket sampler to better accommodate variable-length speech inputs. For long-form audio, speech segmentation is applied before training. The model is trained on four NVIDIA A100 64GB GPUs for approximately three days. Due to time constraints, only one of the two scheduled training epochs is completed by the submission deadline. The final checkpoint is used for evaluation and submission.

4.2.1 Dynamic Bucket Sampler↩︎

To improve training efficiency under large variation in audio duration, we employ a dynamic bucket-based distributed batch sampler. The sampler assigns each sample to a duration bucket defined by a set of boundaries, and each bucket is associated with a dedicated per-replica batch size. This allows shorter utterances to be grouped into larger batches and longer utterances into smaller ones, reducing padding overhead and improving hardware utilization. At the beginning of each epoch, sample indices are deterministically shuffled and grouped by bucket. Within each bucket, synchronized global chunks are formed and evenly partitioned across replicas so that all workers process samples from the same duration range at each step.

When the number of samples in a bucket is not divisible by the global chunk size, the sampler either drops the remainder or pads the bucket by deterministically resampling indices from the same bucket during training. It further supports both sequential and round-robin scheduling across buckets, enabling either contiguous or interleaved traversal of different duration ranges. The sampler maintains explicit state, including the current epoch and iterator position, making it fully compatible with resumable and stateful distributed training.

4.2.2 Speech Segmentation↩︎

To process long-form speech with a model trained primarily on shorter utterances, three segmentation strategies are explored. The first strategy uses fixed windows, where each audio is divided into consecutive non-overlapping segments with durations of 15, 30, 45, or 60 seconds. This setting provides a simple and controlled baseline for analyzing the effect of segment length on downstream performance. Shorter windows can reduce memory and context burden within each segment, while longer windows preserve more local context but increase the risk of deletion and truncation errors.

The second strategy applies voice activity detection using the SpeechBrain CRDNN model to produce speech-based segments [23]. The activation_th, deactivation_th, and speech_th thresholds control speech onset detection, speech offset detection, and speech region selection, respectively. The default setting, (0.5, 0.25, 0.5), is used as the baseline; however, it may produce fragmented segments and limit the long-form context. Therefore, less aggressive variants are evaluated by relaxing these thresholds to (0.3, 0.2, 0.3) and (0.2, 0.1, 0.2), encouraging the detector to preserve more continuous speech regions and produce longer segments with broader context.

The third strategy adopts a hybrid segmentation approach inspired by [24], [25], combining pause-based segmentation with a duration constraint. The double-check step in the original pipeline is replaced with a divide-and-conquer algorithm, as shown in Algorithm 3. For segments exceeding the maximum allowed duration, the algorithm identifies the longest pause within the current interval and recursively splits the segment at that point. If no valid pause is found, the segment is kept unchanged.

4.3 System Evaluation↩︎

The long-form system is evaluated from four perspectives: short-form capability retention, hallucination, the evaluation metric, and speech segmentation. The analysis is conducted on the MCIF long-form tasks, covering ASR, ST, SQA, and SSUM, with task-specific scores and hallucination statistics reported for a comprehensive assessment.

4.3.1 Short-form↩︎

The long-form model is also evaluated on the MCIF short-form tasks to examine whether long-form extension preserves short-form capabilities. As shown in Table ¿tbl:tab:mcif-short?, the resulting contrastive system achieves a SIFS score of 2.0291, lower than the primary short-form score of 2.0708 but still competitive. This degradation may be partially explained by the training setup: to reduce computational cost, only LibriSQA short-form data are used instead of the full short-form training set. Overall, these results suggest that short-form capabilities can largely be retained after long-form extension.

4.3.2 Hallucination↩︎

Hallucination has a substantial impact on long-form evaluation, particularly for ASR and SSUM, and can significantly distort the apparent effectiveness of a segmentation strategy. As shown in Table 3, under CRDNN segmentation, including hallucinated outputs increases ASR WER from 0.1580 to 1.0161 for English, while also reducing ST COMET and SSUM BERTScore across target languages. In contrast, SQA is less affected, with only small differences in BERTScore after hallucinated outputs are included. This suggests that hallucination introduces task-dependent degradation, with generation-heavy tasks being vulnerable.

The ASR error breakdown in Table 1 further shows that hallucination mainly increases insertion errors. For CRDNN segmentation, the number of insertions rises sharply from 271 to 13,969 when hallucinated outputs are included, leading to a much higher WER. A similar but smaller trend is observed for the hybrid strategy, where insertions increase from 312 to 4,841. Manual inspection of the model outputs shows substantial hallucinated repetitions, which explains why excessive insertions become the dominant source of degradation. Inspired by the post-editing strategy in prior work [5], an alternative regular-expression-based post-editing strategy is applied to remove hallucinated repetitions and improve the reliability of long-form outputs before evaluation.

4.3.3 Evaluation Metric↩︎

The Hallucination-Penalized Instruction-Following Score (HIFS) is computed by applying a hallucination penalty to each available task score and then summing the task-level averages, as shown in Equation 2 . Here, \(\mathcal{T}\) denotes the set of evaluated tasks, including ASR, ST, SQA, and SSUM; \(\mathcal{L}t\) denotes the set of language pairs with valid results for task \(t\); \(m{\ell,t}\) denotes the task-specific metric score; and \(h_{\ell,t}/n_{\ell,t}\) denotes the corresponding hallucination rate. For ASR, \(m_{\ell,t}\) is computed as \(1-\mathrm{WER}\).

\[\label{eq:hifs} \mathrm{HIFS} = \sum_{t \in \mathcal{T}} \frac{1}{|\mathcal{L}_t|} \sum_{\ell \in \mathcal{L}_t} m_{\ell,t} \left(1 - \frac{h_{\ell,t}}{n_{\ell,t}}\right).\tag{2}\]

4.3.4 Speech Segmentation↩︎

Three speech segmentation strategies are compared for long-form inference: fixed-window segmentation, CRDNN-based segmentation, and hybrid segmentation. For fixed-window segmentation, Table 4 shows that the 30-second window achieves the best overall performance, with the highest HIFS score of 2.0663. The best ASR accuracy and ST COMET score are also obtained with this setting, while relatively low hallucination rates are maintained. Although slightly better SQA and SSUM scores are achieved by the 15-second and 45-second windows, respectively, these task-specific gains do not lead to the best overall performance. This suggests that a 30-second window provides a better trade-off between preserving sufficient context and avoiding unstable generation. Adaptive segmentation is further evaluated using CRDNN-based VAD with different parameter settings. As shown in Table 5, the \((0.5, 0.25, 0.5)\) setting achieves the highest HIFS score of 1.9591, mainly due to its stronger ST performance. In comparison, better SQA and SSUM scores and lower hallucination rates are achieved by the \((0.3, 0.2, 0.3)\) setting, but its lower ST score results in a slightly lower overall score. These results indicate that CRDNN segmentation is sensitive to its parameter settings, and that optimization for one task does not necessarily yield the best overall long-form performance. Finally, hybrid segmentation, which combines fixed-window and VAD-based segmentation, is evaluated in Table 6, and the scores are visualized in Figure 2 (b). The \((0.3, 0.2, 0.3)\) setting performs best among hybrid configurations, reaching an HIFS score of 2.0139. Compared with CRDNN segmentation, the overall score is improved, and the best ASR accuracy and SQA BERTScore are obtained among hybrid settings. However, this strategy remains below the fixed 30-second window in overall performance. These results suggest that adaptive segmentation can be beneficial but requires careful tuning, while the fixed 30-second window provides the most robust segmentation strategy for long-form inference.

4.4 System Submission↩︎

Based on the segmentation analysis, three systems are selected for submission: fixed 30-second segmentation as the primary system, and CRDNN-based and hybrid segmentation as contrastive systems. As shown in Table ¿tbl:tab:mcif-strategy?, the primary system achieves the best performance, with the highest HIFS score of 2.0663, as well as the best ASR accuracy and ST COMET score. Among the contrastive systems, CRDNN obtains the best SQA and SSUM scores, while hybrid segmentation provides a stronger balance with an HIFS score of 2.0139.

Table ¿tbl:tab:iwslt-long? shows the IWSLT 2026 long-form results [19]. No segmentation method consistently performs best across all tasks and metrics. Hybrid segmentation provides the most evident ASR improvement, whereas gains on other tasks are less stable. SSUM and ACHAP do not function reliably in this setting, with scores remaining low or degenerate across segmentation methods. These results highlight the difficulty of extending short-form to long-form speech processing.

5 Conclusion↩︎

This paper presented SpeechLLM systems for short-form and long-form instruction following. Strong short-form performance was achieved, while long-form performance remained sensitive to segmentation and hallucination. Fixed 30-second segmentation emerged as the most robust strategy, while hallucination mainly occurred as repetitive insertions that degraded ASR and SSUM. Long-form extension preserved much of the short-form capability, but better segmentation and hallucination mitigation remain important future directions.

Limitations↩︎

This work has several limitations. First, only one epoch of long-form training is completed due to time and computational constraints, which may limit adaptation to long-form speech. Second, using only a subset of short-form data during long-form extension may contribute to short-form degradation. Third, only three SpeechBrain VAD parameters are explored, leaving broader hyperparameter tuning and discourse-aware segmentation for future work. Finally, regular-expression-based post-editing provides limited hallucination mitigation, targeting surface-level repetitions but missing subtler hallucinations or semantic inconsistencies.

Acknowledgments↩︎

This paper has received funding from the European Union’s Horizon Europe programme under grant agreement No. 101213369 (project DVPS), and from the Digital Europe Programme (DIGITAL) under grant agreement No. 101198470 (project LLMs4EU). This work was also supported by the CINECA ISCRA-B programme through project LoST (Long-form Speech-to-Text Models via Sequence Compression), which provided access to high-performance computing resources and support.

6 Appendix↩︎

References↩︎

---

1. IWSLT 2026 Instruction Following page↩︎

2. SeamlessM4T-v2-large on Huggingface↩︎

3. Qwen3-4B-Instructon Huggingface↩︎

4. Unbabel/wmt22-cometkiwi-da↩︎