1 Introduction↩︎

Multimodal speech LLMs such as SALMONN [4], Qwen-Audio [5], SpeechGPT [6], and WavLLM [7] couple a frozen speech encoder with an instruction-tuned LLM [8] via a lightweight connector, enabling flexible multi-task inference through natural language prompts. The IWSLT 2026 Instruction-Following Shared Task [9] formalizes this paradigm with the MCIF benchmark, evaluating unified models on ASR, multilingual ST (EN\(\rightarrow\)), and SQA. The NAVER LABS 2025 system [1] demonstrated a competitive three-stage pipeline in the IWSLT 2025 constrained setting [10], but was not publicly released.

We provide the first open-source re-implementation, adapted to the IWSLT 2026 constraints (SeamlessM4T-v2-large encoder, Qwen3-4B-Instruct LLM—replacing the LLaMA-3.1-8B [11] backbone used in 2025). We further construct 100k synthetic instruction-following examples across ten speech-centric task types (Section 2.1) and ablate LoRA rank and learning rate configurations for Stage 2 text pre-training.

2 Task and Data↩︎

2.0.0.1 Shared Task.

We participate in the constrained condition, short audio track [9]. Evaluation on MCIF uses WER (\(\downarrow\)) for ASR, COMET [12] (\(\uparrow\)) for ST, and BERTScore-F1 [13] (\(\uparrow\)) for SQA. Task instructions follow the natural-language prompt format of [1].

2.0.0.2 Training Corpora.

Core speech data is from CoVoST 2 [14] and EuroParlST [15] (ASR/ST) and LibriSQA [16] (SQA); multilingual SQA pairs in DE, IT, ZH are obtained by machine-translating LibriSQA via SeamlessM4T-v2. Stage 3 additionally draws on NUTSHELL [17] for speech summarization and YTSeg [18] for audio chapter detection. Table 1 summarizes corpora per stage.

2.1 Synthetic Instruction-Following Data↩︎

We construct 100k synthetic examples (10k per task) from the provided corpora using open-weight Gemma models [19]. Seven text-grounded tasks are generated by Gemma-4-31B from reference transcripts: keyword extraction (T1), named entity recognition (T2), gist summarization (T3), topic labeling (T4), numeric QA (T5), and gist summarization in DE and ZH (T6–T7). Three audio-grounded tasks (T8–T10) are generated by Gemma-4-E4B-it directly from audio: vocal style description in EN, DE, and ZH.

For example, a NER target (T2) for the transcript “The Luks family eventually moved to Pottsville, in southern Pennsylvania” yields PER=[Luks], LOC=[Pottsville, S. Pennsylvania]. A vocal style target (T8) yields “The speaker has a measured, confident tone, speaking at a moderate pace with a clear articulation.” This data is released with our code.

3 System↩︎

3.0.0.1 Architecture.

Our model follows [1] and is illustrated in Figure 1. A frozen SeamlessM4T-v2-large encoder [2] produces 1024-dim frame representations. A trainable projector downsamples by 3\(\times\) via frame averaging, passes through a 4-layer Transformer encoder [20], and projects to the LLM hidden size. A LoRA-adapted [21] Qwen3-4B-Instruct [3] generates the response, with speech embeddings prepended at a <|speech|> placeholder.

3.0.0.2 Stage 1 — Projector Alignment.

Encoder and LLM are frozen; only the projector is trained (4 epochs, lr \(1\times10^{-4}\), constant, AdamW [22]). A.1 (ASR/ST): sampling 40% ASR, 18% ST-DE, 24% ST-ZH, 18% ST-IT (CoVoST 2 + EuroParlST). A.2 (ASR/ST/SQA): sampling 40% ASR, 10.5% ST-DE, 14% ST-ZH, 10.5% ST-IT, 25% SQA-EN (adds LibriSQA).

3.0.0.3 Stage 2 — Text-Only LoRA.

No audio; projector frozen; LLM adapted via LoRA for 1 epoch. Sampling: MT 60% (20% each DE/IT/ZH, from CoVoST 2 and EuroParlST transcripts) and QA 40% (10% each EN/DE/IT/ZH, LibriSQA + machine-translated). Three configurations:

• V1: rank 8, \(\alpha\)=16, lr \(3\times10^{-4}\), attn+FF layers

• V2: rank 16, \(\alpha\)=32, lr \(1\times10^{-5}\), cosine, all-linear

• V3: rank 32, \(\alpha\)=64, lr \(2\times10^{-4}\), cosine, all-linear

3.0.0.4 Stage 3 — Multimodal Merge.

Both the A.1 projector and V1 LoRA adapters are fine-tuned jointly; the speech encoder remains frozen. Sampling: 20% ASR, 10% each ST-{DE,IT,ZH}, 10% SQA-EN, 5% each SQA-{DE,IT,ZH}, 10% S2TSum, 15% AChap. Each speech batch (ST, SQA) is immediately followed by a paired text-only batch (MT, QA) to prevent catastrophic forgetting. Projector lr: \(1\times10^{-5}\) (constant); LoRA lr: \(3\times10^{-4}\) (cosine); 2 epochs on 4\(\times\)H100.

4 Experiments↩︎

(a) MCIF Benchmark Results (constrained, short audio)

(b) Stage-2 Text Eval (1k CoVoST-2, not MCIF-comparable)

(c) Stage-2 LoRA Rank Ablation

4.0.0.1 Setup.

Stages 1–2 train on one H200 GPU; Stage 3 on 4\(\times\)H100 80GB (DDP). Audio longer than 15 seconds is excluded due to memory constraints. Main evaluation uses the official mcif_eval tool; Stage 2 is separately evaluated on a 1k CoVoST 2 text subset (not MCIF-comparable).

4.0.0.2 Stage 1 & 3 Results (MCIF).

Table ¿tbl:tab:main? (a) shows MCIF results. Stage 1 improves ST and SQA over the SeamlessM4T-v2-large baseline at the cost of higher WER—consistent with [1]. A.2 boosts English SQA (0.267 vs.) but further degrades ASR. Stage 3 recovers ASR (23.49 WER), achieves the best ST COMET across all pairs (EN–ZH: 0.781), and strongly improves English SQA (0.346). Cross-lingual SQA remains lower due to sparse multilingual supervision.

4.0.0.3 Stage 2: Text Evaluation & LoRA Ablation.

Since Stage 2 is text-only (no audio), its results are evaluated on the 1k CoVoST 2 text subset and are not comparable to MCIF. Table ¿tbl:tab:main? (b) shows the gain from LoRA (V1) over the base Qwen3-4B [23], [24]; Table ¿tbl:tab:main? (c) ablates LoRA rank. V3 (rank 32) achieves the highest MT scores; V1 (rank 8) yields the best SQA F1 and is selected for Stage 3.

5 Conclusion↩︎

We re-implemented the NAVER LABS three-stage instruction-following pipeline [1] for the IWSLT 2026 constrained setting, adapting it to SeamlessM4T-v2-large and Qwen3-4B-Instruct. Our Stage 3 model achieves COMET 0.781 on EN–ZH ST and BERTScore-F1 0.346 on English SQA on MCIF, with consistent improvements over projector-only baselines. The 100k synthetic dataset released alongside our code provides a natural extension point for richer Stage 3 fine-tuning or future reinforcement learning [25] with task-specific rewards and LLM-as-judge criteria [26].

Limitations↩︎

Audio longer than 15 seconds is excluded due to GPU memory constraints, potentially limiting performance on longer utterances. Cross-lingual SQA relies on machine-translated QA pairs, which may introduce noise. Stage 2 ablation numbers are evaluated on a text-only subset and are not directly comparable to MCIF results.

6 Synthetic Dataset: Task Definitions and Examples↩︎

Each of the 100k synthetic examples (10k per task) is paired with a natural-language instruction prompt; the model is trained to produce the target output. Tasks T1–T7 are text-grounded: Gemma-4-31B generates targets from reference transcripts. Tasks T8–T10 are audio-grounded: Gemma-4-E4B-it generates targets directly from speech audio. One example per task is shown below.

6.0.0.1 T1 — Keyword Extraction.

Definition: Extract the most salient keywords from the spoken utterance as a comma-separated list.
Input: “These data components in turn serve as the ‘building blocks’ of data exchanges.”
Output: data components, building blocks, data exchanges

6.0.0.2 T2 — Named Entity Recognition.

Definition: Identify and classify named entities (PER, ORG, LOC) present in the transcript.
Input: “Saunders was born in Farnborough, Hampshire, the son of a British Army officer.”
Output: PER:[Saunders], ORG:[British Army], LOC:[Farnborough, Hampshire]

6.0.0.3 T3 — Gist Summarization (EN).

Definition: Produce a one-sentence English summary of the spoken content.
Input: “...if you don’t want to look at all of the man pages, you just want to see a very short description of what a command does—type whatis followed by the command name...”
Output: The whatis command provides a one-line description of any Unix/Linux command, e.g.whatis sort or whatis cat.

6.0.0.4 T4 — Topic Labeling.

Definition: Assign a single short topic label (one to three words) to the utterance.
Input: “These data components in turn serve as the ‘building blocks’ of data exchanges.”
Output: data exchange

6.0.0.5 T5 — Numeric QA.

Definition: Generate a question–answer pair where the answer is a number explicitly mentioned in the utterance.
Input: “Only one referendum has been held at the national level in Guyana.”
Output: Q: How many national-level referendums have occurred in Guyana? A: 1 referendum

6.0.0.6 T6 — Gist Summarization (DE).

Definition: Produce a one-sentence German gist summary of the spoken content.
Input: “...if you don’t want to look at all of the man pages, you just want to see a very short description of what a command does...”
Output: Der Befehl whatis liefert eine einzeilige Beschreibung eines Unix-Kommandos, etwa whatis ls oder whatis sort.

6.0.0.7 T7 — Gist Summarization (ZH).

Definition: Produce a one-sentence Mandarin Chinese gist summary of the spoken content.
Input: “...if you don’t want to look at all of the man pages, you just want to see a very short description of what a command does...”
Output:

6.0.0.8 T8 — Vocal Style Description (EN).

Definition: Describe the speaker’s vocal style, tone, and delivery in English, inferred directly from the speech audio (no transcript used).
Input: [audio: “These data components in turn serve as the ‘building blocks’ of data exchanges.”]
Output: The speaker has a measured, confident tone, speaking at a moderate pace with clear articulation.

6.0.0.9 T9 — Vocal Style Description (DE).

Definition: Describe the speaker’s vocal style in German, inferred directly from the speech audio.
Input: [audio: “These data components in turn serve as the ‘building blocks’ of data exchanges.”]
Output: Die Sprecherin klingt ruhig und professionell mit einem gleichmäßigen Sprechtempo und einer freundlichen, sachlichen Betonung.

6.0.0.10 T10 — Vocal Style Description (ZH).

Definition: Describe the speaker’s vocal style in Mandarin Chinese, inferred directly from the speech audio.
Input: [audio: “These data components in turn serve as the ‘building blocks’ of data exchanges.”]
Output:

References↩︎

---

1. https://github.com/anand-kamble/iwslt2026-instruction-following↩︎