1 Introduction↩︎

Speech recognition has been decoded the same way for a long time. An encoder reads the audio, and an attention decoder spells the transcript one token at a time, feeding each token back to itself to produce the next [1], [2]. The encoders have grown very strong, but the decoder is serial by construction, so a thirty second clip costs a few hundred sequential forward passes no matter how much hardware is available. Diffusion language models decode differently, starting from a canvas of noise and refining the whole sequence in parallel over a small number of denoising steps; they now reach quality competitive with autoregressive models of similar size [3]. The natural question is whether speech can be transcribed the same way, so that the cost of recognition is set by the number of denoising steps and not by the length of the transcript.

Two systems have shown that diffusion can transcribe. TransFusion denoised character sequences with multinomial diffusion [4], and Whisfusion trained a masked-diffusion decoder on top of frozen Whisper features [5]. Both train a decoder specifically for the task, and both are English only. We ask a different question: can an existing, general-purpose diffusion language model be taught to hear, so that recognition runs through its own decoder with no new decoder trained at all?

We take DiffusionGemma, an open-weight 26B mixture-of-experts model that generates text by discrete diffusion (does not have audio support), and give it a native audio input. We keep the backbone frozen. We add a frozen Whisper encoder as an acoustic feature extractor [2], a small trainable projector that maps those features into the model embedding space, and low-rank adapters [6] so the frozen model can learn to attend to the new modality. The projector-into-language-model recipe follows the connectors that gave sight and hearing to text-only models [7]–[9], applied here for the first time to a diffusion decoder. Only about 42M parameters train, which is 0.16 percent of the backbone. The result reaches 6.6 percent word error rate on LibriSpeech test-clean, converges in about eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages.

The challenge lied in grounding, a frozen language model has never seen a spectrogram, so its embedding space has no notion of formants or phonemes, and the two obvious training objectives both route their signal through cross-attention. If attention has already decided the audio is noise, no gradient reaches the projector to make it useful, and the projector stays noise, which in turn gives attention no reason to change. We break this deadlock with a connectionist temporal classification loss applied directly through the frozen output head, which forces the audio embeddings to be linearly predictive of the transcript without asking attention to trust them first.

Our contributions are the following.

• An audio-native pathway for a frozen discrete-diffusion language model, trained only through a projector and low-rank adapters, with no decoder trained from scratch.

• A diagnosis of the grounding failure and a direct-supervision fix that breaks the projector and attention deadlock in a few hundred steps.

• An empirical study of the accuracy and speed of diffusion decoding for speech, including a step count sweep that shows recognition cost decoupled from transcript length.

• A single adapter trained on six languages (English, German, French, Spanish, Hindi, Mandarin) and evaluated here on English, Hindi, and Mandarin, together with an analysis of where it trails autoregressive Whisper and why.

2 Background↩︎

2.1 The diffusion backbone↩︎

DiffusionGemma is a 26B mixture-of-experts model with 128 experts, top-8 routing, and roughly 4B active parameters per token. It has 30 transformer layers, a hidden size of 2816, and a vocabulary of 262144. It generates text by discrete diffusion rather than autoregression.

Most recent diffusion language models use an absorbing state, where corruption replaces tokens with a special <mask> symbol and generation fills masks in [3], [10], [11]. DiffusionGemma instead uses uniform, random-token diffusion. It starts from a fixed-length canvas, up to 256 slots, filled with random tokens drawn from the vocabulary. At each denoising step the model reads the whole canvas at once, keeps the low-entropy predictions it is confident about, and renoises the remaining positions back to fresh random tokens. After a few steps the noise anneals into text (Figure 1). Training mirrors this process. A clean sequence is corrupted by replacing a fraction \(\gamma\) of positions with uniform random tokens, and the model is trained to recover the originals at the corrupted positions. We formalize the process in Section 3.4.

2.2 Encoder-decoder structure↩︎

Architecturally the model is an encoder-decoder with tied transformer weights. The encoder reads the prompt causally into a read-only key-value cache. The decoder refines the canvas with bidirectional self-attention while cross-attending to that cache. Multimodal inputs are injected by scattering projected features into placeholder token positions of the encoder input embeddings. The existing vision path uses an image token identifier for this. Out of the box the model accepts text, images, and video, and produces text. It does not accept audio and providing that modality is one of our key contributions in our work.

2.3 Connecting an encoder to a language model↩︎

A frozen language model can be given a new input modality by training a small connector that maps features from a modality encoder into the model embedding space, while the backbone stays fixed or is only lightly adapted. This recipe added vision to text-only models through a projector over a vision encoder [7], and audio to language models through speech encoders and adapters [8], [9], usually on top of open model families such as Gemma [12]. Our audio pathway follows the same pattern. What is new is the decoder it feeds. Every connector we are aware of feeds an autoregressive model, whereas here the connector conditions a diffusion decoder that produces the transcript in parallel, which changes both how the model must be grounded and how it decodes.

3 Method↩︎

3.1 Notation and setup↩︎

Let \(\mathcal{V}\) be the vocabulary with \(V=|\mathcal{V}|=262{,}144\). A transcript is a token sequence \(\mathbf{y}=(y_1,\dots,y_M)\). We write it onto a fixed canvas of length \(L\) (we use \(L{=}256\) in training and \(L{=}192\) at inference) as the clean target \(\mathbf{x}_0=(\mathbf{y},\mathrm{\small eos},\mathrm{\small pad},\dots)\in\mathcal{V}^{L}\). The audio is a log-mel spectrogram \(\mathbf{m}\in\mathbb{R}^{80\times3000}\), since Whisper pads or trims to thirty seconds. We keep the DiffusionGemma backbone and the Whisper encoder frozen and train only a projector \(\mathcal{P}_\phi\) and low-rank adapters \(\theta\). We write \(\sigma_\tau(z)=\tau\tanh(z/\tau)\) for the final logit softcap with \(\tau{=}30\), and \(W_U\in\mathbb{R}^{V\times d}\) for the frozen tied output head, with model width \(d{=}2816\). The overall architecture is shown in Figure 2.

3.2 Audio pathway↩︎

In our first attempt, we skipped the acoustic encoder entirely. Some unified Gemma models project raw waveforms straight into the embedding space, so we fed the model short audio frames and let the backbone infer the acoustics to test this approach. And it failed completely, as a frozen language model that has never seen audio cannot build an acoustic front end from gradient signal alone, and a shallow projector cannot do feature extraction on its own. Word error rate plateaued near 94 percent while the model produced fluent text unrelated to the audio.

The working design uses a frozen Whisper-small encoder [2] as a feature extractor, not as a decoder. It carries the acoustic modeling that self-supervised representations [13] and convolution-augmented transformers [14] established, which is the map from sound to linguistic features a frozen text model cannot learn on its own. The encoder produces \(\mathbf{a}\in\mathbb{R}^{F\times d_w}\) with \(F{=}1500\) frames and \(d_w{=}768\). The projector maps these to audio tokens \(\mathbf{u}\in\mathbb{R}^{T\times d}\), \[\begin{align} \mathbf{h}^{(0)} &= \mathrm{GELU}(\mathbf{a}\,W_{\text{in}}),\\ \mathbf{h}^{(l)} &= \mathrm{GELU}\!\big(\mathrm{Conv1d}^{(l)}(\mathbf{h}^{(l-1)})\big),\quad l=1,2,3,\\ \mathbf{u} &= \mathrm{LN}\!\big(\mathbf{h}^{(3)}\,W_{\text{out}}\big), \end{align}\] with \(W_{\text{in}}\in\mathbb{R}^{d_w\times d_h}\), \(W_{\text{out}}\in\mathbb{R}^{d_h\times d}\), hidden width \(d_h{=}1280\), and convolutions of kernel 3, stride 2, and padding 1. Each convolution maps a length \(n\) to \(\lfloor(n-1)/2\rfloor+1\), so \(1500\to750\to375\to188\) and \(T{=}188\), about 6.25 tokens per second. The projector, together with the adapters introduced below, is the only trained part of the model.

3.3 Conditioning the diffusion decoder on audio↩︎

We build a fixed-length prompt \[\texttt{[BOS]}\;\text{instr}\;\texttt{[BOA]}\;\big(\texttt{<audio>}\big)^{\!\times T}\;\texttt{[EOA]},\] with audio placeholder positions \(\mathcal{A}\), \(|\mathcal{A}|{=}T\). We embed the prompt and overwrite the placeholders with the projected audio tokens, \[\mathbf{E}_j=\begin{cases}\mathbf{u}_{\rho(j)} & j\in\mathcal{A},\\[2pt] \mathrm{Emb}(c_j) & \text{otherwise,}\end{cases}\] where \(\rho\) indexes projector rows, reusing the scatter mechanism the vision path uses. The frozen encoder, with adapters \(\theta\), reads this into a key-value cache \(\mathbf{K}=\mathrm{Enc}_\theta(\mathbf{E})\), and the decoder cross-attends to \(\mathbf{K}\) while denoising, as in Eq. 2 . The prompt length is constant because \(T\) is fixed. For clips shorter than thirty seconds we zero the attention mask on the silence-pad audio positions so the encoder ignores them. The canvas holds the transcript, an end-of-sequence marker, and padding, and is the decoder target \(\mathbf{x}_0\).

3.4 Uniform discrete diffusion↩︎

With the audio in the cache, we can state the generation process. Unlike the absorbing-state schemes used by most diffusion language models [3], [10], [11], DiffusionGemma corrupts toward uniform random tokens rather than a mask. A single scalar \(\gamma\in[0,1]\) sets the corruption level and plays the role a timestep plays in continuous diffusion. Corruption acts independently at each position, \[q(x_\gamma^i = k \mid x_0^i;\gamma) = (1-\gamma)\,\mathbf{1}[k=x_0^i] + \frac{\gamma}{V}, \label{eq:forward}\tag{1}\] which is the cumulative uniform D3PM kernel \(\bar Q_\gamma=(1-\gamma)\,I+\tfrac{\gamma}{V}\mathbf{1}\mathbf{1}^{\!\top}\) [10]. Equivalently, draw \(b_i\sim\mathrm{Bernoulli}(\gamma)\) and \(r_i\sim\mathrm{Unif}(\mathcal{V})\) and set \(x_\gamma^i=(1-b_i)\,x_0^i+b_i\,r_i\). There is no special mask symbol, so a corrupted position holds a real, random vocabulary token.

The model is trained in the \(\mathbf{x}_0\) parameterization. Given the corrupted canvas and the audio conditioning \(\mathbf{c}\) carried by the cache \(\mathbf{K}\), it predicts the clean token at each position, \[\begin{align} p_{\theta,\phi}(x_0^i\mid \mathbf{x}_\gamma,\mathbf{c})&=\mathrm{softmax}\!\big(\sigma_\tau(W_U\,\mathbf{h}_i)\big),\\ \mathbf{h}&=\mathrm{Dec}_\theta(\mathbf{x}_\gamma,\mathbf{K}). \end{align} \label{eq:denoiser}\tag{2}\]

3.5 Training objectives↩︎

Three losses share one projector and one set of adapters.

3.5.0.1 Diffusion loss.

This is the model’s own objective, the reweighted \(\mathbf{x}_0\)-prediction surrogate of the discrete-diffusion variational bound [10], [15]. Let \(\mathcal{C}\) be the corrupted, non-pad positions. Taking the expectation over \(\gamma\sim\mathrm{Unif}(0,1)\), the data, and the corruption, \[\mathcal{L}_{\text{diff}}=\mathbb{E}\!\left[\frac{w(\gamma)}{|\mathcal{C}|}\sum_{i\in\mathcal{C}}-\log p_{\theta,\phi}(x_0^i\mid \mathbf{x}_\gamma,\mathbf{c})\right], \label{eq:diff}\tag{3}\] with an optional importance weight \(w(\gamma)\in\{1,\,1/\gamma\}\). The loss is applied only at corrupted positions. Because the objective is bidirectional and the frozen text prior explains much of the visible clean context, the gradient asking the model to use audio is diluted (Section 3.6). We therefore force \(\gamma{=}1\) on a fraction of batches, where the audio is the only source of information.

3.5.0.2 Autoregressive auxiliary loss.

The encoder is causal, so we can teacher-force the transcript through it and read a standard next-token loss, \[\mathcal{L}_{\text{ar}}=\mathbb{E}\Big[\frac{1}{M}\sum_{i=1}^{M}-\log\,\mathrm{softmax}\!\big(\sigma_\tau(W_U\mathbf{g}_i)\big)_{y_i}\Big], \label{eq:ar}\tag{4}\] where \(\mathbf{g}_i\) is the encoder state that predicts \(y_i\) from \(\mathbf{y}_{<i}\) and the audio. Because the encoder shares weights with the diffusion decoder and reads the same audio tokens, this dense per-token gradient transfers to parallel denoising.

3.5.0.3 CTC loss.

We decode the projector rows directly through the frozen output head, with no softcap, and align them to the transcript with connectionist temporal classification [16]. With per-frame distributions \(\mathbf{q}_t=\mathrm{softmax}(W_U\mathbf{u}_t)\) and the blank symbol at index 0, \[\mathcal{L}_{\text{ctc}}=-\log\!\!\sum_{\pi\in\mathcal{B}^{-1}(\mathbf{y})}\;\prod_{t=1}^{T}\mathbf{q}_t(\pi_t), \label{eq:ctc}\tag{5}\] where \(\mathcal{B}\) collapses repeats and removes blanks. Feasibility needs \(T\ge M\), so we cap targets to the number of real audio tokens. The full objective is \[\mathcal{L}=\mathcal{L}_{\text{diff}}+\lambda_{\text{ar}}\,\mathcal{L}_{\text{ar}}+\lambda_{\text{ctc}}\,\mathcal{L}_{\text{ctc}}. \label{eq:total}\tag{6}\]

3.5.0.4 Low-rank adaptation.

We add LoRA adapters [6] of rank 16 and scaling 32 on the query, key, value, and output projections of both the encoder and the decoder self-attention. The encoder adapters let the model fold audio into the cache, and the decoder adapters let it attend to that cache during denoising. The experts and the vision tower stay frozen. The projector and the adapters together are 42.3M trainable parameters, which is 0.16 percent of the backbone.

3.6 Why grounding fails, and how CTC fixes it↩︎

With only the diffusion and autoregressive losses, training stalled in a way that took us a while to understand. The curves fell for a few hundred steps and then went flat. The diffusion loss settled near 8 and the autoregressive loss sat around 4.5, close to chance, and neither moved for thousands of steps.

Both losses reach the projector only through attention over the audio positions. Let \(\alpha_t\) be the aggregate attention mass the model places on audio token \(t\). To first order the gradient the attention-routed losses send to \(\mathbf{u}_t\) scales with that mass, \[\frac{\partial \mathcal{L}_{\text{diff}}}{\partial \mathbf{u}_t}\;\sim\;\alpha_t\,W_V^{\!\top}\boldsymbol{\delta}_t, \label{eq:gradstall}\tag{7}\] where \(\boldsymbol{\delta}_t\) is the backpropagated error at the attended positions and \(W_V\) the value projection. At initialization \(\mathbf{u}_t\) is random, the frozen prior explains the visible clean context, and the model learns \(\alpha_t\to0\), which sends this gradient to zero. The gradient that would raise \(\alpha_t\) is also small, because \(\mathbf{u}_t\) carries nothing worth attending to. Both directions vanish at the same fixed point. The projector and the attention each wait for the other to move first.

CTC removes the \(\alpha_t\) factor. Its gradient to the projector is the linear-head cross-entropy gradient composed with the CTC forward-backward occupancy \(\boldsymbol{\rho}_t\), the posterior probability of each label at frame \(t\), \[\frac{\partial \mathcal{L}_{\text{ctc}}}{\partial \mathbf{u}_t}=W_U^{\!\top}\big(\mathbf{q}_t-\boldsymbol{\rho}_t\big), \label{eq:gradctc}\tag{8}\] which is nonzero whatever the attention does. It pushes \(\mathbf{u}_t\) into the region of embedding space where \(W_U\mathbf{u}_t\) predicts transcript tokens, with no attention required. Once \(\mathbf{u}_t\) carries content, \(\alpha_t\) grows and the gradients in Eq. 7 revive. Adding the term broke the plateau on our first attempt: the CTC loss fell from 24 to 8.6 in about 300 steps and held-out token accuracy climbed off the floor. The CTC head is a training-time scaffold, which we drop at inference so that recognition runs entirely through the diffusion decoder.

3.7 Inference as parallel denoising↩︎

Whisper’s encoder is fixed to a thirty second window, so longer audio is split at silence. We take the quietest short frame near a thirteen second target inside a three second search window and cut there, so no word is split and the non-overlapping segments concatenate with no stitching. Each segment is denoised on a canvas of \(L{=}192\) slots by Algorithm 3.

The sampler is the base model’s uniform-diffusion process, conditioned on audio. Each step recomputes the denoiser logits over the whole canvas, accepts the positions whose predictive entropy \(H(\mathbf{p}_i)\) falls below a bound \(\eta\), and renoises the rest to fresh uniform tokens. A linear temperature schedule anneals from 0.8 down to 0.4, self-conditioning [17] feeds the previous prediction forward, and a stability-and-confidence criterion stops early once the canvas settles. Optional classifier-free guidance [18] combines a conditional pass with an unconditional pass that hides the audio cache, \(\boldsymbol{\ell}=\boldsymbol{\ell}^{u}+w(\boldsymbol{\ell}^{c}-\boldsymbol{\ell}^{u})\), to sharpen the effect of the audio without retraining. A short post-processing pass removes repeated words and n-grams, the canvas-fill artifacts a diffusion decoder produces when the transcript is shorter than the canvas.

3.7.0.1 Cost.

Autoregressive decoding needs \(M\) sequential decoder passes to emit \(M\) tokens [19]. Diffusion decoding needs at most \(S\) passes over the whole canvas in parallel, plus one encoder pass, and doubles the decoder passes only when guidance is on. The cost is set by \(S\) and is independent of the transcript length \(M\). With \(S\approx8\ll M\), and the flat accuracy curve of Table 4, this is the source of the speedup.

4 Experimental Setup↩︎

4.0.0.1 Data.

Stage one, we train the English model on the 100 hour clean subset of LibriSpeech [20]. The multilingual model warm-starts from the English checkpoint and continues on FLEURS [21] across six languages, English, German, French, Spanish, Hindi, and Mandarin. A further stage adds VoxPopuli parliamentary speech [22]. The model has seen roughly 219 hours of audio in total.

4.0.0.2 Optimization.

We use AdamW with a learning rate of \(1\mathrm{e}{-3}\), betas of 0.9 and 0.95, and weight decay of 0.01, with 100 warmup steps and a cosine decay to one tenth of the peak. We clip gradients at norm 1.0. The backbone runs in bfloat16 and the projector in float32 for stable updates. Batches hold 4 utterances with 2 gradient accumulation steps, reduced to 2 and 6 on the longer VoxPopuli material. All runs use a single H100.

4.0.0.3 Evaluation.

We report word error rate with the Whisper text normalizer applied to both reference and hypothesis, which is the Open-ASR and Artificial Analysis convention. For Hindi and Mandarin we report character error rate, because word error rate penalizes scripts whose word boundaries do not match the reference segmentation and overstates the error. We also report throughput as a multiple of real time.

4.0.0.4 Configuration.

Table 1 lists the settings used for the reported runs.

4pt

5 Results↩︎

5.1 English recognition and grounding dynamics↩︎

On LibriSpeech test-clean, English word error rate walked down over ten epochs, from roughly 90 percent early in training to about 50 percent, then to 14.6 percent, and finally to 6.6 percent at the last checkpoint (\(n{=}100\) test-clean utterances). The two intermediate points come from small held-out samples and are approximate; the final number is on the full evaluation set. This is single-digit word error rate from a frozen 26B backbone, a frozen Whisper encoder, and about 42M trainable parameters.

Grounding was not monotone in the metrics. Token accuracy reached 0.50 and CTC loss kept falling while the model still said nothing useful. A manual decode at 0.4 epochs showed the greedy CTC output emitting the unigram prior, the most frequent tokens in frequency order such as “the of to he”, with no relation to the audio, even though the loss curve looked healthy. Only repeated exposure over many passes brought the projector and the attention into a configuration that actually transcribes, so we ran a manual decode every half epoch. As training progressed the outputs moved from repeated fragments to fluent text with occasional acoustic slips, such as “tents” for “tense”, the error profile of a model transcribing from the audio rather than falling back on its text prior.

Table 2 shows the same run in numbers. The autoregressive loss stays near chance and token accuracy hovers around 0.4 to 0.5 through the first epoch, then both move sharply once the projector grounds. Without the CTC term the same configuration did not ground within the compute we ran, which is the failure Section 3.6 analyzes; we observed this during development rather than as a controlled ablation.

5.2 Multilingual recognition↩︎

Warm-starting from English and training on FLEURS gave one adapter covering all six languages. We report the three for which we ran held-out evaluation, English, Hindi, and Mandarin. German, French, and Spanish are in the training mix but were not separately evaluated here. Adding VoxPopuli traded a small amount of read-speech accuracy for a gain on conversational and accented audio, a movement along the accuracy frontier rather than a free improvement. Table 3 reports the adapter under the Whisper normalizer.

5.3 Parallel decoding↩︎

Because the decoder refines the entire canvas at once, transcription cost is set by the number of denoising steps and not by the length of the transcript. Table 4 sweeps the step count on FLEURS English. The curve is close to flat. Moving from 8 steps to 48 buys about half a point of word error rate and costs roughly three times the latency. The model converges in about eight parallel passes. For a ten second clip this is on the order of one second of model time, and it does not grow with how much the speaker says.

5.4 Comparison with diffusion and autoregressive systems↩︎

Table 5 places our LibriSpeech test-clean result beside other diffusion and non-autoregressive systems. These are the numbers reported in the respective papers, not controlled reruns, and the systems differ in encoder, training data, tokenization, and decoding, so the table is context rather than a head-to-head comparison. We place our result next to TransFusion [4] and Whisfusion [5] for reference, not to claim a controlled win.

Against autoregressive Whisper, which remains the strongest baseline, our model trails, as Table 6 shows. All numbers use the Whisper normalizer. The gap is largest on FLEURS and VoxPopuli and smallest on LibriSpeech, the one domain where we had a hundred clean hours.

4pt

5.5 What is the bottleneck↩︎

We ran ablations to locate the source of the residual error, because the answer decides whether the method is broken or simply undertrained. We ruled out one suspect at a time.

First, encoder quality. We swapped the Whisper-small front end for Whisper-large-v3 and retrained on the same 100 hour set. Word error rate at matched data moved from 14.75 percent at three epochs to 13.15 at six and 8.95 at eight, which is roughly even with the small-encoder run at the same data scale. A stronger encoder did not move the floor at this data budget, so the encoder is not the current limit.

Second, audio resolution. We reduced the subsample factor below eight to give the projector more audio tokens per second. This regressed to 44.7 percent word error rate rather than improving, so token resolution is not the limit either, and the eight-fold subsample is not starving the decoder.

Third, decode steps. The step sweep in Table 4 is flat, so the sampler is not leaving accuracy on the table.

With encoder quality, resolution, and step count ruled out, the most likely remaining explanation is data scale and adapter alignment. Whisper-large-v3 was trained on a few million hours and Whisper-small on about 680 thousand. This model has seen roughly 219 hours, three orders of magnitude fewer, and it already reached 6.6 percent on the single domain where it had a hundred clean hours. The frontend ablation points the same way, since a stronger encoder did not help at this data budget, which is what one expects when the bottleneck is data rather than features. The evidence is consistent with a data bottleneck rather than a broken method, but we cannot establish this without a scaling curve, which we leave to future work.

6 Related Work↩︎

6.0.0.1 How speech recognition decodes.

End-to-end recognition grew from two ideas. Connectionist temporal classification aligns a frame sequence to a shorter label sequence without explicit segmentation [16]. Attention-based sequence-to-sequence models instead let a decoder attend over the encoded audio and spell the transcript [1], a pattern the transformer generalized [19]. Most progress since then has come from stronger encoders, including self-supervised representations [13] and convolution-augmented transformers [14], and from scale and weak supervision [2]. In all of these the transcript is produced one token at a time.

6.0.0.2 Non-autoregressive and diffusion text generation.

Parallel generation began with non-autoregressive translation, which emits the whole target at once [23]. Diffusion models, developed for continuous data [15], [24], were brought to language both by embedding tokens in a continuous space [25], [26] and by defining the corruption directly on discrete tokens [10], [11]. Self-conditioning improved discrete diffusion by feeding the previous prediction back into the next step [17], a technique our sampler uses. Large masked-diffusion language models now reach quality competitive with autoregressive models of similar size [3]. DiffusionGemma sits in this family but corrupts toward uniform random tokens rather than an absorbing mask.

6.0.0.3 Diffusion speech recognition and audio language models.

Two systems put diffusion decoding directly on speech. TransFusion transcribed characters with multinomial diffusion as a proof of concept [4], and Whisfusion trained a masked-diffusion decoder over frozen Whisper features [5]. Both build that decoder from scratch, whereas we reuse one that already exists. Separately, a large body of work gives new modalities to frozen or lightly adapted language models through a trained connector, for vision [7] and for audio [8], [9], often on top of open model families such as Gemma [12]. Our work joins these two threads. We reuse an off-the-shelf diffusion language model with no new decoder, connect a frozen speech encoder through a projector and low-rank adapters [6], ground it with connectionist temporal classification [16], decode with uniform random-token diffusion rather than absorbing masks, and train one adapter on six languages.

7 Limitations↩︎

The model trails autoregressive Whisper on every benchmark we tested, by the largest margin on multilingual read speech. The evidence points to data scale as the cause rather than architecture, but that remains a hypothesis until we train on substantially more audio. The Whisper encoder fixes the input window at thirty seconds, so real streaming is not possible and long audio is handled by silence segmentation, which can err when speech has no clear pauses. The diffusion decoder can produce repetition artifacts on short utterances, which we currently remove with a post-processing pass rather than at the source. Finally, our comparisons to Whisper use published ranges under the same normalizer rather than a single controlled re-run of every baseline, so the Whisper columns should be read as reference points rather than exact head-to-head numbers.

8 Conclusion↩︎

A frozen language model can learn to hear if the audio projector is supervised directly rather than only through attention. Loss curves and token accuracy will look healthy well before the model is actually grounded, so manual decoding matters during training. Once grounded, diffusion decoding makes the length of what a speaker said stop mattering for cost, because recognition converges in about eight parallel steps. The grounding recipe, the CTC unlock, and the diffusion decoder all work today; what the model still lacks is exposure to more hours of audio.

References↩︎