1 Introduction↩︎

On-device AI inference is becoming increasingly important for mobile applications due to its advantages in privacy, latency, and reliability. Among these applications, real-time speech transcription has emerged as a key capability for intelligent assistants and accessibility services, where timely and accurate transcription is critical to user experience [1], [2]. Enabling such functionality directly on mobile devices is particularly desirable, as it avoids the latency and connectivity limitations of cloud-based processing while preserving user privacy.

Recent systems increasingly rely on large foundation models for robust transcription, and Whisper [3] has emerged as a de facto standard due to its strong accuracy and scalability. As a result, there is growing interest in running Whisper directly on mobile devices for real-time, on-device transcription [4]–[7]. To meet the stringent latency and energy requirements of real-time processing, modern mobile platforms are equipped with Neural Processing Units (NPUs) that offer high-throughput and energy-efficient inference. However, fully leveraging mobile NPUs for real-time Whisper remains challenging, due to fundamental mismatches between Whisper’s streaming inference pipeline and the execution characteristics of NPUs.

First, existing streaming approaches for Whisper incur substantial redundant computation. Whisper is prone to hallucinations when processing short audio inputs that deviate from its 30-second training distribution [5]. To mitigate this, prior systems rely on padding and overlapping audio buffers, effectively reprocessing previously seen inputs to stabilize decoding as illustrated in Figure 1. However, this design treats padding as a necessity, even though its primary role is to suppress hallucinations rather than to improve acoustic coverage. As a result, a large fraction of computation is spent on redundant or artificial inputs, while newly arrived speech accounts for only a small portion of each inference step.

Second, Whisper’s autoregressive decoding is poorly matched to the static execution model of mobile NPUs. NPUs rely on precompiled static graphs with fixed tensor shapes, whereas decoding involves dynamically growing key-value (KV) caches. Existing approaches either execute attention over the maximum sequence length at every decoding step, incurring substantial redundant computation, or rely on step-specific graphs, which introduce significant runtime overhead due to frequent graph dispatch.

To address these challenges, we present NPUsper, a system for real-time on-device Whisper transcription on mobile NPUs. Our key insight is that the inefficiency of streaming Whisper stems from two structural mismatches: (i) hallucination-induced padding overhead caused by unstable temporal grounding under short inputs, and (ii) the mismatch between dynamic autoregressive decoding and static-graph NPU execution. NPUsper resolves these mismatches through temporal-grounding-aware decoding control and NPU-aware controlled unrolling.

We implement and evaluate NPUsper on commercial mobile devices [8], [9] and show that it achieves up to 4.84\(\times\) lower per-word latency, up to 33.2\(\times\) lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy.

2 Background and Related Work↩︎

2.1 Live Transcription Systems with Whisper↩︎

Whisper. Whisper [3] is an encoder–decoder Transformer model widely used for speech transcription due to its strong accuracy and robustness. Although originally designed for offline inference, it has been increasingly adopted in live transcription systems through buffering and specialized decoding strategies [4]–[7].

Live transcription systems. Processing continuously arriving speech in real time requires balancing low latency with transcription accuracy [10]. To achieve this, existing systems repeatedly invoke Whisper on partial inputs and rely on decoding policies such as Local Agreement [11] and AlignAtt [12] to determine when partial hypotheses are stable enough to emit.

2.2 Neural Processing Units↩︎

Neural processing units (NPUs) execute models using precompiled static graphs with fixed tensor shapes, enabling high efficiency [8], [9], [13] but limiting support for dynamic workloads such as autoregressive decoding.

Static-graph execution. To exploit these hardware characteristics, NPUs typically execute models using precompiled static graphs, where tensor shapes, memory layouts, and execution schedules are fixed. This enables efficient mapping of operations onto parallel compute units, improves data reuse through on-chip SRAM buffering, and reduces costly memory movement. As a result, static-graph execution allows NPUs to achieve high utilization of compute resources while minimizing runtime overhead and energy consumption [14], [15].

Implications for live transcription. This execution model is well-suited to workloads with fixed or predictable tensor shapes, such as vision models with static input resolutions [16], [17]. In contrast, live transcription involves continuously arriving audio inputs and incremental decoding, which naturally introduce variability in input sizes and execution patterns [10]. As a result, efficiently mapping such workloads onto static-graph execution often requires structured input handling or approximation strategies that align dynamic inputs with a limited set of predefined execution configurations [18].

3 Motivation↩︎

3.1 Computational Cost of Padding and Hallucination Risk↩︎

Whisper hallucination. Whisper exhibits hallucination behavior, generating text that is not grounded in the input audio [19]–[21]. In speech transcription, this typically manifests as repeated words or phrases [5], [21]. As shown in Figure 2 (a), hallucinations become more frequent when processing short audio inputs without padding. To mitigate this, existing systems extend the input with padding toward the 30-second training window, which stabilizes decoding. Correspondingly, Figure 2 (b) shows that the word error rate (WER) and character error rate (CER) decrease as the input length approaches 30 seconds. This behavior stems from Whisper being trained on fixed 30-second audio segments [3], [5], indicating that padding primarily serves to suppress hallucinations rather than to provide additional acoustic context.

Padding overhead. While padding mitigates hallucinations and improves WER/CER, it incurs significant computational overhead, especially on resource-constrained mobile devices. As shown in Figure 2 (a), the encoder processing time increases substantially as the padded input length approaches 30 seconds, reaching around 720 ms, which is 3.8\(\times\) higher than that without padding. The smaller gap observed in Figure 2 (a) is due to the longer input duration in this experiment (approximately 9 seconds), which exceeds the shorter segments typically used in live transcription. We focus on encoding latency in this motivation study because decoding latency depends on both input length and the number of generated tokens, making it difficult to isolate the effect of padding.

Opportunity. As shown in Figure 2 (b), when hallucinated tokens generated beyond the reference transcription are removed in offline post-processing (w/o hall. in the figure), Whisper inference without padding achieves WER and CER close to those obtained with full 30-second padding. These observations suggest that hallucinations are fundamentally associated with failures in temporal grounding rather than insufficient acoustic context itself. Therefore, if temporally ungrounded tokens can be detected online during decoding, it becomes possible to eliminate padding overhead while preserving transcription accuracy. To realize this opportunity, we analyze cross-attention patterns to detect hallucinations and avoid unnecessary computation in NPUsper, detailed in § 4.3.

3.2 Dynamic Speech Processing on Static-graph NPUs↩︎

NPU execution mismatch. NPUs offer high inference throughput and energy efficiency, making them attractive for on-device live transcription systems. As shown in Figure [fig:motivation-npu](a), on the same 200 Whisper inference tasks, the NPU completes inference 3\(\times\) faster than the GPU and reduces average power consumption from 1.99 W to 1.14 W. However, using NPUs for live transcription introduces an execution mismatch: input length and autoregressive decoding are dynamic, whereas NPUs rely on static graphs with fixed shapes and control flow. While variability in input length can be handled relatively easily using bucketing [18], autoregressive decoding is more challenging to execute efficiently on NPUs. Specifically, the KV cache grows with each generated token, creating a mismatch between dynamic autoregressive decoding and static-graph execution on NPUs.

KV-cache inefficiency. To enable autoregressive decoding on NPUs, Qualcomm adopts a design that fixes the encoder input to 30 seconds and allocates the KV cache for the maximum decoded sequence length, \(M=200\) [22]. We focus on the decoding-side inefficiency, as the encoding-side overhead from input padding was discussed in § 3.1. In autoregressive decoding, the KV cache grows incrementally with the number of generated tokens, so early decoding steps require only minimal attention computation. Under Qualcomm’s design, however, the NPU must execute attention over all \(M=200\) positions at every decoding step, even when the actual sequence length is much shorter. As a result, substantial redundant computation is incurred throughout decoding.

Runtime overhead. A straightforward way to reduce KV-cache inefficiency in Qualcomm’s fixed-shape design is to compile separate step-specific graphs that match the current KV-cache length. However, this requires dispatching a different graph at each token-generation step, causing substantial runtime overhead (Figure [fig:motivation-npu](b)). Another alternative is to unroll \(n\) decoding steps into a single graph to amortize this overhead. Once dispatched, however, the graph must execute all \(n\) steps, even if EOS or a stop condition occurs earlier. This highlights the need for a balanced NPU decoding design that reduces redundant KV-cache computation, minimizes runtime overhead, and preserves early termination (§ 4.4).

4 NPUsper System Design↩︎

Figure 3 presents an overview of NPUsper. At each inference round, NPUsper constructs a short Whisper input from newly arrived audio and carry-over context (§ 4.1). The input is mapped to an NPU-compatible bucket, and decoding is executed with controlled unrolling (§ 4.4). Generated tokens are then validated using cross-attention dynamics (§ 4.2 and § 4.3). Valid tokens are emitted immediately, while hallucinated tokens stop decoding and determine the carry-over state for the next round.

4.1 Audio Buffer Management↩︎

The audio buffer manager maintains the incoming audio stream and constructs each input from carry-over audio and newly arrived audio. Unlike prior systems that rely on padding, NPUsper constructs inputs using only real audio and validated context. The input is mapped to a discrete bucket for NPU execution. For autoregressive decoding, NPUsper initializes the decoder with prefix tokens derived from the carry-over text. NPUsper validates generated tokens using cross-attention and emits only valid tokens. When hallucination is detected, decoding stops and the carry-over state is updated using dynamic time warping (DTW) [23].

4.2 Cross-attention Characteristics in Whisper↩︎

Definition of cross-attention. For the \(i\)-th generated token, we define the frame-wise cross-attention at decoder layer \(l\) as the head-averaged attention: \[a_i^{(l)}(f) = \frac{1}{H} \sum_{h=1}^{H} \mathrm{CrossAttn}_i^{(l,h)}(f), \label{eq:cross-attention}\tag{1}\] where \(H\) is the number of attention heads, \(f\) denotes the audio frame index, and \(\mathrm{CrossAttn}_i^{(l,h)}(f)\) is the softmax-normalized cross-attention weight assigned to frame \(f\) at decoder layer \(l\) and head \(h\) when generating the \(i\)-th token. \(a_i^{(l)}\) provides a token-level distribution over audio time, which we use to analyze temporal grounding.

Cross-attention analysis. Cross-attention in Whisper indicates which audio frames contribute to each generated token. We use this signal to analyze whether tokens are temporally grounded in the input audio. Figure 4 shows that content tokens exhibit localized attention that follows the temporal progression of the input audio. This property is not specific to Whisper, but arises from the alignment structure of encoder-decoder speech models [12], [24]. We hypothesize that valid tokens follow a monotonic progression over audio frames, whereas hallucinated tokens violate this temporal consistency. This suggests that cross-attention can serve as a proxy for token-level grounding. Punctuation tokens exhibit diffuse attention over the audio timeline, and subword tokens can introduce noise in temporal alignment. We therefore restrict our analysis to content tokens for reliable detection (§ 4.3).

Layer selection. The temporal structure of cross-attention also varies across decoder layers. Compared with earlier layers, the final decoder layer shows clearer temporal alignment with the audio timeline (Figure 4), making it more suitable for tracking token-level progression. We therefore use the final layer and compare cross-attention across consecutive content tokens.

4.3 Hallucination Detection↩︎

Based on observations and insights in § 4.2, we design a hallucination detection mechanism that identifies violations of temporal consistency in cross-attention patterns.

Backward cross-attention shift. NPUsper detects hallucinated tokens from cross-attention patterns in the final decoder layer. In Whisper, hallucinated tokens exhibit backward shifts in cross-attention over the audio timeline, rather than following the forward temporal progression of valid tokens.

Token validation. To operationalize this intuition, we define token-level temporal attention dynamics as follows. For each generated token \(i\), we extract the head-averaged cross-attention from the final decoder layer \(L\): \[\tilde{a}_i(f) = a_i^{(L)}(f). \label{eq:final-layer-cross-attn}\tag{2}\] Temporal attention changes are computed only between consecutive content tokens, excluding punctuation and subword tokens to ensure reliable temporal alignment. Let \(p(i)\) denote the most recent preceding content token before token \(i\). We define the frame-wise attention difference as \(\delta_i(f) = \tilde{a}_i(f) - \tilde{a}_{p(i)}(f)\). Let \(\hat{\delta}_i(f)\) denote the filtered attention (Appendix 7) difference between token \(i\) and its preceding content token. We identify the locations of the positive and negative peaks: \[f_i^{+} = \operatorname*{arg\,max}_{f} \hat{\delta}_i(f), \qquad f_i^{-} = \operatorname*{arg\,min}_{f} \hat{\delta}_i(f). \label{eq:peak-locations}\tag{3}\] Intuitively, \(f_i^{+}\) indicates the audio frame toward which attention newly concentrates, whereas \(f_i^{-}\) indicates the frame from which attention moves away. A token is considered valid if attention progresses forward along the audio timeline, i.e., \(f_i^+ \geq f_i^-\). Otherwise, we mark it as hallucinated: \[f_i^{+} < f_i^{-}. \label{eq:attention-reversal}\tag{4}\] This condition captures violations of temporal monotonicity in cross-attention, which we treat as a signature of hallucination. This depends only on the relative ordering of attention dynamics and does not require threshold tuning.

Decoding control. During decoding, NPUsper stops generation when hallucination is detected or when attention reaches the end of the current audio chunk. In both cases, valid tokens are emitted, and the carry-over state is updated using DTW [23]. This enables early termination of decoding and avoids unnecessary computation.

4.4 Controlled Unrolling for Decoding on NPUs↩︎

Controlled unrolling. Our system unrolls autoregressive decoding into \(K\)-step chunk graphs and executes them sequentially on the NPU. This design explicitly bridges the mismatch between dynamic autoregressive decoding and static-graph execution on NPUs. As shown in Figure 5 (), each chunk graph generates up to \(K\) tokens in one NPU execution and returns the cross-attention weights defined in Equation 2 . NPUsper then applies the token validation in § 4.3 to decide whether to emit tokens, stop decoding, or dispatch the next chunk graph. This enables adaptive computation that scales with the actual decoding length rather than the maximum sequence length.

Chunk-level decoding control. A \(K\)-token chunk serves as the unit of both NPU execution and decoding control. After each chunk, NPUsper applies the same decoding control conditions described in § 4.3 to decide whether to dispatch the next graph or proceed to the next inference round. Valid tokens are emitted after each chunk, rather than waiting for the entire decoding round to finish. If a hallucinated token is detected, NPUsper emits only the preceding valid tokens. This chunk-level design reduces graph dispatch overhead while preserving timely token emission and hallucination handling.

Necessary KV-cache computation. Controlled unrolling keeps only the KV-cache attention computation needed at each decoding step. Let \(M\) be the maximum decode length of the fixed full-length graph, and let \(\mathrm{MAC}(s)\) denote the number of multiply-accumulate operations for self-attention at decoding step \(s\). We use \(c\) to denote the per-position self-attention cost, including the query-key dot product and value aggregation across all decoder layers and attention heads. The fixed full-length graph computes attention over all \(M\) positions at every step, so \(\mathrm{MAC}_{\text{fixed}}(s)=cM\). In contrast, NPUsper computes attention only over the \(s\) positions needed at step \(s\), so \(\mathrm{MAC}_{\text{ours}}(s)=cs\). If decoding stops at step \(T\), the resulting self-attention MAC ratio is \[\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}}(s)}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}}(s)} = \frac{2M}{T+1}. \label{eq:mac-ratio}\tag{5}\] This reveals a fundamental inefficiency in fixed-length decoding, where computation scales with the maximum sequence length \(M\) rather than the actual decoding length \(T\). This inefficiency becomes particularly pronounced in live transcription, where decoding often terminates after only a few tokens. For example, with \(M=200\), stopping at \(T=10\) yields a \(36.4\times\) reduction in self-attention MACs, while \(T=20\) still yields a \(19\times\) reduction.

5 Results↩︎

5.1 Evaluation Setup↩︎

We implemented and evaluated NPUsper and baseline systems on two commercial mobile devices: a Samsung Galaxy S25 [8] and a Snapdragon X Plus laptop [9], hereafter referred to as S25 and X Plus, respectively. For a fair comparison, all baselines and our method run on the GPU. We additionally evaluate our method on the NPU, denoted as Ours-N, to measure the benefit of NPU execution. Appendices 9 and  10 provide implementation details.

Baselines. We compare NPUsper with four Whisper-based live transcription systems: Whisper-Streaming (WS) [4], WhisperFlow (WF) [5], SimulStreaming (SS) [7], and Simul-Whisper (SW) [6]. They differ in Whisper input construction, decoder context management, and decoding policy. WS, SS, and SW feed Whisper with 30-second sliding-window inputs, whereas WF reduces the input to roughly 15 seconds using a trained hush word. For decoding, WS and WF use Local Agreement [11], which repeatedly confirms partial hypotheses before emission, while SS and SW use AlignAtt [12] to stop decoding based on cross-attention near chunk boundaries. For decoder context, SS retains tokens from dropped audio segments, whereas SW discards them and uses a trained integrate-and-fire (IF) module to detect word truncation [25].

Dataset. We evaluate on two long-form datasets, TED-LIUM 3 [26] and Meanwhile [3]. Because each end-to-end system evaluation runs in real time with the input audio, the main results use two representative TED-LIUM 3 talks, EricMead (7:39 min) and GaryFlake (5:45 min). Additional results on the remaining TED-LIUM 3 talks and Meanwhile dataset are available in Appendix 8.

Metrics. Per-word latency (\(\downarrow\)) is the average latency between when a word is spoken and when it appears as text to the user [4], [5]. Word error rate (WER, \(\downarrow\)) is the fraction of incorrectly transcribed words relative to the reference transcript [3], [27]. Time-to-first-token (TTFT, \(\downarrow\)) is the delay from the start of inference to the first text output visible to the user [28].

5.2 Latency and Inference Efficiency↩︎

Low-latency transcription. As shown in Figure 6, NPUsper achieves the lowest per-word latency with comparable transcription accuracy, with 1.71–4.62\(\times\) and 1.69–4.84\(\times\) lower per-word latency than the baselines on S25 and X Plus, respectively. This gain comes from eliminating redundant computation in padded and repeated Whisper inputs, which reduces inference time, and from using AlignAtt [12] to emit tokens from a single inference.

Reduced inference cost. Figure 7 shows that NPUsper runs Whisper inference 5.52–13.09\(\times\) and 7.92–19.10\(\times\) faster than the baselines on S25 and X Plus. NPUsper achieves this by avoiding unnecessary padding and processing only newly arrived audio with minimal carryover, resulting in shorter Whisper inputs and lower encoder latency. NPUsper also reduces prefill latency by using only the last valid word as the decoder prefix.

Fast transcript updates. By avoiding redundant computation during each inference, NPUsper also reduces the time required to produce the first visible transcript update. On S25, NPUsper reduces TTFT by 2.2–12.8\(\times\) compared with the baselines, while Ours-N further improves this to 3.9–22.8\(\times\). On X Plus, NPUsper reduces TTFT by 2.9–14.2\(\times\), and Ours-N achieves a 6.9–33.2\(\times\) reduction. This improvement comes from jointly reducing computation cost, hypothesis recomputation, and unnecessary prefill computation.

5.3 Energy and Decoding Efficiency↩︎

We measure power on the S25 using a Monsoon High Voltage Power Monitor [29] and on the Snapdragon X Plus using Qualcomm Profiler [30], reporting power after subtracting idle power. (Detailed in Appendix 8.2)

Energy efficiency. Table 8 shows that NPUsper achieves the best energy efficiency on both devices. On S25, Ours-N reduces average power to 0.46 W and energy per inference to 0.92 J, achieving 6.3–8.8\(\times\) lower power than the baselines. On X Plus, Ours-N also achieves the lowest power, energy per inference, and energy per token. This gain comes from eliminating redundant Whisper computation at the system level, while NPU execution provides additional energy savings. Additional results are provided in Appendix 8.2.

Token-metric analysis. The energy-per-token and throughput metrics should be interpreted with each decoding policy in mind. WS and WF sometimes show competitive energy per token or throughput, not because they are more efficient, but because Local Agreement [11] regenerates many hypothesis tokens within each inference. Moreover, WS and WF failed to process the entire audio (see truncated power traces in Figure 10) because long Whisper inputs and repeated hypothesis recomputation increase per-iteration latency. Slow iterations delay longest-prefix confirmation in Local Agreement, causing the audio buffer to grow beyond what the systems can process in real time. By contrast, SS, SW, and NPUsper all use AlignAtt decoding [12], making their token-generation metrics more directly comparable. NPUsper nevertheless achieves the best energy-per-token and throughput overall.

Decoding efficiency. Figure [fig:decoding-breakdown] compares three NPU decoding strategies on S25. Qualcomm’s fixed full-length graph incurs the highest latency due to excessive KV-cache computation. Using separate graphs removes this redundant KV-cache work but still suffers from per-token graph dispatch overhead. NPUsper reduces both costs by computing only the required KV-cache attention and reducing graph dispatch frequency through controlled unrolling. (Detailed in Appendix 8.3 for additional results.)

6 Conclusion and Limitations↩︎

We propose NPUsper, an on-device live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation on both audio processing and autoregressive decoding. NPUsper removes the need for long padded Whisper inputs by detecting hallucinated tokens online from cross-attention dynamics, allowing each inference round to process only newly arrived audio with minimal carry-over context. For NPU execution, NPUsper introduces controlled unrolling, which reduces redundant KV-cache attention computation and graph dispatch overhead while enabling chunk-level token emission and decoding control. NPUsper achieves the best per-word latency, inference speed, and energy efficiency among the evaluated live transcription systems.

Limitations. This paper focuses on system-level efficiency rather than modifying the Whisper architecture itself. Prior work [31] converts Whisper’s encoder into a causal encoder and applies low-rank adaptation [32], but the resulting model is limited to the dataset used for adaptation. We focus on building and evaluating a full end-to-end on-device system with the Whisper base model under mobile memory and compute constraints. However, our hallucination detection method is not limited to the base model, as it relies on temporal alignment in Whisper’s decoder cross-attention. Appendix 7.4 demonstrates its generalizability across different Whisper model sizes. We leave the design of efficient on-device systems using large speech models such as Canary [33] and SeamlessM4T [34] for future work.

7 Hallucination Detection Analysis↩︎

7.1 Filtered Attention-Difference Signal↩︎

Filtered attention. For each generated token \(i\), we compare its final-layer cross-attention with that of the most recent preceding content token \(p(i)\). The raw attention-difference signal is defined as \[\delta_i(f) = \tilde{a}_i(f) - \tilde{a}_{p(i)}(f), \label{eq:attention-delta}\tag{6}\] where \(f\) denotes the audio frame index. To reduce spurious spikes near the beginning and end of the audio as shown in Figure 4, we apply a median filter followed by a moving-average filter: \[\hat{\delta}_i(f) = \mathrm{MA}_{w_a} \left( \mathrm{Med}_{w_m}(\delta_i) \right)(f), \label{eq:filtered-attention-delta}\tag{7}\] where \(\mathrm{Med}_{w_m}\) denotes a median filter with window size \(w_m\), and \(\mathrm{MA}_{w_a}\) denotes a moving-average filter with window size \(w_a\). We use the filtered signal \(\hat{\delta}_i(f)\) to locate the positive and negative peaks used for hallucination detection.

7.2 Component Ablation Study↩︎

Effect of token-validation components. Table 1 analyzes the effect of each component in our token validation method using 250 samples from the LibriSpeech dataset [35]. Without padding, Whisper produces a high hallucination rate. Backward-shift detection substantially reduces hallucinations, while smoothing and median filtering reduce undesired backward-shift detections. Skipping punctuation and subword tokens further improves robustness by avoiding unreliable temporal anchors. The full configuration eliminates hallucinations in this study and achieves the lowest undesired backward-shift detection rate. In NPUsper, even if a rare undesired backward shift stops decoding early, the remaining audio is retained as carryover and processed together with newly arrived audio in the next inference round.

3pt

Ablation visualization. Figure 9 provides a complementary view of the same ablation. The stacked bars (Figure 9 ( )) show that each component progressively reduces false or undesired detections while preserving the ability to catch hallucinated tokens. The precision–recall plot (Figure 9 ( )) further shows that token filtering improves detection precision while maintaining high recall, indicating that punctuation and subword tokens are unreliable temporal anchors for hallucination detection.

7.3 Filter Parameter Selection↩︎

Parameter selection rationale. We choose the moving-average and median-filter window sizes in two steps. Table 2 reports representative configurations used for this selection. First, without median filtering, we vary the moving-average window and find that \(w_a=10\) reduces the hallucination rate among the tested smoothing settings. Second, with \(w_a=10\) fixed, we add a median filter to remove spurious attention peaks in the attention-difference signal, which are visible in Figure 4. A median-filter width of \(w_m=7\) removes most of these peaks and reduces undesired backward-shift detections while maintaining a low hallucination rate. We therefore use \((w_a, w_m)=(10,7)\) in our implementation. Compared with this parameter selection, the larger improvement comes from the backward-shift criterion and proper handling of punctuation and subword tokens, as shown in Table 1 and Figure 9.

5pt

7.4 Model Selection and Generalizability↩︎

Model selection. Our end-to-end system evaluation focuses on the Whisper base model. This choice reflects the memory and compute constraints of mobile devices, where the goal is to build a complete real-time on-device transcription system rather than to maximize transcription accuracy with a larger model. We exclude Whisper tiny because its transcription quality is substantially weaker and it is rarely used as the main model in recent Whisper-based streaming systems [5]. We also exclude Whisper large from this analysis in this section because it caused an out-of-memory error even on the Galaxy S25, which is a high-end mobile device.

Cross-model evaluation. Although our end-to-end system uses Whisper base, the proposed hallucination detection method (§ 4.3) is not specific to the base model. Its generality comes from relying on temporal alignment in decoder cross-attention, which is consistently observed across different Whisper model sizes. To evaluate this property, we reuse the configuration chosen for Whisper base on Whisper small and medium, including the same filtering and token-skipping parameters. For each model size, we compare a no-detection baseline against our method.

4pt

Generalizability. Table 3 shows that hallucination detection remains effective beyond Whisper base. On Whisper small, our method reduces the hallucination rate from 24.0% to 2.0% for both selected layers. Undesired backward-shift remains modest at 5.2% and 3.2%, respectively. On Whisper medium, our method reduces the hallucination rate from 36.4% to 6.8%, with 6.9% undesired backward-shift. These results indicate that temporal alignment in decoder cross-attention provides a general signal for detecting hallucinated tokens across different Whisper model sizes.

Deployment feasibility. Although our hallucination detection method remains effective, Whisper medium is less practical for real-time deployment because its encoder alone takes more than 4 s on the Snapdragon X Plus laptop, which exceeds the short inference interval in live transcription systems. This makes Whisper medium difficult to use as a full real-time on-device system, despite its potential for higher transcription accuracy. We leave the construction of fully optimized end-to-end systems for larger Whisper model sizes to future work, including model-size-specific hyperparameter tuning for hallucination detection.

8 Additional Results↩︎

8.1 Transcription Quality↩︎

Word and character errors. Table [tab:additional-system-wer-cer] reports additional transcription quality results. All systems are evaluated with the Whisper base model, and WER/CER are computed per sample and reported as mean\(\pm\)standard deviation. Full WER/CER compare each system output against the complete reference transcript. For Whisper-Streaming and WhisperFlow, some runs terminate before processing the full audio, which can inflate full-reference WER/CER due to incomplete coverage. We therefore additionally report truncated WER/CER for these cases, comparing each output only against the reached reference prefix.

8.2 Power Measurement↩︎

Measurement setup. We measured full-system battery-rail power on the Galaxy S25 using a Monsoon High Voltage Power Monitor [29], configured as an external supply for the phone battery rail [36]. The S25 traces used for Table 8 were captured with \(V_{\mathrm{out}}=4.42\) V. The Monsoon capture process starts before each live-transcription run and samples the main rail current and voltage at \(f_s=5000\) Hz. The workload script writes start and end marker files immediately before and after the transcription command; all S25 power statistics are computed over the Monsoon samples whose timestamps fall inside this marker-delimited command window. This measures the complete phone system during the workload, not an isolated GPU or NPU rail.

Power and energy computation. For each command-window sample \(i\), we compute instantaneous power as \[p_i = I_i V_i, \quad \bar{P}_{\mathrm{raw}} = \frac{1}{M}\sum_{i=1}^{M} p_i, \quad E_{\mathrm{raw}} = \frac{1}{f_s}\sum_{i=1}^{M} p_i, \label{eq:s25-power-integration}\tag{8}\] where \(I_i\) is in mA, \(V_i\) is in V, \(p_i\) is in mW, and \(M\) is the number of samples in the command window. Thus, S25 energy is obtained by discrete integration over Monsoon samples rather than by manually multiplying a reported average by wall-clock time. We separately measure an idle baseline with the same Monsoon setup and subtract its command-window mean power, \(P_{\mathrm{idle}}=362.2\) mW, from each S25 workload: \[P_{\mathrm{work}} = \bar{P}_{\mathrm{raw}} - P_{\mathrm{idle}}, \quad E_{\mathrm{work}} = E_{\mathrm{raw}} - P_{\mathrm{idle}}\frac{M}{f_s}. \label{eq:s25-idle-subtraction}\tag{9}\]

Reported metrics. Table 8 reports \(P_{\mathrm{work}}\) and energy metrics derived from \(E_{\mathrm{work}}\), converted to W and J. Energy per inference is \(E_{\mathrm{work}}/N\), where \(N\) is the number of inference iterations in the run logs. Energy per token divides \(E_{\mathrm{work}}\) by the total number of generated tokens, and throughput is computed as generated tokens per active inference time, where active time is the sum of mel, encoder, decoder-prefill, and decoder execution time.

Snapdragon X Plus measurement. For the Snapdragon X Plus laptop, we measured power using Qualcomm Profiler [30] and applied the same base-power subtraction principle to the profiler traces. Because the S25 and X Plus use different power measurement interfaces, their traces have different noise characteristics. We therefore use the traces in Figure 10 to show qualitative results and report the averaged values in Table 8 for quantitative comparison. The numerical values in Table 8 are computed from the raw command-window summaries.

Power traces. Figure 10 shows the power traces used for Table 8. NPUsper consistently maintains much lower power than the baselines because it avoids long padded inputs and repeated hypothesis recomputation. All systems show an initial power peak when transcription starts, reflecting the startup cost of launching the live transcription pipeline. The WS and WF traces on both devices are truncated because these systems terminated before processing the entire audio. This behavior is caused by high per-iteration latency from long Whisper inputs and hypothesis recomputation by Local Agreement.

8.3 Controlled Unrolling on NPUs↩︎

Cross-device consistency. In addition to the S25 results in Figure [fig:decoding-breakdown], we provide supplementary X Plus results in Figure 11. The X Plus results show the same overall trend as S25: the fixed full-length graph has the highest decoding overhead, the separate-graph approach reduces redundant KV-cache computation but still incurs graph-dispatch overhead, and our controlled-unrolling design achieves the lowest overall decoding overhead.

Across decoding lengths. We also evaluate whether this trend remains consistent as the number of generated tokens changes. Figure [fig:decoding-breakdown-s25-token-counts] reports the S25 decoding breakdown for shorter, medium, and longer decoding cases. Across all token-count settings, the relative behavior of the three strategies remains unchanged. The fixed full-length graph remains the least efficient, the separate-graph approach improves over it but retains dispatch overhead, and our method consistently provides the lowest decoding overhead.

8.4 Supplementary CPU-only System Comparison↩︎

CPU-only setup. Table 4 reports a supplementary CPU-only system comparison on Galaxy S25 and Snapdragon X Plus. All systems use the same 2 s inference interval and run without GPU acceleration. This experiment is not intended as our main comparison setting. Instead, it characterizes how each system behaves when real-time streaming must be sustained using CPU execution alone.

2pt

Mobile CPU bottlenecks. The CPU-only results highlight why removing unnecessary computation is important for on-device streaming inference. On the S25 CPU, systems that repeatedly use a fixed 30s encoder input incur substantial encoder overhead: WS, SS, and SW spend 2141.7 ms, 2832.6 ms, and 2814.1 ms, respectively, on encoder execution alone. These costs already exceed the 2 s inference interval before accounting for preprocessing and decoding. SS further spends 1142.2 ms in the decoder, which is the largest decoder cost among the S25 runs, mainly because it performs the most prefill-token computation. As a result, SS completes but falls far behind the real-time latency performance, while SW fails to produce a usable transcript.

Effect of removing redundant computation. NPUsper avoids much of this overhead by using online hallucination detection to remove unnecessary reprocessing for mobile execution. Its average encoder input length is only 3.67s on S25, reducing encoder execution to 161.6 ms and decoder execution to 37.1 ms. This allows NPUsper to complete each CPU-only streaming update within the inference interval even on the mobile CPU. The X Plus results show the same trend under a stronger CPU: all systems become less constrained, but NPUsper still uses substantially shorter encoder inputs and lower encoder/decoder latency than the baselines.

Scope. Note that these results are not intended as the main system comparison, but as a supplementary evaluation showing that on-device AI systems are highly sensitive to redundant computation. Eliminating those redundant computations for encoder and decoder is essential for sustaining real-time streaming on mobile processors. Jointly pipelining the CPU, GPU, and NPU remains an interesting direction for future work.

9 GPU Implementation Details↩︎

Implementation and GPU setup. All GPU implementations were built on whisper.cpp [37], with GGML used as the tensor computation backend. For GPU-accelerated execution, we used the GGML OpenCL backend on Qualcomm Adreno GPUs. Experiments were conducted on two devices: a Samsung Galaxy S25 [8] equipped with a Qualcomm Adreno 830 GPU, and an ASUS Vivobook S 15 [9] powered by Snapdragon X Plus with a Qualcomm Adreno X1-45 GPU. Both platforms used OpenCL 3.0 and were compiled in release mode with Adreno-optimized OpenCL kernels enabled. Unless otherwise stated, all experiments used the ggml-base.bin Whisper model, and streaming inference was performed with a 2 s inference round interval. The motivation experiments in § 3 were conducted on the Snapdragon X Plus using 200 samples from the LibriSpeech dataset [35].

10 NPU Implementation Details↩︎

10.1 Graph Construction↩︎

ONNX/QNN setup. For NPU execution, we use ONNX graphs executed with the ONNX Runtime QNN execution provider and the QAIRT/QNN HTP backend. The NPU implementation follows the QAI Hub Whisper graph construction: transformer linear layers are represented as 1\(\times\)​1 convolution operators for QNN execution, while core Whisper operations such as attention, layer normalization, GELU, and encoder front-end convolutions are preserved.

Lowering effect. This lowering improves S25 NPU latency under the same runtime stack, so we use the lowered QNN graphs for our NPU implementation.

10.2 Input and Model Bucketing↩︎

Bucket selection. The NPU runtime uses input and model bucketing to connect variable-length streaming audio with the static-shape execution model of Qualcomm QNN. Rather than compiling a separate graph for every possible window length or padding every input to Whisper’s 30s context, NPUsper selects the smallest supported bucket that can contain the current effective audio window. Given an effective audio duration \(d\), the selected bucket is \[\mathcal{B}=\{3,4,5,6,30\}\;\text{s}, \qquad b(d)=\min\{\,\beta \in \mathcal{B} \mid d \le \beta\,\}.\] The 3–6s buckets are sufficient for most audio-buffer lengths in our experiments, while the 30s graph is kept as a rare fallback for unstable segments that require reinference with the native Whisper context.

10.3 Controlled Unrolling Decoder Graphs↩︎

Chunked decoder execution. The decoder is executed as a sequence of small static graphs rather than as one graph that always runs the full decoding loop. This keeps each NPU graph shape-specialized while allowing the runtime to stop early when no further tokens are needed. For each bucket, the decoder supports up to 30 output tokens. After each chunk graph, the system emits valid tokens and dispatches the next graph only if decoding must continue.

Chunk schedule and prompt prefill. For each bucket, the decoder is implemented as a controlled-unrolling chain with a maximum decode length of 30 tokens. The default schedule is [4,6,5,5,5,5], where the first chunk handles Whisper’s four special tokens. When carryover text is used as a decoder prompt, we use the prefill schedule [6,4,5,5,5,5] so that prompt and special tokens are handled within the first two chunks. Since both schedules cover 10 positions before the later chunks, the remaining [5,5,5,5] graphs are shared across the default and prefill paths. Each chunk returns logits, cross-attention maps, and updated self-KV cache tensors, and chunk wrappers within each bucket are linked into a weight-shared QNN context binary to minimize decoder weights.

Chunk-size selection. We select the repeated later chunk size while keeping the first two chunks fixed as [4,6] for the default path and [6,4] for the prompt-prefill path. We compare later chunk sizes of 3, 5, and 7, corresponding to default schedules of [4,6,3,3,3,3,3,3], [4,6,5,5,5,5], and [4,6,7,7,7], respectively. This comparison captures the trade-off between graph-dispatch overhead and unused decoder computation: smaller chunks require more graph launches, while larger chunks execute more decoder positions than may be needed.

Selected chunk size. Table 5 shows that 3-token chunks incur excessive decoder calls and graph switches, while 7-token chunks perform more unnecessary decoder-position computation. The 5-token setting best balances graph-dispatch overhead and unused decoder work, achieving the lowest per-word latency. We therefore use [4,6,5,5,5,5] for the default path and [6,4,5,5,5,5] for the prompt-prefill path.

10.4 Cross-Attention Masking↩︎

Padded source positions. In the bucketed NPU runtime, the number of encoder frames attended by the decoder is fixed by the selected bucket rather than by the exact audio duration. When the streaming window is shorter than the bucket, the encoder output contains real audio positions followed by padded positions introduced only to match the static bucket shape. If left unmasked, these padded positions would still participate in the cross-attention softmax and could change the attention distribution over real audio frames.

Masking padded positions. Each decoder chunk receives a cross-attention mask constructed from the non-padded audio length recorded during bucket selection. The mask leaves the columns corresponding to real audio unchanged and adds a finite negative bias of \(-100\) to the padded columns before the softmax. We use a finite value instead of infinity to avoid numerical issues under reduced-precision NPU execution. The final-layer cross-attention maps returned to the host are also cropped to the real-audio region before hallucination detection and DTW-based carryover updates. As a result, the compiled NPU graph keeps a fixed bucket shape, while token validation and streaming decisions are computed only from real audio.

Mask-removal case study. Figure 12 shows the token-level effect of removing the cross-attention mask on the LJ-Speech utterance LJ001-0021 [38]. With the same encoder input, decoder weights, and decoding state, the unmasked decoder leaves the masked top-1 trajectory after step 10. At the first failure point, the masked token is absent from the unmasked top-3 candidates; in the following steps, it is demoted below an incorrect unmasked top-1 token.

10.5 Self-Attention Cache Handling↩︎

Static cache interface. Controlled unrolling executes multiple autoregressive decoding steps inside one NPU graph, but the first chunk would normally start with an empty self-KV cache. This zero-length cache is inconvenient for QNN compilation and would make the first chunk graph a special case. We avoid this by reserving one leading dummy KV slot, so every chunk starts with a non-empty cache and follows the same read–append–return cache interface.

Dummy-slot masking. The dummy slot is only a structural placeholder. It is masked as an invalid self-attention source position, and the host never treats it as a generated token. Thus, the cache interface remains static for NPU execution while token emission, EOS checks, hallucination detection, and prompt carryover operate only on real generated tokens.

10.6 Software Stack↩︎

NPU runtime stack. We implemented the bucketed NPU runtime using ONNX Runtime with Qualcomm’s QNN HTP backend. Encoder and decoder graphs were exported to ONNX and compiled offline with Qualcomm AI Runtime SDK (QAIRT) 2.37.1.250807. The same runtime, NPU backend, and offline compiler stack were used across both target devices. The target-specific details are the application build environments: Visual Studio 2022 ARM64/MSVC for the Snapdragon X Plus laptop and Android NDK r26d with clang 17 for the Galaxy S25.

5pt

References↩︎

---

1. NPUsper combines “NPU” and “Whisper,” with “NP” also referring to no padding to avoid padding overhead.↩︎