1 Introduction↩︎

Vision-language models (VLMs) are crossing a critical threshold: devices already in consumers’ hands—with billion-parameter multimodal models—can now run VLM inference entirely on-device, with no cloud round-trip. Wearables, AR overlays, robotics, and privacy-sensitive pipelines all demand exactly this capability [1]–[3]. The Qualcomm SM8750 (Snapdragon 8 Elite) exposes a stable Hexagon NPU path for full multimodal models via the QNN SDK [4], [5], making it the right deployment platform to ask: what does NPU acceleration concretely deliver for each phase of VLM inference, and how should practitioners configure heterogeneous pipelines?

This question is non-trivial: VLM inference decomposes into three stages with fundamentally different hardware affinities [6]. Prefill saturates NPU tensor engines with batched GEMMs. Decode is bandwidth-bound and shape-dynamic, favouring CPU cache locality [5], [7]. Vision encoding is static-shape and compute-dense—the NPU’s clearest sweet spot, yet the least systematically studied. Treating these phases uniformly leads to measurable waste in latency, energy, and thermal budget. Prior systems work on heterogeneous mobile inference targets text-only LLMs [6], [7]; the vision encoding phase, its thermal behavior, and practical porting paths for unsupported encoders are not covered. We address this gap with a VLM-specific, hardware-in-the-loop study on SM8750, contributing phase-aware deployment guidance and a practical QNN porting workflow for production phone NPUs.

Contributions:

• Phase map: 1.64\(\times\) NPU prefill, 1.18\(\times\) decode, 6.41\(\times\) cold-start overhead (Sec. 3).

• Thermal suite: 10.47\(^\circ\)C cooler, 2.52\(\times\) lower energy, zero throttling on NPU over 100 runs (Sec. 4).

• Pipeline map: Five CPU\(\leftrightarrow\)NPU configurations; W4A8 hybrid at 97.5 tok/s (Sec. 5).

• Latency knob: Token-budget sweep 0.36–0.85 s TTFS, no recompilation (Sec. 5).

• Encoder scaling: Four families \(\times\) CPU/GPU/NPU, 20–45\(\times\) CPU-to-NPU speedups (Sec. 6).

• Porting methodology: Four-step rewrite, 22\(\times\) on Phi-3.5-V (Sec. 7).

2 Platform and Experimental Setup↩︎

2.1 Hardware: Qualcomm SM8750 Snapdragon 8 Elite↩︎

Our target is the Qualcomm SM8750 [8], a 3 nm SoC with a 64-bit Oryon CPU (4.32 GHz), LPDDR5x at 5300 MHz, Adreno GPU, and a Hexagon NPU with scalar/vector/tensor engines and VTCM scratch [5]. W4A8 INT4-weight packing halves VTCM bandwidth demand vs.INT8 [9], [10]. Static-shape NPU paths use QNN SDK [4] via ONNX; dynamic delegation uses LiteRT [11].

2.2 VLM Phase Decomposition↩︎

Following the roofline analysis framework [12], VLM inference separates into three hardware-distinct phases (Table 1). Prefill has high operational intensity \(I = \text{FLOPs}/\text{Byte}\) due to batched GEMM and ViT attention over full image and prompt—NPU systolic engines achieve peak utilization here [5], [6]. Decode processes one token per step while reading a growing KV cache: per-layer traffic scales as \(2LDb\) bytes, making it bandwidth-bound. The NPU delivers 1.18\(\times\) decode throughput; heterogeneous pipelines assign decode to CPU to maximize energy efficiency and preserve NPU capacity for the compute-intensive prefill phase [7]. Vision encoding is entirely static-shape and compute-dense, yielding the highest NPU efficiency of any VLM stage [1], [13], [14].

4pt

2.3 Models, Runtimes, and Protocol↩︎

We use FastVLM-0.5B [1] as the primary model: a 0.5B-parameter VLM built on Qwen-0.5B [15] with a FastViT encoder that applies aggressive token compression for mobile execution. For encoder scaling, we additionally benchmark ViT-B/16 [16], Phi-3.5-V [17], NanoVLM [18], and MobileNetV5 Gemma3n [19]. The CPU path uses LiteRT [11] with XNNPACK FP16. The NPU path uses LiteRT with Qualcomm AI Engine delegate [4]. The W4A8 quantized path uses QAIRT [4] INT4 weight, INT8 activation containerization for the LLM decode stage. All measurements use a fixed 64-tok text prompt, 336\(\times\)​336 image, 128-tok output; 30 runs (100 for thermal); mean \(\pm\) std reported. Device: AC power, 23\(^\circ\)C, CPU governor schedutil, NPU at Sustained Performance mode. Temperature: 1 s-sampled from NPU/CPU cluster thermal zones (/sys/class/thermal/); energy: fuel-gauge I\(\times\)V at 100 ms granularity, capturing total SoC inference draw.

Cache states [20], [21]—S1 (cold-start): process killed, drop_caches=3, QNN context cache deleted; first call reloads model and recompiles QNN graph (Table 4 cold). S2 (cold-cache): process restarted; model hot in page cache; QNN context cache intact; executor init on first call. S3 (warm): process alive, QNN context in VTCM, dispatch only. Table 2 reports S2 executor-init overhead; Table 4 reports full S1 end-to-end pipeline latency—each measuring a distinct, well-defined deployment scenario.

3 End-to-End NPU vs.CPU Throughput↩︎

3.1 Phase-Level Results↩︎

Table 2 reports throughput under warm-cache conditions. NPU delivers 1.64\(\times\) prefill throughput (197.43 vs. tok/s) and 1.18\(\times\) decode throughput (113.06 vs. tok/s). The asymmetry reflects fundamentally different operational intensity: prefill’s large batched GEMMs saturate the Hexagon tensor engine, while decode’s per-step KV-cache reads are memory-bandwidth-limited, with VTCM saturation increasing at longer contexts [5].

3pt

3.2 Cold-Start as a First-Class Deployment Dimension↩︎

The NPU executor-init overhead (536 ms vs. ms CPU, S2 state) is a design dimension steady-state benchmarks conceal. Persistent worker processes and model-cache pinning [20], [21] keep the NPU executor warm, amortizing this cost and recovering a net 1.4–1.6\(\times\) throughput advantage [11]. Warm-state management is therefore a primary deployment objective, not an optimization detail.

3.3 Qualitative Output Consistency↩︎

Table 3 confirms that the NPU backend produces semantically equivalent captions to CPU. Both backends recover identical semantic anchors (autumn park, fluffy cat, wooden bench); differences are stylistic only—NPU includes eye-color and posture details, CPU emphasizes fur texture—confirming functional multimodal equivalence under accelerator execution.

3pt

4 Thermal Characterization↩︎

4.1 100-Run Stability Suite↩︎

Fig. 2 shows SoC temperature during sustained back-to-back inference across a 100-run stability suite. The NPU path stabilizes at a mean of 40.78\(\pm\)​3.03\(^\circ\)C (peak 44.20\(^\circ\)C), while the CPU path reaches 51.25\(\pm\)​3.62\(^\circ\)C (peak 55.30\(^\circ\)C)—a 10.47\(^\circ\)C average gap with 5\(\times\) lower variance. Under deliberate thermal stress (15 minutes of continuous VLM inference), CPU paths reach 82\(^\circ\)C and trigger governor-driven clock reduction, while NPU paths stabilize at 66\(^\circ\)C and remain below the thermal throttle trip point (85\(^\circ\)C skin per Qualcomm thermal HAL configuration).

4.2 Implications for Always-On Edge Deployment↩︎

Mobile SoC governors reduce clock non-linearly past the thermal design point, inflating CPU latency by 1.3–2.1\(\times\) under sustained load—invisible in short-burst benchmarks [2], [3]. CPU paths show three latency outliers (\(Z>2.0\), at runs 20, 53, 76) from governor scaling; NPU paths show zero. The NPU’s 2.52\(\times\) lower energy per request (fuel-gauge measured) directly extends always-on session duration [5].

5 Heterogeneous Pipeline Configurations↩︎

5.1 Five-Configuration Study↩︎

A VLM pipeline on a heterogeneous SoC is not binary—it can assign encoder, prefill, and decode to different backends independently. Table 4 and Fig. 3 report end-to-end latency for five configurations under cold- and warm-cache conditions.

Under warm-cache conditions, NPU-only (2,080 ms) and Hybrid-full (2,190 ms) deliver the lowest latency. Under cold-cache, CPU-only (4,180 ms) is the fastest because warm-state transitions on the NPU amortize over subsequent requests. Hybrid-PD—offloading only the prefill stage to the NPU while the encoder and decode remain on CPU—achieves 2,640 ms warm with a manageable 6,240 ms cold, offering the best cold/warm balance for latency-sensitive deployments [20].

5.2 Visual Token Budget as a Latency Knob↩︎

In the Hybrid configuration (vision encoder on NPU, language model on CPU), the image_tokens_after parameter provides a continuous latency control knob without any model re-export. Sweeping from 24 to 224 tokens, TTFS scales linearly from 0.361 s to 0.850 s while decode throughput remains stable at 95–100 tok/s (Fig. 4). The W4A8-quantized decode path [9] achieves 97.5 tok/s at a TTFS of 0.497 s—a practical operating point for streaming VQA workloads. This knob enables adaptive quality-latency tradeoffs at runtime without model recompilation.

6 Vision Encoder Scaling Across Backends↩︎

6.1 Three-Backend Benchmark↩︎

Table 5 and Fig. 5 report standalone encoder latency for four families across CPU/GPU/NPU on SM8750. FastVLM-0.5B full-pipeline characterization is the subject of Sections 3–5; the standalone encoder benchmark here isolates module-level scaling across four additional families. CPU-to-NPU speedups: 20\(\times\) (MobileNetV5) to 45\(\times\) (ViT-B/16, NanoVLM); GPU-to-NPU: 1.7–6.9\(\times\).

2pt

6.2 Architectural and Deployment Insights↩︎

Three patterns emerge across the encoder benchmark.

Encoder choice is the dominant per-query cost lever. NPU encoder latency spans 14.3 ms (NanoVLM) to 103.5 ms (Phi-3.5-V CLIP) across the four benchmarked families—a 7\(\times\) range within the same backend, larger than any single backend-switch gain. Encoder architecture selection is therefore a higher-leverage design decision than backend selection for latency-constrained mobile VLMs [2], [3].

GPU is not a competitive alternative to NPU for vision encoding. Across all four benchmarked families, GPU trails NPU by 1.7–6.9\(\times\). MLC-LLM [22] and GPU-first frameworks therefore impose a significant encoder latency penalty on Snapdragon relative to QNN-compiled NPU paths.

INT8 NPU quantization preserves output fidelity. We evaluate FastVLM-0.5B FP16 (CPU) vs.INT8 (NPU) using greedy decoding on 500 COCO val2017 images (CIDEr via pycocoevalcap) and 200 VQAv2 open-ended questions (exact-match accuracy after lowercase normalization). CIDEr drops by 0.4 points and VQAv2 accuracy by 0.3%—within the variance of image-quality variation at mobile resolution, confirming deployment-grade equivalence.

7 Inference Code Rewrite for Operator Support↩︎

7.1 The Porting Challenge↩︎

Porting modern VLM encoders to QNN’s Hexagon backend requires resolving three structural adaptations between encoder architectures and the static, integer-typed execution model of mobile NPU compilers [4], [23], [24].

Dynamic shapes. ViT-based encoders use symbolic batch dimensions and resolution-dependent reshapes during tokenization. NPU compilers require fully static tensor shapes at AOT compile time; exporting at a fixed resolution (e.g., \(336{\times}336\), patch size 14, yielding 576 visual tokens) and replacing symbolic dimensions with concrete constants resolves this class of shapes for full NPU dispatch.

Non-standard attention. Flash Attention [25] and SDPA variants use memory-layout fusions outside Hexagon’s microkernel set. Rewriting to explicit Q/K/V matmul sequences before ONNX export yields graphs the QNN compiler re-fuses into Hexagon-native kernels.

Precision boundaries. Projection heads that accumulate in FP32 while the encoder body operates in INT8 require explicit Cast nodes inserted at each precision boundary via onnx-rewriter before conversion. Once made explicit, these boundaries become well-typed compiler barriers rather than implementation assumptions.

7.2 Methodology and Outcomes on Phi-3.5-V↩︎

We applied this methodology to Phi-3.5-V’s [17] CLIP-based encoder. Residual unsupported operators (custom rotary positional embedding and GatherElements nodes) were handled via QNN’s partitioned-model API, which executes them on CPU while keeping 96.8% of FLOPs on Hexagon. Table 6 summarizes the progression. The adapted encoder runs at 103.5 ms versus 2,286.1 ms CPU-only: a 22.1\(\times\) speedup that closes the gap between architecturally complex encoders and natively efficient ones like ViT-B/16.

4pt

This methodology generalizes: the same four steps apply to any encoder whose operators lie within the ViT family with standard transformer backbones [13], [14], [16]. For newer architectures with GroupQueryAttention [26] or RoPE [27] variants, the attention decomposition step requires additional specialization, but the partitioned fallback mechanism provides a practical safety net that keeps the bulk of compute on the NPU.

8 Discussion↩︎

Phase disaggregation is essential. The 1.64\(\times\) prefill vs.\(\times\) decode gap is fundamental [6]: single end-to-end numbers hide backend placement opportunities [7].

Encoder architecture dominates backend selection. NPU encoder latency spans 14.3–103.5 ms across four families—a 7\(\times\) range larger than any backend switch gain. Treat the vision backbone as an independently profiled, swappable component [2], [3].

Thermal and cold-start are deployment-class effects. Both determine battery life and perceived latency in always-on scenarios [20], [21]. The 2.52\(\times\) NPU energy advantage directly extends inference sessions.

Token budget is a zero-cost latency knob. The 0.36–0.85 s TTFS sweep needs no recompilation—a practical quality-latency handle under thermal pressure, demonstrated on FastVLM-0.5B on SM8750.

9 Related Work↩︎

Mobile VLM efficiency. FastVLM [1] proposes efficient image tokenization via FastViT for mobile deployment. MobileVLM [2] and LiteVLM [3] similarly reduce VLM cost for resource-constrained inference. Qwen2-VL [28] and Phi-3.5-V [17] target capable edge deployment with efficient architectures. These model-level optimizations are orthogonal to—and complementary with—our deployment characterization.

Heterogeneous on-device inference. LLM.npu [6] and HeteroInfer [7] characterize heterogeneous LLM inference on mobile SoCs, showing per-phase backend specialization outperforms uniform placement. Our work extends this in two directions absent from LLM-only studies: (1) we add the vision encoding phase as a distinct hardware domain, quantifying 20–45\(\times\) CPU-to-NPU speedups across four encoder families; (2) we provide a concrete four-step porting methodology for encoders that fail QNN compilation out of the box—a practical gap not addressed by prior work targeting text-only LLM pipelines.

Cold-start and quantization. NNV12 [20] and PASK [21] motivate our three-state cache taxonomy. W4A8 [9] and AWQ [10] reduce VTCM pressure, enabling 97.5 tok/s decode. FlashAttention [25] and GQA [26] are resolved in the operator adaptation step.

Compilation and runtime toolchains. ExecuTorch [23] and MLC-LLM [22] target edge AOT export; our encoder porting methodology complements these by resolving QNN-specific operator adaptations required for Hexagon deployment. RoPE [27] embeddings require additional decomposition in our attention rewrite step.

10 Conclusion↩︎

We presented a VLM-specific phase characterization and deployment study on a production mobile NPU (Qualcomm SM8750), yielding six actionable insights: phase-dependent NPU gain (1.64\(\times\) prefill, 1.18\(\times\) decode); 10.47\(^\circ\)C thermal advantage with 2.52\(\times\) lower energy eliminating throttling; Hybrid-PD as the best cold/warm pipeline balance; a zero-recompilation TTFS knob spanning 2.4\(\times\); encoder architecture as the dominant latency lever (20–45\(\times\) CPU-to-NPU speedups across four families); and a four-step porting methodology at 22\(\times\) on Phi-3.5-V. Together these results constitute a deployment map for heterogeneous VLM inference across the latency, energy, and thermal axes that define always-on edge intelligence.

References↩︎