1 Introduction↩︎

On-device AI has evolved from narrow perception tasks, such as image classification and keyword spotting, to workloads that demand sustained reasoning over long contexts, multimodal fusion, and real-time decision-making. To efficiently handle these diverse demands, modern edge System-on-Chips (SoCs) integrate increasingly heterogeneous processing units (PUs): CPUs, GPUs, and neural processing units (NPUs).

Concurrently, the landscape of neural network architectures deployed at the edge is diversifying. Beyond the Transformer and convolutional neural network (CNN) architectures that current deployment stacks are designed around, emerging models introduce operators with distinct compute requirements: FFT-based long convolutions and elementwise gating (Hyena [@hyena]), learnable spline evaluations with control-heavy logic (KANs [@kan]), and selective scan kernels (Mamba [@mamba]). In addition, multi-stage pipelines such as vision-language-action (VLA) models (\(\pi_{0.5}\) [@phi05]) chain encoders, decoders, and iterative denoising stages, each with its own operator mix. These architectures rely on non-standard operations that do not uniformly map onto a single PU. As a result, the choice of PU becomes critical, since certain operators may suffer order-of-magnitude latency and energy penalties on a mismatched PU [@hkn]. Furthermore, edge systems increasingly serve multiple inference streams concurrently (e.g., object detection alongside speech recognition, or a vision encoder co-executing with a language decoder in a VLA pipeline), amplifying the cost of suboptimal PU assignment.

Despite the growing operator diversity, current deployment frameworks map an entire model’s execution graph to a single target PU, typically the one expected to offer the best average performance. This monolithic approach has three limitations. First, it ignores per-PU capability differences. Each PU excels at different operator types, yet monolithic execution forces all operators onto one PU, causing non-standard operators to incur significant latency and energy penalties [@hkn; @das2025grannite; @das2025xamba; @raha2025llm]. Second, single-PU mapping leaves hardware underutilized. Though many operators in an inference graph have no data dependencies between them and could execute concurrently across PUs, single-PU execution serializes the entire graph. Third, forcing the entire graph onto one backend exposes unsupported operators, causing the backend to either fall back to an inefficient execution path, silently substitute an approximation, or fail to compile altogether [@hkn].

The root cause of these inefficiencies is the absence of a system-level orchestration layer that is aware of operator-to-PU affinity. Existing deployment frameworks (e.g., OpenVINO [@openvino], TensorRT [@tensorrt], ONNX Runtime [@onnxruntime]) optimize within a single-PU’s compilation boundary but do not provide cross-PU partitioning, scheduling, or parallel orchestration. Existing heterogeneous inference engines target specific accelerator pairs or model families, or else rely on coarse-grain model heuristics rather than general-purpose, model-agnostic orchestration across arbitrary PU configurations [@heterollm; @axonn; @haxconn; @agent.xpu].

More fundamentally, the core challenge in heterogeneous inference is not merely selecting a target PU, but jointly reasoning about operator-level heterogeneity, inter-operator independence, and multi-model concurrency. We observe that these dimensions can be captured within a single graph-search formulation, enabling optimal operator-to-PU assignment across diverse models and execution scenarios.

To address this gap, we propose BIDENT, an operator-level orchestration framework for heterogeneous edge SoCs that realizes this unified graph-based formulation by mapping operators from single or concurrent inference requests to appropriate PUs based on profiled per-operator affinity and cross-PU contention. Figure 1 illustrates the high-level overview.

The key contributions of this work are:

• A unified operator-level abstraction for heterogeneous inference. We introduce a model-agnostic formulation that captures operator-level heterogeneity, inter-operator independence, and multi-model concurrency within a single graph-based representation.

• A principled graph-search formulation with optimality guarantees. We cast operator-to-PU assignment as a shortest-path problem over a weighted execution graph, reducing combinatorial scheduling to a polynomial-time optimization under both latency and energy objectives.

• A unified orchestration framework across execution regimes. BIDENT supports sequential heterogeneous mapping, intra-model parallel execution of independent operators, and multi-model concurrent co-scheduling within a single framework.

• A full-system implementation on a real heterogeneous SoC. We build BIDENT end-to-end, including profiling, execution graph construction, and scheduling, integrated with a production inference backend.

• Comprehensive evaluation across diverse models and workloads. Across ten model families and 190 multi-model combinations, BIDENT achieves up to 1.60\(\times\) speedup via intra-model parallelism and a 3.42\(\times\) geometric mean speedup in concurrent settings, demonstrating the importance of operator-level orchestration.

2 Background and Motivation↩︎

This section provides background on diverse neural network architectures for edge inference, and our motivational observations that reveal the shortcomings of single-PU execution.

2.1 Diverse Neural Network Architectures and Their Compute Demands↩︎

Modern edge SoCs integrate multiple processing units (CPU, GPU, and NPU) connected through an on-chip interconnect and sharing a common off-chip memory [@apple_m5_2025; @intelcoreultra]. The CPU provides general-purpose scalar and vector execution well-suited for control-heavy and irregular operations that benefit from instruction-level parallelism. The GPU employs a single-instruction, multiple-threads (SIMT) execution model with massively parallel threads, enabling high throughput for data-parallel workloads such as large general matrix multiplications (GEMMs). The NPU is a domain-specific accelerator built around fixed-function multiply-accumulate (MAC) arrays optimized for convolutions and energy-efficient GEMMs, complemented by vector DSPs that handle the elementwise and non-linear operations [@intelcoreultra].

The operators that compose neural network architectures exhibit distinct computational patterns, each with varying affinity to these processing units. Convolutions, dominant in CNNs, map directly to the NPU’s MAC arrays and achieve high utilization. GEMMs and matrix-vector products (GEMVs), which dominate Transformer attention and feed-forward layers, are compute-bound during prefill and memory-bound during decode [@agent.xpu], with the optimal PU depending on hardware resources. FFT-based long convolutions and elementwise gating, central to the Hyena operator [@hyena], are largely memory-bound and rely on operations that fall outside the GPU and NPU’s MAC-centric datapath, making the CPU a more natural fit. Selective state-space recurrences in models such as Mamba [@mamba] introduce sequential dependencies that limit data-level parallelism, favoring CPU execution. Recursive spline evaluations in KANs [@kan] are control-heavy and dominated by DSP-bound operators such as comparisons and conditional logic, aligning best with the CPU’s instruction-level parallelism.

Critically, practical neural network architectures comprise combinations of these operators rather than relying on any single type. A Transformer, for instance, interleaves GEMM-heavy attention with non-linear activations and normalization layers, while hybrid models further mix attention with SSM or Hyena operators. Multi-stage pipelines such as vision-language-action (VLA) models chain vision encoders, language decoders, and iterative denoising stages, each with distinct PU affinity and natural opportunities for inter-stage parallelism. Furthermore, edge systems often execute multiple models concurrently to serve diverse use-cases, leading to PU contention and resource underutilization when each model is statically assigned to a single PU. Thus, mapping models to a single PU inevitably leaves some operators suboptimally executed, underscoring the need for operator-aware heterogeneous orchestration across all available PUs.

2.2 PU Capability Variation↩︎

A key motivating observation is that the optimal processing unit varies not only across operator types but also across operand sizes for the same operator.

Observation 1: Best PU depends on operator type. To quantify how operator diversity influences PU affinity, Figure 2 compares seven key operators. Collectively, these operators represent the dominant computational patterns found in our selected model families. MatMul and Conv2D represent the dense GEMM-like kernels at the core of Transformer attention and feed-forward layers and CNN feature extraction. DWConv and Elementwise ADD capture the low-arithmetic-intensity operators found in efficient backbones and residual connections. RDFT, CumSum, and Gather correspond to the non-GEMM kernels that characterize, respectively, Hyena’s FFT-based long-convolution, Mamba’s SSM sequential-scan recurrence, and KAN’s spline evaluation. Unlike the GPU-favoring GEMM operators, these three all favor the CPU. Profiling results on the CPU, GPU, and NPU confirm that no single PU is optimal across these operators. The GPU delivers lowest latency for dense linear operators: MatMul is \(2.8\times\) faster than CPU and \(1.6\times\) faster than NPU, while Conv2D is \(2.2\times\) faster than CPU and \(1.1\times\) faster than NPU. For all remaining operators, the CPU is fastest. The non-GEMM operators (RDFT, CumSum, and Gather) suffer the most severe NPU penalty, with latencies \(4.7\times\), \(8.7\times\), and \(4.1\times\) higher than on the CPU, respectively, because these operations fall outside the NPU’s MAC-centric datapath and must route through its vector DSP or fall back to the CPU, incurring fixed per-kernel dispatch overhead on top of reduced throughput. Even NPU-native operators such as DWConv and Elementwise ADD are outperformed by the CPU at the tested shapes, where the NPU’s dispatch overhead dominates the total compute time.

Observation 2: Best PU depends on operand size. Figure 3 sweeps square MatMul of shape \([1, N, N] \times [N, N]\) for \(N \in \{32, \ldots, 2048\}\) at both FP16 and INT8 precision. At FP16 (Figure 3a), the CPU is fastest for \(N \leq 64\); the GPU crosses over at \(N=128\) and widens its lead to \(4.8\times\) at \(N=2048\). At INT8 (Figure 3b), the CPU holds the lead through \(N=128\) while the GPU crosses over at \(N=256\). The NPU overtakes the GPU at \(N=2048\), the only configuration in which the NPU is the fastest PU. This crossover demonstrates that the NPU’s INT8 compute density is sufficient to overcome its dispatch overhead only at large operand sizes, while the GPU remains preferable for mid-range sizes.

Together, these observations imply that no single PU dominates across all operator types and operand sizes, motivating a mapping strategy that assigns the best PU for each operator individually.

2.3 SoC Underutilization↩︎

When an inference execution graph is serialized on a single PU, the remaining PUs on the SoC sit idle during the entire inference. Inference execution graphs frequently contain operator pairs with no mutual data dependency, such as parallel encoder branches, concurrent feature-extraction heads, or adjacent operators drawn from distinct model components in multi-task workloads. Serializing such independent subgraphs on one PU wastes hardware resources and unnecessarily extends end-to-end latency.

Observation 3: Parallel execution recovers idle PU capacity. Figure 4 measures the speedup from dispatching independent operator pairs to separate PUs concurrently, against the best serial single-PU baseline. Three pairs are profiled, each representing a distinct PU-affinity configuration drawn from the operator spectrum in Observation 1. P1 (MatMul \(\|\) Conv2D) pairs two GPU-favoring operators from Transformer and CNN workloads. P2 (MatMul \(\|\) CumSum) pairs a GPU-favoring GEMM with a CPU-favoring sequential-scan operator, representative of hybrid Transformer–Mamba inference graphs. P3 (Conv2D \(\|\) DWConv) pairs two convolution variants with split PU preferences, as from efficient CNN backbones.

Across all six ordered assignments per pair, GPU\(\|\)CPU consistently achieves the highest speedup: \(1.41\times\) for P1, \(1.38\times\) for P2, and \(1.46\times\) for P3. In each case the CPU operator completes entirely within the GPU’s execution window, hiding its latency. These measured speedups already reflect shared-memory bandwidth contention between the two active PUs; BIDENT’s contention model (Section 3.2.2) accounts for this effect when computing parallel makespans. Assignments that place the operator on the slower PU (e.g., placing MatMul or Conv2D on the CPU) severely fall below the serial baseline, confirming the importance of optimal PU selection for each operator. This operator-level parallelism generalizes to multi-model workloads where entire inference graphs of independent models can be interleaved across PUs.

Together, Observations 1–3 motivate a system-level orchestration framework that (1) maps each operator to the PU best suited for its type and operand size, (2) exploits inter-operator parallelism for data-independent subgraphs, and (3) co-schedules concurrent inference streams across idle PUs, the three core capabilities of BIDENT.

3 BIDENT Framework↩︎

This section describes our BIDENT, an operator-level orchestration framework for heterogeneous edge SoCs. BIDENT takes as input one or more neural network models (supporting concurrent multi-model inference) and a description of the target SoC’s processing units. Then, it produces an optimized execution strategy that maps individual operators to PUs, minimizing end-to-end inference latency or energy, with the objective chosen by the user. Throughout this work, we use the term operator to refer to a group of primitive operations fused by the backend compiler (e.g., Conv--BN--ReLU), consistent with the granularity used in [@axonn; @haxconn]. BIDENT consists of three main components: (1) a profiler that characterizes per-operator execution costs on each PU, (2) an execution graph builder that encodes the mapping problem as a weighted directed graph, and (3) a search engine that finds the optimal mapping via shortest-path algorithms. Algorithm 5 summarizes the end-to-end pipeline.

BIDENT is designed to integrate with existing inference compilers and ML frameworks. In our implementation, we integrate BIDENT with OpenVINO [@openvino], using it as the backend for per-PU model compilation and inference. However, the framework is compiler-agnostic and can be adapted to other backends such as TensorRT [@tensorrt], ONNX Runtime [@onnxruntime], or QNN [@qnn].

3.1 Profiler↩︎

The profiler (Algorithm 5, lines 2–9) measures four categories of per-operator latency and energy on each PU:

• Host-to-Device transfer (H2D): The time to make input tensors available to the target PU, including cache flush/invalidation and direct memory access (DMA) descriptor setup.

• Kernel dispatch: The time to submit the pre-compiled kernel to the PU’s execution queue.

• Kernel execution: The actual compute time on the PU, measured using hardware performance counters.

• Device-to-Host transfer (D2H): The time to make output tensors available to the host or to a subsequent PU, symmetric to H2D.

Profiling is performed offline and once per model–SoC pair. In case of concurrent execution, the profiler also profiles each model pair–PU pair combination. The profiler decomposes the model into fused operators (line 3) by running PERF_COUNT profiling on the NPU. The profiler extracts each fused operator as a standalone sub-model (line 5), compiles it independently for each PU, and measures the above four timing components and power to obtain \(w(O_i, P_j)\) and \(p(O_i, P_j)\), respectively (line 6–7). The resulting cost table, indexed by (operator, PU), serves as input to the execution graph builder. Since each operator is profiled at its actual operand shape within the model, the cost table inherently captures size-dependent PU affinity (cf.Observation 2, Section 2.2). PU crossover points shift with operand size, and this variation is reflected directly in the measured costs without any analytical model. If an operator fails to compile or cannot be profiled on a given PU (as occurs for non-standard operations on mismatched backends), its entry is omitted from the cost table, and the graph builder creates no corresponding node. The search then naturally routes such operators to a supported PU, eliminating the fallbacks and compilation failures that monolithic single-PU execution would otherwise incur (cf.Section 1).

3.2 Execution Graph Builder↩︎

The core of BIDENT is the formulation of the operator-to-PU mapping problem as a shortest-path problem over a weighted directed graph (Algorithm 5, lines 11–19).

3.2.1 Sequential Execution Graph↩︎

For models with a linear (sequential) data dependency chain \(O_1 \rightarrow O_2 \rightarrow \cdots \rightarrow O_N\), BIDENT constructs the execution graph \(G = (V, E)\) as follows (lines 12–16).

Nodes. For each operator \(O_i\) and each processing unit \(P_j \in \{P_1, \ldots, P_K\}\) (where \(K\) is the number of PUs), we create a node \(v_{i,j}\) representing the execution of \(O_i\) on \(P_j\). The node weight \(w(v_{i,j})\) is the profiled execution cost (kernel dispatch + kernel execution) of \(O_i\) on \(P_j\). Additionally, a virtual start node \(s\) and a virtual end node \(t\) are added.

Edges. Edges connect nodes between consecutive operators:

• From \(s\) to each \(v_{1,j}\): weight = H2D cost for \(O_1\) on \(P_j\).

• From \(v_{i,j}\) to \(v_{i+1,k}\) for all PU pairs \((j, k)\): if \(j = k\) (same PU), the edge weight is zero (no transition cost); if \(j \neq k\) (PU transition), the edge weight is the data-transfer overhead profiled for \(O_{i+1}\) on \(P_k\): \(\text{H2D}(O_{i+1}, P_k)\) when \(P_k\) is an accelerator (GPU or NPU), plus \(\text{D2H}(O_i, P_j)\) for transitions between two accelerators or from an accelerator to CPU.

• From each \(v_{N,j}\) to \(t\): weight = D2H cost for \(O_N\) on \(P_j\).

The total cost of a path from \(s\) to \(t\) equals the end-to-end inference latency under the corresponding operator-to-PU mapping. The optimal mapping is the shortest path from \(s\) to \(t\).

3.2.2 Parallel Execution Graph↩︎

Many neural network architectures, particularly the hybrid models discussed in Section 2.1, contain operators with no mutual data dependencies that can execute concurrently on separate PUs, the idle-PU waste quantified in Observation 3 (Section 2.3). Exploiting this intra-model parallelism is the primary objective of BIDENT’s parallel orchestration engine. BIDENT identifies these parallelization opportunities by analyzing the model’s data dependency structure and partitioning the execution into phases (line 16).

Phase and branch partitioning. BIDENT performs a topological traversal of the fused-operator DAG and partitions it into phases bounded by fork points (out-degree \(>1\)) and join points (in-degree \(>1\)). In each phase, operators are further grouped into branches, independent computation paths identified by depth-first search from the phase’s root operators. Branches within a phase are mutually data-independent and can execute concurrently on separate PUs, while operators within a single branch form a sequential chain.

The latency or energy of a phase, its makespan, is the maximum branch cost across all concurrent branches, adjusted for cross-PU contention (see below). Phase boundaries act as synchronization barriers (i.e., all branches must complete before the next phase begins).

Parallel execution graph. For each branch within a phase, BIDENT constructs a sequential execution subgraph and runs per-branch Dijkstra (the same algorithm as the sequential search, allowing per-operator PU mixing within a branch). The phase makespan is the maximum of the contention-adjusted branch costs. The total parallel latency is the sum of phase makespans.

Memory contention modeling. When multiple PUs execute concurrently, they contend for shared memory bandwidth [@haxconn]. BIDENT uses two empirically grounded contention models:

• Intra-model parallel: When branches execute concurrently on different PUs, each operator’s cost is adjusted by a measured cross-PU slowdown factor \(\text{SF}(P_{\text{run}}, P_{\text{interfere}})\) derived from cross-PU profiling experiments. In our evaluation, the NPU shows the highest sensitivity (\(1.17\times\) when the CPU is active, \(1.09\times\) when the GPU is active); CPU and GPU show negligible cross-PU interference.

• Multi-model concurrent: For co-scheduled operators from different models on the same PU, BIDENT profiles each model pair under barrier-synchronized simultaneous execution, capturing the actual co-execution latencies without parametric assumptions.

Multi-model concurrent inference. BIDENT extends to concurrent execution of multiple inference requests (line 18) via two Dijkstra modes:

• Aligned Dijkstra (same-model pairs): Both requests advance in lockstep. State \(= (\text{step}, d_0, d_1)\); at each step the search selects a PU pair \((d_0, d_1)\). Same-PU cost uses the average of measured concurrent execution times; cross-PU cost uses the maximum of solo times.

• Joint \((i,j)\) Dijkstra (mixed-model pairs): Each request’s progress is tracked independently. State \(= (i, j)\) where \(i\) and \(j\) count completed operators per request. Three transitions are available: advance both (\(i{+}1, j{+}1\)), advance request 0 solo (\(i{+}1, j\)), or advance request 1 solo (\(i, j{+}1\)). This allows asymmetric completion with solo tails when one model finishes before the other.

Co-scheduled operators on the same PU use empirically measured concurrent execution costs; operators assigned to different PUs use their respective profiled costs.

3.3 Optimal Mapping Search↩︎

3.3.1 Sequential Search↩︎

Given the sequential execution graph \(G\), the optimal operator-to-PU mapping reduces to finding the shortest path from \(s\) to \(t\) (Algorithm 5, line 22). Since \(G\) is a directed acyclic graph (DAG) with non-negative edge weights, this can be solved exactly using Dijkstra’s algorithm [@dijkstra] in \(O(NK^2\log(NK))\) time, or via dynamic programming in \(O(NK^2)\) time by processing operators in topological order.

Specifically, let \(\text{cost}(i, j)\) denote the minimum total latency (or energy) to execute operators \(O_1, \ldots, O_i\) with \(O_i\) assigned to PU \(P_j\). The recurrence is: \[\label{eq:dp95sequential} \text{cost}(i, j) = w(v_{i,j}) + \min_{k \in \{1, \ldots, K\}} \left[ \text{cost}(i-1, k) + \text{trans}(i-1, k, i, j) \right]\tag{1}\] where \(\text{trans}(i-1, k, i, j)\) is the transition cost from PU \(P_k\) (after \(O_{i-1}\)) to PU \(P_j\) (for \(O_i\)): zero if \(k = j\); otherwise, the data-transfer overhead profiled for \(O_i\) on \(P_j\): \(\text{H2D}(O_i, P_j)\) when \(P_j\) is an accelerator, plus \(\text{D2H}(O_{i-1}, P_k)\) for accelerator-to-accelerator or accelerator-to-CPU transitions.

The base case is \(\text{cost}(1, j) = \text{H2D}(O_1, P_j) + w(v_{1,j})\) for each PU \(P_j\). The optimal end-to-end latency is \(\min_j [\text{cost}(N, j) + \text{D2H}(O_N, P_j)]\).

3.3.2 Intra-Model Parallel Search↩︎

For the parallel execution graph (lines 23–27), BIDENT runs a per-branch Dijkstra within each phase. Each branch’s operators form a sequential chain, so the same Dijkstra algorithm from Section 3.3.1 applies, allowing per-operator PU mixing within each branch. After all branches are solved, the phase makespan is computed as the maximum of the contention-adjusted branch costs. For each branch, the cost of every operator assigned to PU \(P_j\) is multiplied by \(\max_{P_k \in \mathcal{P}_{\text{other}}} \text{SF}(P_j, P_k)\), where \(\mathcal{P}_{\text{other}}\) is the set of PUs used by the other concurrent branches and \(\text{SF}\) is the measured cross-PU slowdown factor. The total parallel latency is the sum of per-phase makespans.

3.3.3 Multi-Model Concurrent Search↩︎

For concurrent execution of multiple inference requests (line 29), BIDENT searches over the product state space using the aligned or joint \((i,j)\) Dijkstra modes described in Section 3.2.2. This is tractable for typical values of \(K\) (e.g., \(K = 2\)–\(3\)).

3.3.4 Energy-Optimal Scheduling↩︎

The same graph and search framework supports energy-optimal scheduling by replacing the latency edge weight \(w(O_i, P_j)\) with energy \(w(O_i, P_j) \times p(O_i, P_j)\), where \(p(O_i, P_j)\) is the sustained package power measured on PU \(P_j\) via hardware sensors during inference. The Dijkstra search then minimizes total energy rather than total latency. For parallel phases, the contention slowdown factor increases per-operator latency, which in turn increases the energy cost of concurrent execution; consequently, the energy-optimal parallel schedule may differ from the energy-optimal sequential schedule.

3.4 Framework Overhead↩︎

BIDENT’s overhead is minimal and amortized:

Profiling. Profiling is a one-time, offline cost per model–SoC pair for single execution, and per model pair–PU pair for concurrent execution.

Graph construction. The execution graph has \(O(NK)\) nodes and \(O(NK^2)\) edges. For typical models (\(N \approx 10\)–\(500\), \(K = 2\)–\(3\)), the graph contains hundreds to thousands of nodes. For multi-stage VLA pipelines with repeated denoising steps, \(N\) can reach \({\sim}4{,}600\), yielding \({\sim}14{,}000\) nodes, still tractable for Dijkstra and consuming negligible memory.

Search. The sequential search runs in \(O(NK^2)\) time, sub-millisecond for practical problem sizes. The parallel search adds the intra-phase assignment cost, which is polynomial in \(K\) for each phase. The total offline cost is dominated by profiling, not search, and is amortized as a one-time cost.

Since all three components execute before inference, BIDENT introduces zero runtime overhead to the inference critical path. The output schedule is a static mapping that is applied directly by the execution orchestrator.

3.5 System Model↩︎

BIDENT targets modern unified-memory edge SoCs and exploits two structural properties of this platform class:

Unified memory address space. On modern heterogeneous SoCs, the CPU, GPU, and NPU share a common off-chip DRAM [@apple_m5_2025; @intelcoreultra], so tensor allocation is a one-time cost performed before inference begins. The host-to-device (H2D) and device-to-host (D2H) costs measured by BIDENT’s profiler therefore correspond to cache coherency operations, input-output memory management unit (IOMMU) page-table walks, and DMA descriptor setup, rather than physical data copies. BIDENT accounts for these costs directly in the execution graph as PU transition edges.

Ahead-of-time kernel compilation. Modern inference frameworks such as OpenVINO compile models and offload kernels once during initialization (e.g., core.compile_model()) before the inference loop begins [@openvino]. Compilation is thus outside the per-inference critical path. The per-inference overhead BIDENT captures is the kernel dispatch latency, the time to submit a pre-compiled kernel to the PU’s command queue, which is measured and included as part of each operator’s execution cost.

4 Evaluation↩︎

4.1 Experimental Setup↩︎

Hardware platform. All experiments are conducted on an Intel Core Ultra (Lunar Lake, Series 2) platform [@intelcoreultra] integrating three PUs: a CPU with 8 logical cores (Performance + Efficient) at 3.3 GHz base frequency, an Intel Arc 140V integrated GPU with 64 Xe Vector Engines, and an Intel AI Boost NPU 4 with 6 compute tiles. The three PUs share a 16 GB unified DDR5 memory subsystem, consistent with BIDENT’s unified-memory system model (Section 3.5). The host OS is Windows 11.

Models. We evaluate BIDENT on ten models (Table 1) spanning CNNs, Transformers, SSMs, VLAs, and emerging architectures (19 configurations total). Nine models are profiled at both FP16 and INT8 precision; INT8 models are obtained via NNCF [@nncf] post-training quantization. KAN cannot compile on the NPU (unsupported BitwiseAnd on float inputs) and is therefore profiled on CPU and GPU only. \(\pi_{0.5}\) is a multi-stage VLA pipeline comprising a text embedder, an INT8 vision encoder, a prefix-cache decoder, and 10 iterative denoising steps; it is evaluated as a single mixed-precision configuration on CPU and NPU (the prefix-cache and denoising stages exceed GPU memory).

Profiling. We use OpenVINO 2025.3 [@openvino] as the inference backend. We extract each fused operator from the whole models as a standalone sub-model, compile it independently on each PU, and measure its execution time (20 warm-up, 200 measurement iterations). Host-to-Device (H2D) and Device-to-Host (D2H) memory transfer latencies are measured with Intel VTune Profiler 2025.7 [@vtune] using its gpu-offload analysis for the GPU and NPU hardware-event queries for the NPU. Kernel dispatch overhead, Deferred Procedure Call (DPC) and Interrupt Service Routine (ISR) latency incurred when submitting kernels to the GPU and NPU command queues, is captured via Event Tracing for Windows (ETW) using xperf. Power is measured with HWInfo64 [@hwinfo] (CPU package power sensor; equivalent to full package power) during sustained whole-model inference; energy is derived as power \(\times\) latency.

Baselines. We compare BIDENT against three monolithic single-PU baselines (CPU-only, GPU-only, NPU-only) and report speedup relative to the best-performing single-PU for each model.

4.2 Sequential Orchestration↩︎

Figure 6 illustrates BIDENT’s sequential execution graph for five representative operators from LLaMA-7B (1L) FP16. The optimal path assigns Gate Proj and Up Proj to the NPU, SiLU and element-wise Mul to the CPU, and Down Proj to the GPU, demonstrating that different operator types favor different PUs even within a single layer.

Table 2 reports the results of BIDENT’s sequential orchestration across all 19 model–precision configurations. BIDENT-lat shows the latency-optimized heterogeneous latency; BIDENT-energy shows the energy reduction achieved by the energy-optimized search (Section 3.3.4).

Latency. BIDENT achieves consistent speedups of 1.01–1.58\(\times\) across all 19 configurations (geometric mean 1.09\(\times\)). Since the schedule is computed offline and applied as a static mapping, these gains come with zero runtime overhead.

Emerging architectures see the largest gains. SNN-VGG9 FP16 achieves 1.58\(\times\): its 93 fused operators split across NPU (50 convolution ops that benefit from the MAC arrays), CPU (39 lightweight spiking-accumulation ops where per-op CPU execution avoids the NPU dispatch overhead), and GPU (4 ops), directly exploiting the operator-type affinity differences from Observation 1. Mamba-370M FP16 (1.14\(\times\)) and Hyena FP16 (1.16\(\times\)) benefit similarly: their non-GEMM operators (selective scan and FFT-based long convolution, respectively) are cheaper on the CPU, while standard MatMul operators are assigned to the GPU. \(\pi_{0.5}\) achieves 1.23\(\times\). BIDENT routes 3,992 of its 4,595 fused operators to the NPU (vision-encoder and denoiser attention blocks) and 603 to the CPU (text embedding and lightweight non-linear ops), exploiting the distinct PU affinity of each pipeline stage.

For conventional architectures (ResNet-50, ViT-B/16), BIDENT achieves 1.04–1.10\(\times\) since most operators are already well-served by a single PU, with only a few normalization or element-wise ops benefiting from redirection. For models whose operator mix is nearly uniform (LLaMA, KAN), the heterogeneous mapping provides small benefit (\(\leq\)​1.02\(\times\)). We profile a single LLaMA-7B decode layer to isolate the Transformer operator mix; since decode layers are structurally identical, per-layer behavior is representative of the full model’s per-layer scheduling potential.

The near-unity speedup confirms that standard Transformer decode layers, dominated by large MatMul operators, are already well-served by a single PU, validating that BIDENT’s gains are concentrated where operator heterogeneity exists. More broadly, this result highlights that the primary opportunity for heterogeneous orchestration lies not in further optimizing homogeneous workloads, but in exploiting heterogeneity across operators and across workloads. Sequential orchestration therefore establishes a lower bound; Section 4.3 shows that substantially larger gains emerge when intra-model parallelism and multi-model concurrency are considered.

Energy. The BIDENT-energy column in Table 2 shows the energy reduction achieved by the energy-optimized search (Section 3.3.4), which replaces edge weights with energy \(= w \times \text{power}\). Conceptually, there are two distinct paths to energy reduction: (i) indirectly, by minimizing latency—since energy equals power \(\times\) latency, a shorter execution time can lower total energy even on a higher-power PU—and (ii) directly, by minimizing the profiled energy \(w \times p\) during the search, which sacrifices some speedup on operators where a lower-power PU suffices. BIDENT-lat and BIDENT-energy realize these two paths, respectively. The energy-optimal schedule always reduces energy versus the best single-PU baseline (average 9.2% across 19 configs). SNN-VGG9 FP16 saves 44%, \(\pi_{0.5}\) saves 19%, Hyena FP16 saves 11%, and KAN achieves a slight reduction (0.2–0.7%). The energy-optimal assignment incurs a latency trade-off: its geometric mean latency speedup is 1.03\(\times\) (vs.\(\times\) for latency-optimal), since the lowest-energy PU is not always the fastest.

Figure 7 visualizes this trade-off, comparing the energy reduction of the latency-optimized and energy-optimized schedules for all 19 configurations. The latency-optimized schedule reduces energy by 3.7% on average but increases energy for 5 of 19 configurations, while the energy-optimal schedule eliminates all such cases. Comparing the two paths head-to-head, the direct energy-driven path (9.2% average reduction, zero regressions) dominates the indirect latency-driven path (3.7%, 5 regressions), because the latency objective is blind to per-PU power and will pay any power premium for a latency gain. This divergence arises because the three PUs differ substantially in sustained power. In our measurements the GPU draws the highest package power under GEMM load, the NPU the lowest (its MAC arrays being the most power-efficient datapath), and the CPU is intermediate, so an operator with similar latency on two PUs is routed to the faster one under the latency objective and to the lower-power one under the energy objective. For example, BIDENT-lat offloads KAN’s operators to the GPU for only a 1.01\(\times\) speedup yet raises energy 18% above the CPU baseline, while BIDENT-energy keeps them on the CPU. This highlights the inherent latency–energy tradeoff that the fastest PU for an operator is not always the most energy-efficient, so the two objectives select different PUs whenever the power gap outweighs the latency gap. Conversely, the NPU being the lowest-power PU does not imply it is always the most energy-efficient. An operator that runs much slower on the NPU (e.g., one that falls outside the MAC datapath and spills to the vector DSP) can consume more total energy there than on the CPU despite the NPU’s lower instantaneous power. The energy-optimal search captures this by operating on the profiled product \(w \times p\) rather than on power alone. Two caveats generalize these results. First, the optimal partitioning is hardware-dependent. Successive NPU generations differ in power efficiency, peak throughput, and dispatch overhead, so the same model may prefer a different PU assignment on a different SoC, and the energy winner on one generation may not be the winner on the next. Second, both latency and power shift at runtime under thermal throttling, memory-bandwidth contention, and concurrent workload pressure. This variability is more prominent in dynamic multi-model settings and motivates the dynamic scheduling extension discussed in Section 6.

4.3 Parallel and Concurrent Orchestration↩︎

Beyond sequential scheduling, BIDENT exploits two additional levels of concurrency: (1) intra-model phase-based parallelism, where data-independent operators within a single model execute simultaneously on separate PUs, and (2) multi-model concurrent inference, where operators from two independent inference requests are co-scheduled across PUs.

Intra-model parallel results. Figure 8 (a) annotates the parallel phases in the sequential execution graph from Fig. 6. Phase \(\Phi_8\) contains Up Proj and SiLU, which have no data dependency and can run concurrently on NPU and CPU. Table 3 reports the parallel execution speedup relative to the best single-PU and gain compared to sequential orchestration.

The parallel gain over the sequential schedule depends on the model’s internal data-dependency structure. All makespans in Table 3 include the measured cross-PU contention factors from Section 3.2.2. ViT-B/16 benefits the most (+29%) since its multi-head attention produces 51 concurrent phases in which independent attention heads execute on different PUs simultaneously, raising the overall speedup from 1.10\(\times\) to 1.55\(\times\). \(\pi_{0.5}\) gains +16% (1.46\(\times\)) from 615 concurrent phases. The text-embedding and vision-encoding stages execute in parallel, and each denoising iteration contains intra-layer parallel branches. Mamba-370M (+12%, 25 concurrent phases) and LAVISH (+9%, 4 concurrent phases) also see gains; Mamba from its parallel SSM branches and LAVISH from its dual visual-audio encoder. BitNet, whose 36 operators form a single sequential chain (0 concurrent phases), sees no parallel gain. KAN and LLaMA have 2–3 concurrent phases but negligible gain (\(\leq\)​3%) because the parallel branches are heavily imbalanced.

Parallel energy. Because the contention slowdown factor increases per-operator latency during concurrent phases, the energy-optimal parallel schedule consumes more energy than the sequential one: 7.6% average reduction (parallel) versus 9.2% (sequential). The 1.6% gap is the measured cost of memory bandwidth contention. Models with many concurrent phases (ViT, Mamba, \(\pi_{0.5}\)) show the largest parallel–sequential energy gap; models with zero concurrent phases (BitNet) show no difference.

Multi-model concurrent results. Edge SoCs increasingly serve multiple inference streams simultaneously, for example, running object detection alongside speech recognition, or co-executing a vision encoder with a language decoder in a vision-language-action (VLA) pipeline. In such settings, effective utilization of heterogeneous resources requires coordinating execution across models, as concurrent workloads introduce both memory-bandwidth contention [@haxconn] and diverse operator-level execution patterns [@agent.xpu]. BIDENT addresses this scenario by co-scheduling operators from both models across all available PUs via its Dijkstra-based concurrent search (Section 3.3.3), enabling complementary operator-to-PU affinities to be exploited while recovering otherwise idle hardware capacity.

Figure 8 (b) shows a concrete example. When LLaMA-7B (1L) INT8 and Mamba-370M FP16 are co-scheduled, BIDENT assigns each step’s operators to different PUs so that both models make progress concurrently. After LLaMA completes, Mamba’s remaining 39 operators continue on NPU and CPU.

Figure 9 presents the speedup for all 190 model pairs (10 models, 19 configurations) versus homogeneous serial execution, where both models run sequentially on their best single PU.

Across all 190 pairs, BIDENT achieves a geometric mean speedup of 3.42\(\times\) (range: 0.86–22.4\(\times\)). Three patterns emerge from the heatmap:

(1) Complementary-affinity pairs yield the largest gains. KAN\(_{INT8}\)+LAVISH\(_{FP16}\) reaches 22.4\(\times\). KAN’s CPU-bound spline evaluations overlap almost entirely with LAVISH’s GPU-bound convolution blocks. Similarly, Mamba\(_{FP16}\)+SNN\(_{INT8}\) achieves 21.7\(\times\) by interleaving selective-scan ops on one PU with spiking-accumulation ops on another.

(2) Same-model pairs still benefit substantially. BitNet\(_{FP16}\)+BitNet\(_{FP16}\) reaches 14.1\(\times\) and Mamba\(_{FP16}\)+Mamba\(_{FP16}\) reaches 10.5\(\times\), because BIDENT distributes the two instances’ operators across GPU and CPU, recovering the idle-PU capacity that serial single-PU execution wastes.

(3) Large-model pairs show moderate gains. Pairs involving LLaMA or ViT achieve 2–5\(\times\) because their dominant large MatMul operators saturate a single PU’s memory bandwidth. \(\pi_{0.5}\) pairs range from 1.2–18.7\(\times\): with KAN the smaller model runs entirely within \(\pi_{0.5}\)’s execution window (18.7\(\times\)), while pairs with similarly large models (Hyena FP16, ViT) achieve 1.3–1.4\(\times\). Two pairs (1.1% of total) show speedup \(<\)​1\(\times\), where concurrent memory-bandwidth contention outweighs the parallelism benefit.

Concurrent energy. Thanks to energy-optimal concurrent scheduling, BIDENT achieves an average energy reduction of 48.2%; 16 pairs see an energy increase where the contention penalty exceeds the energy benefit.

5 Related Work↩︎

Existing heterogeneous inference systems typically address only a subset of the orchestration problem. Prior approaches either focus on specific model families, operate at coarse granularity (model or phase level), or target only inter-model scheduling. As a result, they lack a unified mechanism to jointly reason about operator-level PU affinity, intra-model parallelism, and multi-model concurrency. BIDENT addresses this gap through a single, model-agnostic formulation that captures all three dimensions.

Heterogeneous DNN execution on SoCs. AxoNN [@axonn] partitions CNN inference across CPU and DLA on NVIDIA Jetson SoCs using an energy-aware pipelined execution model. HaX-CoNN [@haxconn] models shared-memory contention when co-scheduling multiple DNN instances on CPU, GPU, and DLA, and proposes contention-aware inter-model scheduling. Both works advance heterogeneous SoC utilization, though AxoNN focuses on a two-PU CNN-specific setting and HaX-CoNN addresses inter-model scheduling rather than intra-model operator partitioning. BIDENT generalizes to three or more PUs (including NPUs), operates at per-operator granularity across arbitrary architectures, and supports both intra-model partitioning and multi-model concurrent scheduling within a unified graph formulation. While HaX-CoNN models contention at the whole-model level, BIDENT measures cross-PU interference empirically at per-operator granularity for both intra-model parallel and multi-model concurrent execution.

Heterogeneous LLM inference. HeteroLLM [@heterollm] partitions the prefill and decode phases of LLM inference across CPU, GPU, and NPU on mobile SoCs. Agent.xpu [@agent.xpu] extends heterogeneous scheduling to agentic LLM workloads with diverse tool-calling and reasoning operators. These systems demonstrate the value of heterogeneous execution for LLMs, though they rely on phase-level (prefill/decode) heuristics tailored to Transformer-specific workload characteristics. BIDENT is model-agnostic. It accommodates CNNs, Transformers, SSMs, KANs, Hyena, spiking networks, and multi-stage VLA pipelines with the same profiling-driven shortest-path formulation, without requiring architecture-specific scheduling rules.

NPU architecture and flexibility. FlexNPU [@flexnpu] investigates flexible NPU datapath designs that support a wider range of operators at the hardware level. This approach is complementary to BIDENT’s operator-level orchestration. A more flexible NPU reduces the set of operators that must be routed to alternative PUs, but as long as capability differences exist among PUs, BIDENT’s operator-level scheduling continues to provide benefit.

Summary. Table 4 positions BIDENT among these systems. Direct quantitative comparison is infeasible because each system targets a different SoC family and none provide public implementations on the Intel Core Ultra platform used in our evaluation.

6 Future Work↩︎

Dynamic operator-level scheduling. BIDENT currently produces a static mapping determined offline. However, as shown by Agent.xpu [@agent.xpu], neural network workloads on edge SoCs are increasingly introduced dynamically. Agentic LLM workflows invoke tool-calling models on demand, and multi-application environments launch and terminate inference streams at runtime. In such scenarios, the optimal assignment of PU changes as resource availability changes. Thermal throttling reduces PU throughput, concurrent system processes compete for memory bandwidth, and input-dependent execution paths (e.g., early exit, mixture-of-experts gating) alter operator counts mid-inference. A dynamic extension of BIDENT would require (1) lightweight runtime monitoring of PU utilization and memory bandwidth, and (2) fast re-profiling or analytical modeling of per-operator costs under varying resource contention, so that remapping decisions can be made at operator granularity without incurring scheduling overhead that negates the latency benefit.

Intra-PU tile-level mapping. Even with optimal inter-PU scheduling, BIDENT’s execution schedules exhibit underutilization bubbles (visible in Figure 8 (b)) where a PU sits idle while waiting for a phase boundary. Moreover, within a single PU, not all internal components are utilized simultaneously. For example, the NPU contains both MAC arrays and vector DSP units, yet a given operator typically exercises only one datapath [@hkn].

Modern SoCs incorporate tiled architectures. The Intel Core Ultra NPU exposes 6 compute tiles [@intelcoreultra], and GPUs use similar Xe Vector Engine clusters. A finer-grained mapping strategy could assign a variable number of tiles per operator based on its compute- or memory-boundedness, determined via roofline analysis. Operators below the ridge point (memory-bound) would receive fewer tiles, freeing the remaining tiles for concurrent data-independent operators, while compute-bound operators would receive more. This intra-PU tiling naturally extends to dynamic scheduling. Since resource availability changes at runtime, the roofline shifts and the tile assignment adapts accordingly.

7 Conclusion↩︎

We present BIDENT, an operator-level orchestration framework for heterogeneous edge SoCs that casts operator-to-PU assignment as a shortest-path search and jointly supports sequential, contention-aware intra-model parallel, and multi-model concurrent execution under latency and energy objectives. Evaluated on ten diverse architectures on an Intel Core Ultra SoC, BIDENT achieves up to 1.58\(\times\) sequential and 1.60\(\times\) intra-model parallel speedup over the best single-PU baseline, a 3.42\(\times\) geometric mean speedup across 190 multi-model pairs, and average energy reductions of 9.2% (sequential) and 48.2% (concurrent).

Extending BIDENT with runtime-adaptive scheduling and intra-PU tile-level mapping (Section 6) will further close the gap between available hardware capacity and realized inference performance. More broadly, this work suggests that operator-level orchestration, not model-level mapping, should serve as the fundamental abstraction for heterogeneous inference on future edge systems.

Acknowledgments↩︎

This work was supported in part by the Center for the Co-Design of Cognitive Systems (CoCoSYS) and the Center on Cognitive Multispectral Sensors (CogniSense), two research centers under the Joint University Microelectronics Program (JUMP) 2.0, a Semiconductor Research Corporation (SRC) initiative sponsored by DARPA.