1

1 Introduction↩︎

Efficient execution of ml workloads requires new hardware architectures. cim fuses computation and memory to address the von Neumann bottleneck [1], executing mvms with \(\mathcal{O}\mleft(1\mright)\) latency using crossbar arrays [2]–[4]. rram is a promising cim candidate due to its high device density, low power consumption, non-volatility, and CMOS compatibility [1], [5], [6]. However, cim accelerators rely on a host cpu to execute unsupported operators, requiring workload partitioning between cpu and cim accelerator [4].

Furthermore, cim storage is limited by the available crossbars [7]. Modern cnns often exceed this capacity [7]–[11]. Moreover, rram write latency and endurance limits make dynamic weight replacement impractical [5], [6], [12]. Hence, deployment requires static assignment of selected operators to the cim accelerator.

Existing ml partitioning approaches [13]–[16] treat devices as black boxes, neglect low-level architectural characteristics or assume sequential execution. cim-specific methods either assume full-model fit [17], [18] or rely on weight reprogramming [7], [11]. Furthermore, workload partitioning between the host cpu and the cim accelerator remains an open problem.

We propose a system-level workload partitioning framework for heterogeneous cpu–cim systems. It covers latency characterization and static operator assignment. The resulting partitioning between cpu and cim achieves up to \(\require{physics} \qty{30.9}{\times}\) and \(\require{physics} \qty{7.3}{\times}\) speedup over cpu-only execution on ARM edge and x86 cpus, respectively. 1 illustrates the framework’s components. We highlight the following contributions:

1. A hybrid latency characterization methodology, integrating empirical host cpu profiling with an analytical cim accelerator and interconnect performance model.

2. An ilp formulation for static operator assignment that minimizes the end-to-end inference latency accounting for limited cim memory budget, operator support, and parallelism constraints.

3. A dse using the proposed framework to analyze the impact of model architectures and hardware parameters on system performance, providing design insights for future cim accelerators.

[sec:background] [sec:related_work] provide background and review prior work. 4 presents our framework. 5 reports results and a dse. 6 concludes the paper.

2 Background↩︎

This section presents background related to rram-based cim accelerators and the ilp concepts used in this work.

2.1 rram-based cim Accelerators↩︎

rram is a memristive non-volatile memory technology that stores data as resistance states [1], [5], [6]. By arranging rram cells in crossbar arrays, cim accelerators perform analog mvms via Ohm’s and Kirchhoff’s laws [2]–[4]. Since programming rram cells is slow and repeated writes degrade device endurance [6], [12], a weight-stationary dataflow, where weights are programmed once before inference, is particularly well-suited for rram-based cim accelerators [2], [4].

The PUMA architecture [4] is hierarchically organized into cim cores, connected by an internal bus. Each core contains a control unit, instruction memory, a mvmu, a gpeu, and a data buffer. The mvmu comprises multiple rram crossbars with peripheral circuits. The gpeu performs simple arithmetic and a limited set of functions, including activation functions.

2.2 ilp↩︎

ilp is a class of optimization problems where decision variables are restricted to integers. The objective is to maximize or minimize a linear function with linear constraints. lp allows continuous variables. mip mixes integer and continuous variables.

State-of-the-art solvers [19] combine fundamental methods such as branch-and-bound with heuristics to identify optimal or near-optimal solutions. The relative mip gap measures solution quality as \(\frac{z_{\mathrm{inc}} - z_{\mathrm{bound}}}{\mleft|z_{\mathrm{inc}}\mright|}\), where \(z_{\mathrm{inc}}\), the incumbent, is the best known feasible integer solution and \(z_{\mathrm{bound}}\) is the proven bound obtained from the lp relaxation. However, a large gap does not necessarily indicate a suboptimal solution, since the lp bound may be significantly looser than the true ilp optimum.

3 Related Work↩︎

Prior research has explored ml workload partitioning for heterogeneous systems and cim accelerators specifically.

Neurosurgeon [13] splits ml execution between a mobile device and a datacenter at a layer granularity, selecting the optimal split point via per-layer prediction models. Viramontes et al. [14] extend this to a multi-device ilp formulation for edge-hub-cloud hierarchies, using weight preloading and multiple device transitions to minimize inference latency. CoEdge [15] and HiDP [16] further generalize to cooperative and hierarchical partitioning across heterogeneous edge nodes. These methods largely treat devices as black boxes, rely on high-level latency estimates, or assume sequential layer execution.

Regarding cim-specific approaches, Li et al. [17] optimize crossbar allocation by determining the per-layer weight replication factor for static mapping, and CIM-MLC [18] provides a multi-level compilation stack for operator scheduling across device, circuit, and architecture tiers. Both assume the complete model fits on the accelerator. Gao et al. [7] address the resource-constrained case by statically scheduling weight programming to minimize reprogramming overhead. COMPASS [11] addresses capacity limitations through weight partitioning, replication, and dynamic weight replacement. For rram-based targets, however, this degrades device endurance and introduces write latency, making reliable long-term deployment infeasible. Furthermore, none of the prior works specifically addresses joint cpu–cim execution of ml workloads.

In contrast, we target system-level cpu–cim partitioning, combining empirical cpu profiling with analytical models. Our ilp formulation accounts for memory constraints, operator support, and parallelism, with a weight-stationary dataflow.

4 Partitioning Framework↩︎

The framework targets a hardware as illustrated in 2. The platform comprises a single-core host cpu and a PUMA-based cim accelerator [4] connected through a shared bus. The cim accelerator comprises multiple cim cores (see 2.1).

4.1 Hybrid Latency Characterization↩︎

To accurately characterize operator execution latencies on both devices, we adopt a hybrid approach combining empirical cpu profiling with analytical cim and interconnect modeling.

4.1.1 Host cpu Profiling↩︎

The operator latency on the cpu, \(T_{\mathrm{host},v}\), is measured by profiling each operator using onnx Runtime [20], [21] on the target hardware. This empirical approach captures all architectural and optimization effects, which are difficult to model analytically, while supporting flexible replacement of the host cpu. We perform multiple iterations with warmup runs to eliminate startup effects, using the median latency for robustness. For model-level statistics, we enable all available graph and layout optimizations.

4.1.2 CIM Accelerator Modeling↩︎

We model the cim accelerator latency analytically, which enables evaluation across a broad range of accelerator configurations. rram crossbars execute one mvm per input vector in \(\mathcal{O}\mleft(1\mright)\) time [2]–[4]. However, limited crossbar size requires large operators to span multiple crossbars [2], [17], [22], [23]. The number of crossbars \(N_{\mathrm{crossbar},v}\) required for an operator \(v\) is defined as: \[N_{\mathrm{crossbar},v} = \mleft\lceil \frac{M_v}{M} \mright\rceil \cdot \mleft\lceil \frac{N_v}{N} \mright\rceil, \label{eq:number95of95crossbars}\tag{1}\] where \(M_v\) and \(N_v\) are the dimensions of the weight matrix for operator \(v\), and \(M\) and \(N\) are the dimensions of a single rram crossbar, as illustrated in 2.

Operators without a 2D weight matrix are converted to a gemm. For convolutions, this is achieved using im2col [22], [24]. The number of mvms \(N_{\mathrm{mvm},v}\) required to execute a gemm equals the number of input vectors applied to the crossbars. For instance, a Conv2D layer \(\mleft(C_{\mathrm{in}}, H_{\mathrm{in}}, W_{\mathrm{in}}\mright) \rightarrow \mleft(C_{\mathrm{out}}, H_{\mathrm{out}}, W_{\mathrm{out}}\mright)\) with kernel size \(\mleft(H_{\mathrm{K}},W_{\mathrm{K}}\mright)\) can be expressed as \(N_{\mathrm{mvm},v} = H_{\mathrm{out}} \cdot W_{\mathrm{out}}\) mvms with dimensions \(M_v = C_{\mathrm{out}}\) and \(N_v = C_{\mathrm{in}} \cdot H_{\mathrm{K}} \cdot W_{\mathrm{K}}\).

The operator latency on the accelerator, \(T_{\mathrm{acc},v}\), is modeled as the sum of the load, compute, and store latency [4], [22]: \[T_{\mathrm{acc},v} = T_{\mathrm{load},v} + T_{\mathrm{mvm},v} + T_{\mathrm{store},v}.\] Here, \(T_{\mathrm{mvm},v} = N_{\mathrm{mvm},v} \cdot T_{\mathrm{mvm}}\), with \(T_{\mathrm{mvm}}\) denoting the latency of a single mvm. The latency is independent of the number of crossbars, as these operate in parallel [4], [22].

Load and store latencies depend on the internal bandwidth \(B_\mathrm{cim}\), element size \(b_\mathrm{elem}\), and element counts \(N_{\mathrm{in},v}\) and \(N_{\mathrm{out},v}\): \[T_{\mathrm{load},v} = \frac{b_\mathrm{elem} \cdot N_{\mathrm{in},v}}{B_\mathrm{cim}}, \quad T_{\mathrm{store},v} = \frac{b_\mathrm{elem} \cdot N_{\mathrm{out},v}}{B_\mathrm{cim}}.\]

We assume the partial-sum accumulation and activation application on the gpeu are pipelined, thus not contributing to the overall latency [4].

4.1.3 Inter-device Transfer Modeling↩︎

For directly data-dependent operators \(u\) and \(v\) mapped to different devices, the output of \(u\) is transferred over the shared bus. The transfer latency is: \[T_{\mathrm{transfer},\mleft(u,v\mright)} = \frac{b_\mathrm{elem} \cdot N_{\mathrm{out},u}}{B_\mathrm{shared}},\] where \(B_\mathrm{shared}\) is the bus bandwidth between cpu and cim.

4.2 ilp-based Workload Partitioning↩︎

We formulate the operator partitioning as an ilp on a dag \(G = (V, E)\), where \(V\) is the operator set and each directed edge \((u,v) \in E \subseteq V \times V\) encodes a direct data dependency from \(u\) to \(v\).

Each operator \(v\) has a binary decision variable \(x_v\), where \(x_v = 1\) maps to host cpu and \(x_v = 0\) maps to cim. Only a subset of operators \(V_{\mathrm{acc}} \subseteq V\) is supported by the accelerator: \[x_v \in \mleft\{0,1\mright\}, \;\: \forall v \in V_{\mathrm{acc}}, \quad x_v = 1, \;\: \forall v \in V \setminus V_{\mathrm{acc}}.\]

The objective of the ilp is to minimize the makespan \({T \in \mathbb{R}_{\geq 0}}\), constrained by the finish times of all terminal nodes: \[T \geq f_v, \quad \forall v \in \mleft\{u \in V \mid \nexists\, w \in V \colon \mleft(u,w\mright) \in E\mright\}.\]

The finish time \(f_v\) depends on the start time \(s_v \in \mathbb{R}_{\geq 0}\) and the device-specific computation latency: \[f_v = s_v + x_v \cdot T_{\mathrm{host},v} + (1 - x_v) \cdot T_{\mathrm{acc},v}, \quad \forall v \in V.\]

The start time \(s_v\) is bounded by all predecessors \(\mathop{\mathrm{pred}}\mleft(v\mright) = \{u \in V \mid \mleft(u,v\mright) \in E\}\) and any required transfer latency: \[s_v \geq f_u + z_{u,v} \cdot T_{\mathrm{transfer},\mleft(u,v\mright)}, \quad \forall u \in \mathop{\mathrm{pred}}\mleft(v\mright),\]

The binary auxiliary variable \(z_{u,v}\) is 1 when \(u\) and \(v\) are assigned to different devices, and 0 otherwise, enforced by: \[z_{u,v} \geq x_u - x_v, \quad z_{u,v} \geq x_v - x_u, \quad \forall \mleft(u,v\mright) \in E.\]

4.2.1 Parallel Execution↩︎

Operators without data dependencies can execute in parallel on separate crossbars or across the host and the cim accelerator. We identify such pairs via the transitive closure \(G^{+} = (V, E^{+})\), where \(E^{+}\) contains all pairs \((u,v)\) connected by a path in \(G\). The parallelizable pairs are: \[P = \mleft\{\mleft\{u,v\mright\} \subseteq V \mid u \neq v, \mleft(u,v\mright) \notin E^{+}, \mleft(v,u\mright) \notin E^{+}\mright\}.\]

To serialize pairs when both operators are assigned to the host, we introduce a binary ordering variable \(o_{u,v}\) for each \(\{u,v\} \in P\), where \(o_{u,v} = 1\) if \(u\) precedes \(v\): \[\begin{align} {2} s_v &\geq f_u - M \cdot \mleft(3 - x_u - x_v - o_{u,v}\mright), &\quad& \forall \{u,v\} \in P, \\ s_u &\geq f_v - M \cdot \mleft(2 - x_u - x_v + o_{u,v}\mright), &\quad& \forall \{u,v\} \in P. \end{align}\]

We set \(M\) to an upper bound on the inference latency, where \(T_{\mathrm{acc},v} \mathrel{\vcenter{:}}= 0\) for \(v \notin V_{\mathrm{acc}}\): \[M = \sum_{v \in V} \max\mleft(T_{\mathrm{host},v}, T_{\mathrm{acc},v}\mright) + {\sum_{(u,v) \in E}} T_{\mathrm{transfer},\mleft(u,v\mright)},\] which deactivates the serializing constraint if either operator is assigned to the cim accelerator or the order does not apply.

4.2.2 Resource Constraint↩︎

Finally, the available rram crossbar count \(C \in \mathbb{N}\) limits how many operators can be offloaded: \[\sum_{v \in V_{\mathrm{acc}}} (1 - x_v) \cdot N_{\mathrm{crossbar},v} \leq C,\] where \(N_{\mathrm{crossbar},v}\) is the number of rram crossbars required to execute operator \(v\) on the accelerator, as defined in 1 .

5 Results↩︎

In this section, we evaluate our framework in terms of speedup, operator allocation, and ilp scalability on representative cnns. All ilp instances are solved using Gurobi [19].

Experiments are conducted on two single-core hosts: an AMD Ryzen 9 3900X (3.8 GHz, 32   DDR4) for high-performance x86 and an ARM Cortex-A72 (1.5 GHz, 4   LPDDR4) for edge deployment. The cpu–cim bus bandwidth is 20.48   s⁻¹ on x86 and 10.24   s⁻¹ on ARM. We sweep 25, 50, 100, & 200 crossbars and 50 ns, 100 ns, 500 ns, & 1000 ns mvm latency. This dse covers both hardware parameters and model architecture effects on system performance. Each crossbar stores up to weights, and the internal cim bus bandwidth matches the shared cpu–cim bus bandwidth.

We evaluate resnet-18 and resnet-50 [8] as convolution-heavy architectures, MobileNetV2 [9] for depthwise separable convolutions, and yolov5n [10] for complex branching patterns. All models are quantized to 8 integer precision [25].

5.1 cim Acceleration Speedup↩︎

3 reports resnet-18 speedups over cpu-only execution. On ARM, the speedup reaches \(\require{physics} \qty{30.9}{\times}\) at 50 ns mvm latency with 100 or more crossbars available. Even at 100 ns mvm latency, the speedup remains high at \(\require{physics} \qty{21.1}{\times}\), demonstrating substantial acceleration potential for edge deployments. Beyond 100 crossbars, speedups saturate because all profitable operators are already offloaded. On x86, the peak speedup is \(\require{physics} \qty{7.3}{\times}\) at 50 ns mvm latency, reflecting the stronger baseline performance of the desktop processor. For mvm latencies above 500 ns, offloading on x86 is no longer beneficial, because the layers can be executed faster on the cpu alone.

4 extends the analysis to all evaluated architectures at 50 ns mvm latency. resnet-18 exhibits the highest speedups, followed by resnet-50. MobileNetV2 shows limited acceleration potential because its depthwise separable convolutions cannot be efficiently mapped to rram crossbars. Each depthwise filter operates on a single input channel, reducing the operation to a vector-vector multiplication that leaves most crossbar cells idle. yolov5n achieves peak speedups of \(\require{physics} \qty{3.1}{\times}\) on ARM and \(\require{physics} \qty{1.9}{\times}\) on x86. For most models, scaling beyond 100 crossbars yields diminishing returns. As mvm latency increases, speedups decrease proportionally across all models.

5.2 Operator Allocation↩︎

5 shows the latency share and operator counts for yolov5n, which is representative of all evaluated models.

On ARM with 25 crossbars at 50 ns mvm latency, only 16 of 251 operators are offloaded, as the crossbar budget cannot accommodate more layers. Scaling to 100 crossbars nearly quadruples the offloaded count to 59, more than halving the total latency from 270 ms to 127 ms. Increasing to 200 crossbars does not change the allocation, confirming that crossbar scaling saturates once all compute-intensive layers are covered. A higher mvm latency reduces offloading drastically, as it becomes less profitable. For instance, at 1000 ns mvm latency and 100 crossbars, only 33 operators are offloaded.

On x86, the stronger host performance leads to more conservative offloading. At 50 ns mvm latency and 100 crossbars, only 48 operators are offloaded compared to 59 on ARM. Furthermore, the impact of a high mvm latency is more severe. At 1000 ns mvm latency, offloading reduces to 6 operators for all crossbar counts.

This confirms that the ilp adapts to platform and hardware parameters by balancing computation and communication costs.

5.3 Scalability of the ilp Approach↩︎

1 summarizes the ilp sizes and solver metrics on ARM. The ilps of resnet-18 and MobileNetV2 are solved in under 0.01 s with a mean mip gap below 0.1 %. The larger resnet-50 converges in 0.24 s with a mean mip gap of 0.37 %. This confirms that larger linear networks remain manageable.

yolov5n exhibits a substantial increase in problem complexity due to its heavily branched architecture. Despite having only \(\require{physics} \qty{3.2}{\times}\) more operators than resnet-50, its branching topology leads to \(\require{physics} \qty{33}{\times}\) more decision variables and \(\require{physics} \qty{17}{\times}\) more constraints. Both arise from the complex execution orderings across parallel paths. The solver often reaches the 600 s timeout, yielding solutions with a mean mip gap of 6.97 %. As discussed in 2.2, the lp relaxation bound may be significantly looser than the true ilp optimum, so these solutions may still be optimal or near-optimal.

These results show that graph branching, rather than operator count, drives ilp complexity and solving time.

6 Conclusion↩︎

We presented an ilp-based ml workload partitioning framework for heterogeneous systems comprising a host cpu and an rram-based cim accelerator. It combines empirical host cpu profiling with analytical cim and bus latency models. The ilp formulation minimizes the end-to-end inference latency under rram memory budget and operator support constraints, exploiting both intra-accelerator and inter-device parallelism.

Our experiments on resnet-18, resnet-50, MobileNetV2, and yolov5n demonstrate speedups of up to \(\require{physics} \qty{30.9}{\times}\) over an ARM edge cpu and \(\require{physics} \qty{7.3}{\times}\) over a high-performance x86 cpu. The optimal strategy consistently offloads compute-intensive convolutions. The dse shows that the mvm latency dominates hardware sensitivity, while crossbar scaling saturates once all compute-intensive layers are covered. The ilp formulation scales well for linear and sparsely branched architectures and remains practical for all evaluated edge-relevant models. Future work could explore heuristic or hybrid methods to reduce solving time for complex branching topologies.

References↩︎