Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals

1 Introduction↩︎

Deep Neural Networks (DNNs) have been widely deployed on edge devices to eliminate the issues of data confidentiality breaches and unstable network bandwidth in accessing cloud servers [1]. However, current models grow in complexity to improve accuracy, rendering them unsuitable for resource-limited edge devices. Neural architecture search (NAS) can design customized DNN models for edge devices, but it is time-consuming and requires heavy engineering efforts [2]. Model compression is promising to improve DNNs’ efficiency on edge devices. Many DNN applications, such as virtual/augmented reality, and autonomous driving, demand strict response latency, thus, the inference latency of DNNs should be treated as a hard constraint in model compression.

Model compression based on real latency provides additional benefits from better exploration of hardware features [3]. However, traditionally, considering latency as a compression objective requires interaction with the physical device to obtain the real latency, which is time-consuming. Each latency measurement process may take several minutes [4], making it prohibitively expensive in large design spaces. Some approaches are proposed to avoid the high cost of measuring latency, including using look-up tables (LUTs) [3], [4] or hardware simulators [5], [6] to predict the latency. However, LUTs cannot predict the latency of new models that are not recorded in the table; and it is difficult to build hardware simulators by analyzing the device resource and scheduling algorithms due to the opacity of most existing hardware. Thus, LUTs and hardware simulators are unsuitable for various existing edge devices and model compression.

Recently, some learning/regression-based methods [7], [8] are proposed to estimate the inference latency of models on different devices. They first sample some model layers and measure them on the device to obtain precise latency, creating a dataset. Then, these methods use machine learning or regression analysis techniques to derive the patterns from the dataset to predict the latency of unseen models. However, the sampling and measuring remain expensive and can take up to 4.4 days for nn-Meter [7]. Furthermore, unlike NAS, during model compression, the number of input channels and output channels of each layer changes continuously, thus the search space is much huger, rendering these methods unaffordable.

To address the aforementioned problems, we present a novel framework to accurately predict the model inference latency upon targeted hardware utilizing Multi-Layer Perceptron (MLP) neural network [9] and self-evolution scheme. Specifically, the main contributions of this letter are summarized:

1. We present a novel and efficient latency predictor, named EvoLP, to accurately estimate the inference latency of DNN models on diverse edge devices.

2. We reduce the search space to include only the to-be-compressed model and the proposed method can automatically analyze this model to generate a small number of samples, greatly improving the building efficiency.

3. We proposed an evolution scheme for our predictor during model compression, allowing it to evolve for higher precision without interrupting the compression process.

4. We evaluate EvoLP on three edge devices and show that it achieves much higher prediction accuracy compared to existing approaches. And we also demonstrate the benefits of EvoLP in the model compression technique.

2 Background & Related Works↩︎

Complex models require compression to enable efficient inference on low-power and resource-limited edge devices [1]. Recent researches [10] propose to compress the model with one training process to reduce the training overhead. When used in real-time edge systems, it is essential to ensure that the compressed model meets the system latency constraint. Meanwhile, as introduced in Section 1, the costly on-device latency measurement must be replaced with latency predictors.

Latency predictors based on LUTs [4] and hardware simulators [5] are unsuitable in compressing models for various real-time edge systems. Also, predictors based on floating point operations (FLOPs) tend to be highly inaccurate as they do not account for runtime optimization of various operations [7]. Meanwhile, recent predictors, such as nn-Meter [7] and TALM [8], can predict kernel-level latency while considering some optimizations of different operations during runtime. However, these methods require lots of sampling for training the predictor, especially used for model compression.

DNN models are typically composed of a limited set of operations, such as convolution, batch normalization, activation functions, multiplication, etc. However, the parameter configuration range of each operation is infinite [11]. Therefore, even if only predicting the latency of each operation, it poses a significant challenge due to the extensive search space involved. Specifically, considering the commonly used configuration range of convolution in all layers of different models, which encompasses input size (\(1-299\)), input channel (\(1-1024\)), kernel size (\(1-7\)), number of kernels (\(1-1024\)), stride (\(1-3\)), and padding (\(0-1\)), the search space is approximately \(1.3 \times 10^{10}\). Consequently, creating a training set from such a massive search space is highly challenging, and the prediction performance is typically poor when dealing with unseen configurations. Thus, in this letter, we propose to generate a small number of samples for measuring by only considering the model under compression, which accelerates the building process. Moreover, the proposed predictor can evolve during the compression process to achieve higher precision, particularly near the latency constraint.

3 Self-Evolving Latency Predictor↩︎

Traditional latency predictors estimate the latency of models by sampling from the total search space and learning their patterns, but they present two major drawbacks, especially used for model compression. Firstly, the search space is extensive, and numerous samples must be collected, leading to the time cost of measuring these samples on the device being prohibitively expensive. Also, the large search space poses a significant challenge in learning the latency patterns and decreases the precision. Secondly, while some existing methods consider model runtime optimization, some runtime features remain unaccounted for. For instance, the time of data transfer between layers, the input and output time, the parallel processing of branches, etc. In light of these two drawbacks, this letter proposes a model-specific self-evolving latency predictor to efficiently and accurately predict the on-device latency of a model during model compression.

Our latency predictor consists of three processes: model sampling (left), latency predictor generation (right-top), and latency predictor evolution (right-bottom), as shown in Fig. 1. The first two processes are for constructing the predictor, and they reduce the search space to the model under compression, improving both the building efficiency and prediction precision of the latency predictor. The last process is for the evolution scheme and can efficiently learn various model runtime features on the device, enabling accurate prediction of models with varying degrees of compression. We provide details of each phase in the following sections and demonstrate the usage of our latency predictor in model compression.

The first phase is the model sampling and we only need to know the architecture of the model to be compressed without training. Then according to the latency variation throughout the compression process, the layers are categorized into variable layers and const layers. Specifically, if the latency variation of a layer during the compression procedure is trivial, that layer is identified as a const layer. Conversely, it is classified as a variable layer. On-device latency is directly measured and recorded for const layers. For variable layers, we will sample according to the configuration ranges of each layer. As models only include limited layers, the sampling is highly efficient, taking only a few milliseconds. After sampling, we construct a layer pool that comprises thousands of single-layer models with varying operations (e.g., Convolution, Pooling) and configuration sizes, to train the latency predictor. The latency of a single layer typically falls within the range of a few milliseconds, and therefore, the process of measuring all models in the pool is completed within a few hours.

Subsequently, we generate a latency predictor for single layers by training a lightweight three-layer MLP neural network [9], [10]. The forward propagation of this network is shown in Eq. (1 ), where \(x\) denotes the input vector, \(W^L\) represents the weight matrix, \(L\) is the index of layers, and \(\sigma^L\) is the activation function at layer \(L\). Due to the simplicity of this network, only a few hundred multiplication and addition operations are enough to obtain the predicted latency. The updating scheme of the MLP network is also simple, which makes the training process efficient. As MLP can learn the patterns in the training data [9], the trained MLP can predict the latency of unseen layers by providing their configurations. Thus, we obtain a layer-wise latency predictor (LWLP).

\[\label{bp} g(x) = \sigma^2(W^2\sigma^{1}(W^{1} \sigma^0(W^0x)))\tag{1}\] \[\label{em} \begin{align} P = \sum_{i=1}^{n} ReLU(T_i + \sum_{m=0}^{k}a_{im}*T_i^m) \end{align}\tag{2}\]

To predict the overall latency of the entire model when compressed, we first get the real latency of all const layers (CL). Then, for each variable layer, its current configuration is passed to the LWLP to obtain its predicted latency. So far, we obtain the latency of all layers under current compression, denoted as \(T_n, (n=1,2,...)\). To further improve the precision of the overall model latency prediction, an evolution matrix (EM) is employed. The EM is all \(a_{im}\) in Eq. (2 ), and they are trainable parameters for the overall predicted latency (\(P\)) closer to the overall real latency (\(R\)). This approximation is similar to Taylor expansion, and it is used to emulate the edge device runtime features. From our experiments, the approximation is accurate enough under \(k=1\). Meanwhile, the ReLU activation layer is used for predicting the parallel execution of the branch layers.

To provide a clearer understanding of the usage and evolution of our latency predictor, Fig. 2 depicts the model compression process that integrates our latency predictor. As previously mentioned, compression-in-one-training is more promising and is implemented in a soft-compression way [10]. Specifically, unimportant weights are updated to 0 rather than being directly removed from the model. The purpose is that these zeroized weights are not eliminated prematurely and may become important in future training processes. Fig. 2 illustrates the compression training process in chronological order, with the oblique rectangles representing the weights, and different colors indicating different layers. The importance factors (i.e., \(a_1\), \(a_2\), ..., \(b_5\), \(b_6\)) indicate the relative significance of the weights, and weights with small importance factors are “removed" to compress the model.

The compression algorithm first sorts the importance factors and utilizes the latency predictor to identify a suitable compression ratio that meets the latency constraints, while simultaneously recording the predicted latency (\(p\)). The importance factors in the white dashed box correspond to the weight that needs to be “removed". Subsequently, an automatic script [12] is employed to measure the real latency of the compressed model on the target edge device. It is worth noting that the latency test and model training are conducted separately on the cloud and edge device, respectively, and are executed in parallel without any mutual interference. In the next training epoch, the rank of importance factors may change, requiring the latency predictor to determine a new compression ratio. However, at this stage, the edge device still tests the latency of the previous model, thus, the automatic measurement is not triggered to obtain the real latency of the current model.

Upon completion of the on-device measurement, the pair of the real latency (i.e., \(r_1\)) and its corresponding predicted latency (i.e., \(p_1\)) is used to update the EM by stochastic gradient descent. In subsequent training epochs, the updated latency predictor is invoked to find the compression ratio. However, to ensure the convergence of the EM, we still record the uncorrected prediction latency (i.e., \(p_2\)) in the pair. In each update of EM, all recorded latency pairs are utilized, and this process is repeated until the end of training. After training, unimportant weights are removed to obtain a high-accuracy model that satisfies the latency constraints. The evolution process of the latency predictor also offers an additional advantage. In each epoch, the predicted latency is close to the latency constraint, which enables the predictor to become more acquainted with the device execution near the constraint by updating the EM. Consequently, our latency predictor becomes more precise in predicting latency near the constraint.

Furthermore, the proposed latency predictor is not limited to model compression, but can also be used in latency-constraint NAS, etc. Besides, this method can be extended beyond latency prediction and utilized to build power consumption predictors, accuracy predictors, and others for various devices. We will further explore this aspect in our future research.

4 Experimental Evaluation↩︎

4.1 Experimental Setup↩︎

We evaluate EvoLP on four DNNs (VGG, ResNet, Inception, MobileNet) and on three edge devices (one CPU platform – ARMv8 rev 3; and two GPU platforms – 1: NVIDIA Pascal GPU; 2: NVIDIA Maxwell GPU). For a comprehensive comparison, we employ five widely-used baselines, FLOPs, FLOPsE (FLOPs predictor with evolution scheme), ZBNLP (utilizing MLP and more training data) [10], nn-Meter [7], and TALM [8], each of which can be used for estimating model latency in model compression. For FLOPs and FLOPsE, we use FLOPs with/without the evolution scheme to estimate latency, for ZBNLP, we use its open-source code. And we refer to the results reported in the papers of nn-Meter and TALM for comparison. To evaluate the prediction performance, we employ three widely-used metrics in regression: the Root Mean Square Error (RMSE), the relative Root Mean Square Percentage Error (RMSPE), and the ±5% accuracy (the proportion of models with predicted latency within a ±5% error margin). Better prediction performance is indicated by lower RMSE, RMSPE, and higher ±5% accuracy. Finally, we show the benefits of our predictor to model compression, in terms of model accuracy and inference latency.

4.2 Prediction Evaluation↩︎

Our proposed predictor incorporates two key techniques: the building method (i.e., utilizing MLP) and the evolution scheme during model compression. To demonstrate the distinct impacts of these two methods, we present a comparison of our approach with FLOPs, FLOPsE, and ZBNLP in a model compression process, as shown in Fig. 3. Considering the stringent latency constraints inherent to real-time edge systems, we employ a ±5% accuracy for evaluating these methods. From Fig. 3, we can find that our method achieves the highest ±5% accuracy across various devices and model variants (A variant means a model under different compression ratios, represented by +s). Furthermore, the accuracy curves of ZBNLP and EvoLP reveal that the evolution scheme significantly enhances the precision of the latency predictor. A comparison between FLOPsE and EvoLP demonstrates the superior effectiveness of the MLP-based method for constructing the layer predictor.

3pt

3pt

In addition, we present an evaluation of the final EvoLP, compared with nn-Meter and TALM (TALM only reports ±10% accuracy) in TABLE 1³. The results from TALM show that predicting latency for Edge GPUs is more challenging compared to Desktop GPUs. Compared to the results of TALM for Edge GPUs, we can find that our approach attains considerably higher ±5% accuracy and lower RMSE values. Meanwhile, when compared to nn-Meter, our method achieves higher ±5% accuracy across the majority of devices and models. This further substantiates the efficacy of our approach in model compression under strict latency constraints.

4.3 Model Compression with EvoLP↩︎

Finally, we integrate our latency predictor into a model compression framework and present the final model accuracy under varying latency constraints on diverse devices using the ImageNet-100 dataset [10], as shown in TABLE 2. The upper part in this table represents the “hard" latency constraint, and the lower part is the achieved model accuracy while meeting the constraint by model compression with our latency predictor. The term”Baseline" represents the model without compression, while “Comp." is the model compressed under different latency constraints. Due to the latency predictor’s error, we employ real latency measurements in the final epochs of the model compression process to ensure meeting latency constraints. TABLE 2 shows that, for a specific model, larger latency corresponds to higher accuracy, suggesting that it is crucial to approach the latency constraint as closely as possible. Consequently, our precise latency predictor enables the model compression framework to be close to latency constraints, achieving higher model accuracy in one training.

5 Conclusion↩︎

In this letter, we proposed a novel framework called EvoLP, which utilized an MLP neural network and self-evolution scheme to accurately predict the model inference latency on targeted hardware. EvoLP can automatically analyze the model to generate a small number of samples for measuring, thus accelerating the building process and enhancing the prediction precision. By introducing the EM, EvoLP can evolve during the compression process to further improve precision, particularly near the latency constraint. Evaluation results showed that EvoLP achieved much higher prediction precision compared to existing approaches. We also demonstrated the benefits and effectiveness of EvoLP in the model compression technique. EvoLP accurately predicted the on-device latency, facilitating the deployment of DNNs on real-time edge systems.

References↩︎