1 Introduction↩︎

In recent years, deep neural networks have achieved unprecedented success across a wide variety of domains such as computer vision, natural language processing, and automatic speech recognition. Unfortunately, as these networks continue to improve and surpass human-level performance, the computational resources and memory usage required also increases as the architecture becomes more complex. To reduce the model size and accelerate inference operations, many researchers have attempted diverse compression techniques such as network quantization [1] and network pruning [2]. In this paper, we concentrate on network quantization due to the advantage that INT4 or INT8 quantization allows us to accelerate quantized neural networks using off-the-shelf accelerators such as the NVIDIA A100 Tensor Core GPU [3] or ARM Cortex MCUs [4].

Network quantization techniques can be broadly divided into two categories: quantization-aware training (QAT) and post-training quantization (PTQ). QAT is a method where the quantization of the networks is incorporated during the trianing process, as proposed by various research works such as [5]–[9]. We note that QAT results in a marginal performance difference between the full-precision and quantized versions of the neural network. Yet, QAT requires end-to-end retraining or fine-tuning on a full training dataset, which often causes an enormous amount of time and resources to obtain a quantized neural network with competitive performance. Furthermore, a whole training dataset may not be available due to data privacy issues or demands to utilize legacy models. Such drawbacks of QAT are the reasons why researchers recently pay more attention to PTQ [10]–[12] that needs neither a full training dataset nor end-to-end learning at all.

PTQ had been initially performed via rounding-to-nearest by minimizing the quantization error in the parameter space. However, this approach suffers from severe performance degradation. Since it is reported that the loss degradation resulting from quantization can be approximated as the second-order error in Taylor Expansion by viewing quantized weights as perturbed weights, [13] and [14] substantiate that reconstructing each output of layer or block is equivalent to minimizing the approximation of loss degradation resulting from quantization under some assumptions. Accordingly, recent works [13]–[16] have suggested to reconstruct each output of layer or block by devising and learning a new weight-rounding scheme, deviating from rounding-to-nearest, as an effort to preserve the performance of a full-precision model even after PTQ. However, all those new rounding schemes designed in existing studies either round or quantize pre-trained weights adaptively via element-wise addition.

We propose a novel post-training weight quantization method, called FlexRound, which departs from the typical element-wise addition approaches and instead employs an element-wise division perspective. By jointly learning a common quantization grid size and the division factor for pre-trained weights, FlexRound offers a new approach to PTQ. Interestingly, thanks to the reciprocal rule of derivatives induced by element-wise division, FlexRound can inherently leverage pre-trained weights when updating an individual scale for each pre-trained weight. Specifically, we corroborate that a relatively wider range of discrete values needs to be explored when quantizing pre-trained weights of large magnitude. The rationale behind such an approach is that the magnitude of a weight can be interpreted as its relative importance within the network. Given that weights of larger magnitude have a greater impact on the network’s performance than those of smaller magnitude, as demonstrated by research such as [2], to maintain the performance of a pre-trained model even after quantization, it is important to relax the constraints associated with quantizing weights of large absolute value compared to those of small absolute value (i.e., important weights can be quantized to one of not only its two nearest discrete values but also to discrete values further away from it). Accordingly, FlexRound can quantize pre-trained weights flexibly depending on their own magnitudes, thereby leading to better performance.

Our contributions are threefold:

• We propose FlexRound as a new rounding scheme for post-training weight quantization based on the principle of element-wise division in order to allow for jointly learning not only a separate scale for every pre-trained weight but also a common quantization grid size across a group (e.g., a channel or a layer).

• We theoretically and empirically demonstrate that such a new rounding scheme based on element-wise division takes into consideration the magnitude of pre-trained weights when updating their corresponding scales so that FlexRound can quantize pre-trained weights of large magnitude (i.e., important pre-trained weights) more flexibly than rounding either up or down only.

• To the best of our knowledge, we are the first to perform extensive experiments in a per-tensor uniform PTQ setting on natural language generation as well as image classification and natural language understanding, using numerous models such as ResNet, MobileNetV2, BERT, GPT-Neo, OPT, and GPT-2. We also, for the first time, conduct the uniform PTQ reconstruction for large language models like LLaMA on both common sense reasoning and causal language modeling tasks.

2 Related Work↩︎

Recently, many researchers have attempted to quantize a wide range of models for various tasks such as computer vision and natural language understanding/generation without any (re)training. Outlier channel splitting (OCS) [10] replicates channels entailing outliers, and then, halves outliers of those channels. Despite the fact that OCS explicitly addresses outliers, it still experiences severe accuracy degradation when both weights and activations are quantized to low-bit. As an alternative solution, [11] proposed Bit-Split that splits an integer into several bits and optimizes them separately. While the performance of Bit-Split is comparable to that of a full-precision model in a low-bit setting, it may not be as effective for certain architectures such as MobileNetV2.

To overcome the limitations discussed above, [13] and [15] minimize the mean squared error (in a layer-by-layer fashion) between the full-precision layer’s output and its quantized layer’s output by inventing and learning a new weight-rounding mechanism dubbed as AdaRound and AdaQuant, respectively. As such a layer-wise reconstruction error minimization opens the door to \(4\)-bit PTQ regime, [14] proposed block-wise reconstruction, titled BRECQ, to consider cross-layer dependency along with the possibility of fully quantizing MobileNetV2 to \(4\)-bit. In addition to block-wise reconstruction, [16] proposed QDrop that drops the quantization of activations at random during the PTQ reconstruction to induce activation quantization to be synchronized with weight quantization. Both BRECQ and QDrop, however, are based on AdaRound that rounds weights only either up or down at most with a ‘fixed’ quantization grid size. AdaQuant can simultaneously learn a quantization grid size and quantize weights adaptively, but incurs severe performance degradation when quantizing MobileNetV2 in low-bit regimes. As another line of PTQ research, some PTQ techniques are exclusively specialized in quantizing language models such as BERT and GPT-like models. [17] first applied PTQ to BERT by introducing a per-embedding-group activation quantization scheme to deal with highly dynamic activation ranges. [18] studied the PTQ reconstruction in parallel for BERT. [19] proposed ZeroQuant that quantizes BERT and GPT-3 in a group-wise weight quantization manner driven by token-wise activation quantization via layer-by-layer knowledge distillation (while a dedicated CUDA kernel is required for ZeroQuant). [20] quantizes large language models (LLMs) like OPT with vector-wise weight quantization and mixed-precision decomposition with FP16 activations. To avoid the use of FP16 activations, [21] proposed SmoothQuant that shifts the difficulty of activation quantization to weight quantization, allowing for INT8 quantization of both weights and activations in LLMs. Unfortunately, both [20] and [21] assume that the outliers in activations would appear in a certain pattern.

Most of the aforementioned PTQ studies are targeted to either vision models or language models only, not to both. To the best of our knowledge, our work is the first to carry out extensive experiments on diverse tasks ranging from image classification and natural language understanding to natural language generation under a per-tensor uniform PTQ setting. Additionally, we for the first time show that LLMs can be efficiently quantized, with only a minor impact on accuracy compared to half-precision baselines, attained by reconstructing each block output, without the assumption that the activation outliers would appear in a certain pattern.

3 Methodology↩︎

This section begins by introducing the notations used throughout the paper and the background of post-training quantization (PTQ). We then provide the concept and design of FlexRound for the uniform PTQ reconstruction method. We finally delve into the advantages of utilizing the principle of element-wise division in FlexRound.

3.1 Preliminaries↩︎

3.1.0.1 Notations

A scalar, a vector, and a matrix (or a tensor) are expressed as a non-bold letter, a small bold letter, and a capital bold letter (e.g. \(s\), \({\boldsymbol{s}}\) and \({\boldsymbol{S}}\)) respectively. \(\widehat{{\boldsymbol{W}}}\) indicates the quantized counterpart of \({\boldsymbol{W}}\). The input to a 2D convolution or a linear layer is represented as \({\boldsymbol{X}}\) if all previous layers are intact, or as \(\widetilde{{\boldsymbol{X}}}\) if all previous layers are quantized. The entries of a matrix \({\boldsymbol{A}}\) are denoted as \(A_{(i, j)}\), while the entries of a 4-dimensional tensor \({\boldsymbol{A}}\) are denoted as \(A_{(i, j, k, l)}\). We let \(\odot\) and \(\mathbin{/}\) indicate element-wise product and element-wise division, respectively, similar to the broadcasting process in Python Numpy. \(\lfloor \cdot \rceil\) and \(\lfloor \cdot \rfloor\) express the rounding function and the floor function. \(|| \cdot ||_F\) represents the Frobenius norm.

3.1.0.2 PTQ Background

The conventional uniform PTQ approach is to quantize pre-trained weights \({\boldsymbol{W}}\) to be \(\widehat{{\boldsymbol{W}}} = s_1 \Big\lfloor {{\boldsymbol{W}}\over s_1} \Big\rceil\) via rounding-to-nearest, where a quantization grid size \(s_1\) \(\in \mathbb{R}_{> 0}\) is set to minimize \(\|{\boldsymbol{W}}- \widehat{{\boldsymbol{W}}}\|^2_F\), but the minimization of the quantization error in the parameter space is not equivalent to that of the final task loss. As [14] proves that the loss degradation resulting from quantization can be approximated as the quadratic form of the network output and its Hessian matrix, several studies have strove to minimize \(\|{\boldsymbol{W}}{\boldsymbol{X}}- \widehat{{\boldsymbol{W}}}\widetilde{{\boldsymbol{X}}}\|^2_F\) layer-by-layer or block-by-block with respect to continuous variables \({\boldsymbol{V}}\) with a small amount of data, where \(\widehat{{\boldsymbol{W}}}\) is either \(s_1 (\lfloor {{\boldsymbol{W}}\over s_1} \rfloor + h({\boldsymbol{V}}))\) with a certain function \(h(\cdot)\) [13] or \(s_1 \Big\lfloor {{\boldsymbol{W}}+ {\boldsymbol{V}}\over s_1} \Big\rceil\) [15]. However, all these rounding mechanisms are founded on element-wise addition.

3.2 FlexRound↩︎

Unlike prior works based on element-wise addition, we exploit element-wise division for quantizing pre-trained weights. We can formulate our proposed weight-rounding scheme based on element-wise division as follows: \[\begin{align} \widehat{{\boldsymbol{W}}} = s_1 \Big\lfloor {{\boldsymbol{W}}\over {\boldsymbol{S}}} \Big\rceil, \label{eq:flexround} \end{align}\tag{1}\] where \({\boldsymbol{S}}\) is the division factor for \({\boldsymbol{W}}\) whose shape is equal to that of \({\boldsymbol{W}}\) while all entries of \({\boldsymbol{S}}\) as well as \(s_1\) are positive and learnable. Similarly to preceding studies, both \(s_1\) and \({\boldsymbol{S}}\) are updated as an attempt to minimize \(\|{\boldsymbol{W}}{\boldsymbol{X}}- \widehat{{\boldsymbol{W}}}\widetilde{{\boldsymbol{X}}}\|^2_F\).

Eq. 1 indicates that the basic formula of FlexRound supports per-tensor uniform PTQ. Although FlexRound can also adopt per-channel weight quantization by simply replacing a scalar \(s_1\) with a vector \({\boldsymbol{s}}_1\), as we show later, per-tensor uniform PTQ (via FlexRound) can be sufficient to achieve the performance of a full-precision model. Therefore, we focus on the per-tensor uniform PTQ reconstruction unless otherwise specified. The overall procedure of FlexRound in a per-tenor uniform PTQ setting is described in Figure 1.

Let us discuss how to design \({\boldsymbol{S}}\) in detail. We first start formulating \({\boldsymbol{S}}\) as \({\boldsymbol{S}}= s_1 \odot {\boldsymbol{S}}_2\), where \({\boldsymbol{S}}_2\) is the matrix or tensor scaling \({\boldsymbol{W}}\) whose shape is equal to that of \({\boldsymbol{W}}\) while every element of \({\boldsymbol{S}}_2\) is positive and learnable. When \({\boldsymbol{S}}= s_1 \odot {\boldsymbol{S}}_2\), Eq. 1 is enough to perform well compared to existing weight-rounding schemes based on element-wise addition in a per-tensor uniform PTQ setting, as we show later. However, to further improve the performance of a new weight-rounding scheme based on element-wise division, we complement \({\boldsymbol{S}}_2\) as follows. For a linear layer \({\boldsymbol{W}}\in \mathbb{R}^{C_{out} \times C_{in}}\), \({\boldsymbol{S}}_2\) is complemented with an additional learnable tensor \({\boldsymbol{s}}_3 \in \mathbb{R}_{> 0}^{C_{out} \times 1}\). Motivated from a wide acknowledgement that the statistics of output channels can vary greatly [22], [23], we take into account the variation of output channel’s statistics by supplementing \({\boldsymbol{S}}_2\) with \({\boldsymbol{s}}_3\). For a 2D convolution \({\boldsymbol{W}}\in \mathbb{R}^{C_{out} \times C_{in} \times H \times W}\), in particular, \({\boldsymbol{S}}_2\) is complemented with two additional learnable tensors \({\boldsymbol{s}}_3 \in \mathbb{R}_{> 0}^{C_{out} \times 1 \times 1 \times 1}\) and \({\boldsymbol{s}}_4 \in \mathbb{R}_{> 0}^{1 \times C_{in} \times 1 \times 1}\). Hence, \({\boldsymbol{S}}\) is formulated as \(s_1 \odot {\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3\) (as illustrated in Figure 2) for a linear layer, or as \(s_1 \odot {\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3 \odot {\boldsymbol{s}}_4\) for a 2D convolution so that Eq. 1 is transformed into \[\widehat{{\boldsymbol{W}}} = \begin{cases} s_1 \Big\lfloor {{\boldsymbol{W}}\over {s_1 \odot {\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3}} \Big\rceil \quad\,\, \text{ for a linear layer} \\ s_1 \Big\lfloor {{\boldsymbol{W}}\over {s_1 \odot {\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3 \odot {\boldsymbol{s}}_4}} \Big\rceil \,\text{for a 2D convolution} \end{cases}\label{eq:flexround2}.\tag{2}\] We refer to Eq. 2 as ‘FlexRound.’ Here, every element of \({\boldsymbol{S}}_2\), \({\boldsymbol{s}}_3\), and \({\boldsymbol{s}}_4\) is initialized to 1 in order to facilitate learning from the traditional rounding-to-nearest method, namely, \(s_1 \Big\lfloor {{\boldsymbol{W}}\over s_1} \Big\rceil\). All parameters (\(s_1\), \({\boldsymbol{S}}_2\), \({\boldsymbol{s}}_3\), and \({\boldsymbol{s}}_4\)) are updated to minimize \(\|{\boldsymbol{W}}{\boldsymbol{X}}- \widehat{{\boldsymbol{W}}}\widetilde{{\boldsymbol{X}}}\|^2_F\) subject to the constraint that all entries of \(s_1\), \({\boldsymbol{S}}_2\), \({\boldsymbol{s}}_3\), and \({\boldsymbol{s}}_4\) are positive.

In Eq. 2 , element-wise division serves a similar purpose as element-wise addition in creating a more effective rounding scheme than rounding-to-nearest. By implementing such a new rounding policy through element-wise division, we can make \(s_1\), \({\boldsymbol{S}}_2\), \({\boldsymbol{s}}_3\), and \({\boldsymbol{s}}_4\) all learnable. This allows FlexRound to learn a common quantization grid size (i.e., \(s_1\)) jointly with the rounding process (e.g., \({\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3\) or \({\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3 \odot {\boldsymbol{s}}_4\) in FlexRound). Furthermore, the reciprocal rule of derivatives induced by element-wise division enables FlexRound to leverage pre-trained weights when learning the corresponding scales, as demonstrated both theoretically and empirically by the following proposition.

Proposition 1 implies that, for a linear layer, an element \(S'_{(i, j)}\) is (partially) affected by \(W_{(i, j)}\) so that \(\overline{W}_{(i, j)} = \Big\lfloor{W_{(i, j)} \over {s_1 \odot S'_{(i, j)}}}\Big\rceil\) can also be updated and influenced by \(W_{(i, j)}\). In other words, as the magnitude of a pre-trained weight \(W_{(i, j)}\) is larger, the chance of \(\overline{W}_{(i, j)}\) receiving a larger update during the PTQ reconstruction becomes higher. The magnitude of a weight can be regarded as a metric to measure the importance of a weight for pruning unimportant weights [25]. Consequently, weights of larger magnitude play a more important role than those of smaller magnitude [2]. To reduce the performance gap between a full-precision pre-trained model and its quantized version, it would be reasonable to relax the constraint on quantizing pre-trained weights of large magnitude (i.e., potentially important pre-trained weights) by allowing them to have higher chances of being quantized to one of not just the two closest quantization grids but also more distant ones than those of smaller magnitude. The above implication is also identically applicable to a 2D convolution.

Figure 3 shows the amount of weight updates via FlexRound for MobileNetV2 and ResNet-18. On the left side and the center side of Figure 3, histograms describe the change of \(\overline{W}_{(i, j, k, l)}\) grouped for small pre-trained weights (\(|W| < 1\), left) and large pre-trained weights (\(|W| > 1\), center). On the right side, scatter plots show the amount of grid shifts from the grids obtainable from rounding-to-nearest (RTN). We note that MobileNetV2 and ResNet-18 are quantized distinctively due to FlexRound. For example, in the case of MobileNetV2 as in Figure 3 (a), the change of \(\overline{W}_{(i, j, k, l)}\) attained by minimizing \(\mathcal{L}\) is more aggressive (i.e., rounding can be deviated from more than one-step up or one-step down) when the absolute value of \(W_{(i, j, k, l)}\) is larger than one, which means that FlexRound more flexibly quantizes pre-trained weights of large magnitude as illustrated in red dotted squares in Figure 3 (a). The amount of aggressively rounded weights in the first 2D convolution of the first block of MobileNetV2 is around \(12.8\%\) of the total. For ResNet-18, however, there are no pre-trained weights whose magnitudes are larger than one. Thus, most pre-trained weights are rounded either up or down as seen in Figure 3 (b) (e.g., only about \(1.5\%\) weights are rounded aggressively in the first 2D convolution of the first block of ResNet-18). Different rounding results of AdaRound, AdaQuant, and FlexRound are visually compared in Appendix 6.

\begin{table}[t]

-0.2in \end{table}

Even if FlexRound takes into account the magnitude of pre-trained weights when updating their corresponding scales, one might question that FlexRound seems to quantize pre-trained weights of moderate magnitude more flexibly than those of large magnitude as seen in the right side of Figure 3. Our aim with FlexRound is to emphasize that pre-trained weights with relatively larger magnitude are more likely to be quantized with higher flexibility compared to those with relatively smaller magnitude. As explained in Appendix 7, \(\left| {{\partial\mathcal{L}} \over {\partial S'_{(i, j)}}} \right|\) is directly proportional to \(\left| W_{(i, j)} {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \right|\). No matter how large the magnitude of \(W_{(i, j)}\) is, if \(\left| {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \right|\) is close to zero, \(\left| {{\partial\mathcal{L}} \over {\partial S'_{(i, j)}}} \right|\) would be also zero. In this sense, pre-trained weights of large magnitude can be quantized to the grids obtainable from RTN. If \(\left| {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \right|\) is (significantly) larger than zero, pre-trained weights of large magnitude can be quantized to the grids far from two nearest ones as seen in Figure 4. In short, while the magnitude of pre-trained weights influences the updates to their corresponding scales in FlexRound, it does not necessarily imply that larger weights must be quantized more flexibly than smaller ones.

Note that FlexRound can quantize weights more flexibly as the bit-width increases. Comparing the right side of Figure 3 with Figure 5, FlexRound can provide more grid shifts from the grids obtainable from RTN as a higher bit-width is used. Unlike AdaRound that must round weights either up or down regardless of the number of bits used, FlexRound enables more flexible weight quantization as the bit-width increases, thus being better suited for quantizing models that require higher bit-widths (e.g., LLMs) than AdaRound.

\begin{table}[t]

-0.15in \end{table}

\begin{table}[t]

-0.15in \end{table}

4 Experiments↩︎

In this section, we first empirically confirm the importance of learning a quantization grid size \(s_1\) jointly with the rounding process and the distinct contribution of additional tensors \({\boldsymbol{s}}_3\) and \({\boldsymbol{s}}_4\) to FlexRound. Then, we compare the performance of FlexRound with that of the state-of-the-art PTQ methods in a per-tensor uniform PTQ setting in the following cases: image classification on ImageNet [26] with ResNet [27] and MobileNetV2 [28] (Section 4.2), natural language understanding (NLU) on GLUE [29] with BERT [30] and GPT-Neo [31] (Section 4.3.0.1), natural language generation (NLG) on WikiText2 [32] and Penn Treebank (PTB) [33] with GPT-Neo and OPT [34], and NLG on WebNLG [35] with GPT-2 [36] (Section 4.3.0.2). Finally, we validate that large language models (LLMs) can be quantized with only a marginal impact on accuracy compared to half-precision baselines by block-wise output reconstruction, without assuming that the activation outliers would occur in a certain pattern. We study LLaMA [37] by adopting per-channel weight quantization and per-tensor activation quantization for six common sense reasoning benchmarks: BoolQ [38], PIQA [39], HellaSwag [40], WinoGrande [41], ARC easy and challenge [42], and OpenBookQA [43], and the causal language modeling task on WikiText2 (Section 4.3.0.2).

For brevity, we let “B + X” and “Q + X” indicate that a certain rounding scheme ‘X’ is performed in the experimental setup described in BRECQ [14] or QDrop [16], respectively (an experimental setup includes the definition of a block unit for reconstruction error minimization or how much the probability of dropping the quantization of activations is). As introduced in BRECQ and QDrop, we also use the LSQ technique [8] when updating an activation step size for activation quantization. All experimental results are conducted by our own implementation based on open-source codes.

\begin{table}[t]

-0.15in \end{table}

4.1 Ablation Study↩︎

4.1.0.1 Ablation Study 1

Although AdaRound demonstrates the state-of-the-art performance among previous PTQ approaches, it is unable to learn the quantization grid size \(s_1\) jointly with the rounding process, as discussed in Section 2. To understand the significance of learning \(s_1\) jointly with the rounding process, we evaluate the performance of FlexRound with a fixed \(s_1\) (Ablation Study 1 in Table ¿tbl:tab:ablation?) on the ImageNet dataset with weights quantized to \(4\)-bit (activations are not quantized). As seen in Table ¿tbl:tab:ablation?, when \(s_1\) is fixed, FlexRound performs similarly to AdaRound for all models except for ResNet-18. This indicates that regardless of the quantization method used, whether it be AdaRound or FlexRound, using a fixed \(s_1\) prevents further improvements in the performance of the quantized model. However, when learning \(s_1\) jointly with the rounding process, FlexRound outperforms AdaRound for every model. The ability to learn \(s_1\) jointly with the rounding process is a critical aspect in closing the performance gap between a full-precision model and its quantized counterpart. FlexRound possesses this capability in contrast to AdaRound since it is based on element-wise division, as mentioned in Section 3.2.

4.1.0.2 Ablation Study 2

To justify the inclusion of additional tensors \({\boldsymbol{s}}_3\) and \({\boldsymbol{s}}_4\) in FlexRound, we conduct an ablation study in which FlexRound is tested on the ImageNet dataset with weights quantized to \(4\)-bit while keeping activations unquantized, and the results are compared with FlexRound without the use of \({\boldsymbol{s}}_3\) and \({\boldsymbol{s}}_4\) (Ablation Study 2 in Table ¿tbl:tab:ablation?). As shown in the last two rows in Table ¿tbl:tab:ablation?, the presence of \({\boldsymbol{s}}_3\) and \({\boldsymbol{s}}_4\) increases the top-1 accuracy for all models. Interestingly, FlexRound without the use of \({\boldsymbol{s}}_3\) and \({\boldsymbol{s}}_4\) also outperforms both AdaQuant and AdaRound, which would support our claim that a new rounding scheme shifted from element-wise addition to element-wise division is the key to improving the quantization quality significantly.

\begin{table}[t]

-0.15in \end{table}

4.2 ResNet and MobileNetV2 on ImageNet↩︎

We quantize ResNet-18, ResNet-50, and MobileNetV2 in the low-bit PTQ reconstruction with \(1024\) randomly sampled images. Linear symmetric per-tensor quantization format is assumed for quantizing weights and/or activations, whereas in contrast, [14] and [16] adopt linear asymmetric per-channel quantization format, which causes discrepancies between the results obtained in our own implementation of BRECQ and QDrop and those reported in [14] and [16]. For FlexRound, the output of each layer or block is reconstructed during \(5k\) iterations while all learnable parameters (i.e., \(s_1\), \({\boldsymbol{S}}_2\), \({\boldsymbol{s}}_3\), and \({\boldsymbol{s}}_4\)) are updated by using one learning rate (e.g., \(4\)e-\(4\) for the ResNet models quantized by \(3\)-bit or \(4\)-bit, or \(1\)e-\(3\) for the ResNet models quantized by \(2\)-bit and MobileNetV2). The first and last layers are quantized to \(8\)-bit and the batch normalization layer is folded into convolution, as in [14]. Our experiments are performed based on full-precision pre-trained models provided in the BRECQ github repository¹, unless otherwise noted. The experiments based on full-precision pre-trained models available from the official PyTorch repository are given in Appendix 8. We report the median over five random trials.

Assuming the quantization of weights only, we compare FlexRound with AdaRound and AdaQuant, which both utilize the principle of element-wise addition. Table ¿tbl:tab:imagenet95w95brecq? shows that FlexRound consistently outperforms those two addition-based rounding policies. Note that the performance of AdaQuant is inferior to that of AdaRound in Table ¿tbl:tab:imagenet95w95brecq?. Correspondingly, FlexRound would be compared to AdaRound only to save space hereafter. Table ¿tbl:tab:imagenet95wa95brecq? provides model accuracy when AdaRound and FlexRound (quantizing both weights and activations) are associated with the settings of BRECQ or QDrop. It is worth noting that in Table ¿tbl:tab:imagenet95wa95brecq?, FlexRound is particularly effective for MobileNetV2 (which includes weights of large magnitude) for the reasons explained in Section 3.2. It is also interesting to see that even when both weights and activations of the ResNet models are quantized to \(4\)-bit under a per-tensor uniform PTQ setting, the performance degradation (compared to a full-precision pre-trained model) is negligible (less than \(2\%\)) in Table ¿tbl:tab:imagenet95wa95brecq?.

4.3 Language Models↩︎

\begin{table}[t]

-0.15in \end{table}

\begin{table}[t]

-0.15in \end{table}

All language models in this paper are based on the structure of Transformer [44]. To reduce the precision of such models to \(8\)-bit, unless otherwise stated, we employ a linear asymmetric per-tensor quantization scheme for both weights and activations. The reconstruction step for PTQ is applied to each Transformer layer, including both attention and feed-forward sub-layers. All weights in attention and feed-forward sub-layers are quantized to \(8\)-bit. Activations are quantized to \(8\)-bit on-the-fly before each linear layer, while the inputs of the softmax and normalization layers remain at full-precision as suggested in [45] and [46]. We utilize pre-trained language models (PLMs) and datasets from the HuggingFace [47] repository, with the exception of GPT-2 and LLaMA experiments. The experiments for the question-answering task with a fine-tuned BERT on the SQuADv1 [48] dataset is presented in Appendix 12.

4.3.0.1 BERT and GPT-Neo on GLUE

We evaluate the natural language understanding (NLU) performance of FlexRound using a variety of models including \(\text{BERT}_{\text{Base}}\), \(\text{BERT}_{\text{Large}}\), \(\text{GPT-Neo}_{125\text{M}}\), \(\text{GPT-Neo}_{1.3\text{B}}\), and \(\text{GPT-Neo}_{2.7\text{B}}\) fine-tuned on the GLUE benchmark. We only report the experimental results on the MNLI, QQP, and MRPC datasets due to space limit. All experimental results are presented in Appendix 13. The learning rate applied to all learnable parameters (\(s_1\), \({\boldsymbol{S}}_2\), and \({\boldsymbol{s}}_3\)) is selected to be \(2\)e-\(4\) for BERT and to be \(3\)e-\(4\) for GPT-Neo regardless of the task to demonstrate that ‘Q + FlexRound’ can broadly surpass ‘Q + AdaRound’ without the need of significant efforts to select the optimal learning rate for each task. Reconstruction process is performed by using \(1024\) random samples for \(20K\) iterations. The last, randomly initialized layer remains in full-precision. Further experimental details are deferred to Appendix 13. In Table ¿tbl:tab:glue?, we report the performance of ‘Q + AdaRound’ and ‘Q + FlexRound’ that are potentially promising as shown in Table ¿tbl:tab:imagenet95wa95brecq?. We can notice that ‘Q + FlexRound’ yields better NLU scores than ‘Q + AdaRound’ for all models and NLU tasks. In particular, for the MNLI and QQP datasets, ‘Q + FlexRound’ can achieve comparable or even superior performance to a full-precision model in a per-tensor uniform PTQ setting with the exception of \(\text{GPT-Neo}_{125\text{M}}\).

4.3.0.2 GPT-Neo and OPT on WikiText2 and PTB

We test the natural language generation (NLG) performance of FlexRound using fine-tuned PLMs including \(\text{GPT-Neo}_{125\text{M}}\), \(\text{GPT-Neo}_{1.3\text{B}}\), \(\text{GPT-Neo}_{2.7\text{B}}\), \(\text{OPT}_{125\text{M}}\), \(\text{OPT}_{1.3\text{B}}\), and \(\text{OPT}_{2.7\text{B}}\) on the WikiText2 dataset and PTB dataset. Fine-tuned PLMs (for NLG) are quantized by AdaRound and FlexRound in a per-tensor quantization manner with \(128\) random samples drawn from downstream task training data. More details on the experimental setup are provided in Appendix 14. As presented in Table ¿tbl:tab:clm95finetuned?, it is clear that ‘Q + FlexRound’ is superior to ‘Q + AdaRound’ for all models and datasets, which means that FlexRound is also effective for NLG as well as image classification and NLU. Notice that even for the OPT models, the performance of ‘Q + FlexRound’ is close to that of a full-precision model.

4.3.0.3 GPT-2 on WebNLG

To this point, we have applied full fine-tuning for downstream tasks to BERT, GPT-Neo, and OPT. For language models, however, there are various fine-tuning techniques [49]–[51] that can perform better with fewer trainable parameters than full fine-tuning. To evaluate the compatibility of FlexRound with other fine-tuning methods, we perform experiments on quantizing GPT-2 merged with LoRA [51], one of the state-of-the-art fine-tuning methods. We choose \(128\) examples from the training set of WebNLG at random for reconstruction. More experimental details are given in Appendix 15. Table ¿tbl:tab:lora? shows that ‘Q + FlexRound’ excels ‘Q + AdaRound’, and performs similarly or even better than the full-precision model with LoRA. Hence, FlexRound is also compatible with other state-of-the-art fine-tuning techniques in addition to full fine-tuning.

4.3.0.4 LLaMA on Common Sense Reasoning and WikiText2

Finally, we evaluate the zero-shot performance of LLaMA-\(33\)B on six common sense reasoning benchmarks and one casual language modeling task on WikiText2. It is intended to justify that LLMs can be efficiently quantized with only negligible accuracy degradation compared to half-precision baselines by block-by-block reconstructing output, without assuming that the outliers in activations would emerge in a certain pattern. In Table ¿tbl:tab:plm95llama9533b?, for reconstruction, \(512\) samples are randomly selected from the training dataset of C4 [52]. We use linear asymmetric per-channel quantization for weights but linear asymmetric per-tensor quantization for activations. The zero-shot and five-shot performances of LLaMA-\(7\)B, LLaMA-\(13\)B, and LLaMA-\(33\)B as well as those experimental details are given in Appendix 16. Table ¿tbl:tab:plm95llama9533b? shows that ‘Q + FlexRound’ can maintain the accuracy of the half-precision baseline, surpassing ‘Q + AdaRound’. Without any assumption about the activation outliers in LLMs, FlexRound can quantize LLMs while preserving the performance of half-precision baselines.

5 Conclusion↩︎

We propose a new rounding scheme, FlexRound, for post-training weight quantization under the principle of element-wise division, to enable jointly learning both a common quantization grid size and an individual scale for each pre-trained weight. We validate that FlexRound can flexibly quantize pre-trained weights by updating their corresponding scales depending on their own magnitudes. Hence, FlexRound can be applied to various models including even large language models with negligible accuracy degradation.

langley00?

6 Comparison of Rounding Results of AdaRound, AdaQuant, and FlexRound↩︎

Figure 6 shows the comparison of rounding results of AdaRound, AdaQuant, and FlexRound. As shown in Figure 6 (a), FlexRound can quantize pre-trained weights more flexibly than AdaRound and AdaQuant for both ResNet-18 and MobileNetV2, thereby obtaining better performance than AdaRound and AdaQuant.

7 Proof of Proposition 1↩︎

Let \({\boldsymbol{S}}'\) be \({\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3\) for a linear layer, or \({\boldsymbol{S}}_2 \odot {\boldsymbol{s}}_3 \odot {\boldsymbol{s}}_4\) for a 2D convolution.

For a linear layer, \[\begin{align} {{\partial\mathcal{L}} \over {\partial S'_{(i, j)}}} &= {{\partial \widehat{W}_{(i, j)}} \over {\partial S'_{(i, j)}}} {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \\ &= {{\partial \over {\partial S'_{(i, j)}}}}\Big(s_1 \Big\lfloor {W_{(i, j)} \over {s_1 S'_{(i, j)}}} \Big\rceil \Big) {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \\ &= s_1 {{\partial \over {\partial S'_{(i, j)}}}}\Big(\Big\lfloor {W_{(i, j)} \over {s_1 S'_{(i, j)}}} \Big\rceil \Big) {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \\ &= s_1 {{\partial \over {\partial S'_{(i, j)}}}} \Big( {W_{(i, j)} \over {s_1 S'_{(i, j)}}} \Big) {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \quad (\because \text{Straight-Through Estimator}) \\ &= s_1 {W_{(i, j)} \over s_1} {{\partial \over {\partial S'_{(i, j)}}}} \Big( {1 \over {S'_{(i, j)}}} \Big) {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \\ &= W_{(i, j)} \Big( - {1 \over {S'^2_{(i, j)}}} \Big) {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}} \\ &= - {W_{(i, j)} \over {S'^2_{(i, j)}}} {{\partial\mathcal{L}} \over {\partial \widehat{W}_{(i, j)}}}. \end{align}\]

For a 2D convolution, Proposition 1 can be proved by just replacing \(\widehat{W}_{(i, j)}\) and \(S'_{(i, j)}\) with \(\widehat{W}_{(i, j, k, l)}\) and \(S'_{(i, j, k, l)}\), respectively.

8 ResNet-18, ResNet-50, and MobileNetV2 on ImageNet with Pre-trained Models from the Official PyTorch Repository²↩︎

9 Cross-Layer Equalization and Absorbing High Biases as Preprocessing↩︎

It is known that preprocessing pre-trained weights through cross-layer equalization (CLE) and absorbing high biases (AHB) exhibits a noticeable enhancement for the per-tensor quantization performance in vision models [22], [53]. To detect the effect of CLE and AHB on AdaRound and FlexRound as preprocessing, as seen in Table ¿tbl:tab:cle?, we also quantize the weights of MobileNetV2 preprocessed via CLE and AHB to \(4\)-bit using AdaRound and FlexRound in a linear symmetric per-tensor quantization format. Following [22], every ReLU6 in MobileNetV2 is replaced by ReLU when applying CLE and AHB to MobileNetV2. When using the pre-trained model provided from the official PyTorch repository³, utilizing CLE and AHB as preprocessing enhances the performance of ‘B + AdaRound’ but not ‘B + FlexRound’ so that ‘Replacing ReLU6 + CLE + AHB + B + AdaRound’ shows better accuracy than ‘B + FlexRound’ as well as ‘B + AdaRound’. In contrast, when using the pre-trained model provided from the BRECQ github repository⁴, utilizing CLE and AHB as preprocessing seriously hinders both ‘B + AdaRound’ and ‘B + FlexRound’ from performing well. Depending on how a model is pre-trained, exploiting CLE and AHB as preprocessing can or cannot be effective. However, no matter which pre-trained model is chosen, ‘B + FlexRound’ can consistently quantize weights well without any preprocessing, which implies that FlexRound would have its own advantages compared to other post-training weight quantization methods (that might need preprocessing for better performance).

10 Ablation Study on Sample Size↩︎

No matter how much data is used, B+FlexRound always outperforms B+AdaRound. When the sample size decreases from 64 to 32, the accuracy of B+FlexRound declines by almost one percent. Correspondingly, a sample size of 32 would be a breakthrough point.

11 Combining an Additive Approach with a Division-based Approach↩︎

One might wonder whether or not there comes any benefit from combining both element-wise addition and element-wise division. Although it would be interesting to combine AdaRound with FlexRound, such a combination would be challenging due to the fact that AdaRound cannot learn a quantization grid size, \(s_1\) jointly with rounding. Alternatively, we combine AdaQuant with FlexRound. AdaQuant + FlexRound is superior to AdaQuant but inferior to FlexRound. This might be due to the naive combination of AdaQuant with FlexRound. Considering both element-wise addition and element-wise division would be an interesting future work.

12 BERT on SQuAD↩︎

Table ¿tbl:tab:squad? shows the performace of FlexRound on the SQuADv1 [48] dataset⁵ for the BERT models. Both \(\text{BERT}_{\text{Base}}\) and \(\text{BERT}_{\text{Large}}\) are uncased models. For reconstruction, we select \(1024\) samples from the training dataset of SQuADv1 at random without any modification. For ‘Q + FlexRound’, the learning rate is set to \(1\)e-\(4\) for both models. For both ‘Q + AdaRound’ and ‘Q + FlexRound’, the batch size and the number of iterations for reconstruction are \(64\) and \(20k\) respectively. We use the Adam optimizer for all methods and models. The other experimental setting of ‘Q + AdaRound’ follows [16].

Table ¿tbl:tab:squad95finetune? shows the hyper-parameter selection for fine-tuning. The same configuration is used for both \(\text{BERT}_{\text{Base}}\) and \(\text{BERT}_{\text{Large}}\). The other setting for fine-tuning and the evaluation method are the same as HuggingFace repository⁶.

13 BERT and GPT-Neo on GLUE↩︎

\begin{table}[h]

-3pt \end{table}

To investigate the natural language understanding performance of FlexRound for BERT⁷ to GPT-Neo⁸, we directly fine-tune pre-trained models on the GLUE⁹ benchmark. For BERT, we use uncased models. Hyper-parameter selection for fine-tuning a pre-trained model is given in Table ¿tbl:tab:glue-ft?. We use the Huggingface repository¹⁰ for fine-tuning without any modification.

In Table ¿tbl:tab:glue95appendix?, for reconstruction, we randomly sample \(1024\) examples from the training dataset without any modification. For all experiments, the batch size is 64, and the maximum sequence length of all experiments is 128. We use the Adam optimizer for all methods and models. In the QDrop’s setting, the probability of dropping activation quantization is set to \(0.5\). The experimental setting of ‘Q + AdaRound’ follows [16]. We also utilize the Huggingface repository¹¹ for the evaluation method without any modification.

For some datasets (QNLI, SST-2, and STS-B), ‘Q + FlexRound’ does not outperform ‘Q + AdaRound’ as shown in Table ¿tbl:tab:glue95appendix?. This suggests that there may be certain tasks where FlexRound has room for improvement. However, this outcome is due to the fact that the learning rate for \(s_1\), \({\boldsymbol{S}}_2\), and \({\boldsymbol{s}}_3\) is set to \(2\)e-\(4\) for BERT and \(3\)e-\(4\) for GPT-Neo to demonstrate that ‘Q + FlexRound’ can broadly surpass ‘Q + AdaRound’ without the need of significant efforts to select the optimal learning rate for each task. When the learning rate is fine-tuned for the datasets where ‘Q + FlexRound’ falls short of ‘Q + AdaRound’, we can observe that ‘Q + FlexRound’ outperforms ‘Q + AdaRound’ in most cases, as depicted in the table below.

14 GPT-Neo and OPT on WikiText2 and PTB↩︎

To evaluate FlexRound for natural language generation tasks, we utilize GPT-Neo¹² and OPT¹³ fine-tuned on the WikiText2 ¹⁴ and PTB ¹⁵ datasets for \(10\) epochs. Table ¿tbl:tab:hyperparameter95nlg? reports hyper-parameter selection for fine-tuning a pre-trained language model. We utilize the Huggingface repository¹⁶ for fine-tuning without any modification.

For reconstruction, We extract \(128\) random samples from the training dataset without any modification, and the number of iterations is fixed to \(500\). We use the Adam optimizer for all methods and models. The learning rate and batch size for ‘Q + FlexRound’ in Table ¿tbl:tab:clm95finetuned? are shown in Table ¿tbl:tab:hyperparameter95nlg95finetuned?. The batch size of ‘Q + AdaRound’ is same as the batch size of ‘Q + FlexRound’. The other experimental setting of ‘Q + AdaRound’ follows [16]. The probability of dropping activation quantization is set to \(0.5\) in the QDrop’s setting. We also use the Huggingface repository¹⁷ for the evaluation method without any modification.

15 GPT-2 on WebNLG↩︎

In Table ¿tbl:tab:lora?, we utilize the GPT-2 models and the WebNLG dataset available from the LoRA repository¹⁸. Namely, all LoRA checkpoints are loaded from the repository and merged to GPT-2. For reconstruction in all experiments, we use \(128\) random samples from the training dataset of WebNLG without any modification, and the number of iterations and the batch size are set to \(500\) and \(8\) respectively. For ‘Q + FlexRound’, the learning rate is set to \(5\)e-\(3\) for GPT-2 medium and \(3\)e-\(3\) for GPT-2 large, respectively. The other experimental setup of ‘Q + AdaRound’ follows [16].

16 LLaMA on Common Sense Reasoning and WikiText2↩︎

For all experiments, we employ the evaluation code from Eleuther AI’s lm-evaluation-harness [54] for common sense reasoning bechmarks and the evaluation method in the Huggingface repository¹⁹ for the causal language modeling task on WikiText2 without any modification. For reconstruction in all experiments, we use \(512\) random samples from the training dataset of C4, and the number of iterations is set to \(5000\). We use the Adam optimizer for all methods and models. For ‘Q + FlexRound’ in Table ¿tbl:tab:plm95llama95zero95appendix? and Table ¿tbl:tab:plm95llama95five95appendix?, the batch size and the learning rate are set to \(4\) and \(3\)e-\(3\) for LLaMA-\(7\)B and LLaMA-\(13\)B, and \(2\) and \(1\)e-\(3\) for LLaMA-\(33\)B. For ‘B + FlexRound’ in Table ¿tbl:tab:plm95llama95zero95appendix95w4a16? and Table ¿tbl:tab:plm95llama95five95appendix95w4a16?, the batch size and the learning rate are set to \(4\) and \(2\)e-\(4\) for LLaMA-\(7\)B, \(4\) and \(1\)e-\(4\) for LLaMA-\(13\)B, and \(2\) and \(1\)e-\(4\) for LLaMA-\(33\)B. The probability of dropping activation quantization is set to \(0.5\) in the QDrop’s setting. The other experimental setups of ‘B + AdaRound’ and ‘Q + AdaRound’ follow [14] and [16], respectively.

17 LLaMA fine-tuned via LoRA on WikiText2 and PTB↩︎

For the LoRA configuration, we apply LoRA to the query and value projection weights with a LoRA rank of \(4\). The batch size and the number of epochs are set to \(128\) and \(15\), respectively. For LLaMA-\(7\)B, LLaMA-\(13\)B, and LLaMA-\(33\)B, the learning rate is set to \(1\)e-\(4\), \(2\)e-\(4\), and \(4\)e-\(5\) for Wikitext2 and \(5\)e-\(4\), \(4\)e-\(4\), and \(6\)e-\(4\) for PTB.

For all experiments, we employ the evaluation method in the Huggingface repository²⁰ for WikiText2 and PTB without any modification. For reconstruction in all experiments, we use \(256\) random samples from the training dataset of WikiText2 and PTB respectively, and the number of iterations is set to \(5000\). We use the Adam optimizer for all methods and models. For the experiments of ‘Q + FlexRound’ on WikiText2 in Table ¿tbl:tab:llama95finetuned95lora?, the batch size and the learning rate are set to \(4\) and \(5\)e-\(3\) for LLaMA-\(7\)B, \(4\) and \(2\)e-\(3\) for LLaMA-\(13\)B, and \(2\) and \(2\)e-\(3\) for LLaMA-\(33\)B. For the experiments of ‘Q + FlexRound’ on PTB in Table ¿tbl:tab:llama95finetuned95lora?, the batch size and the learning rate are set to \(4\) and \(2\)e-\(3\) for LLaMA-\(7\)B, \(4\) and \(1\)e-\(3\) for LLaMA-\(13\)B, and \(2\) and \(3\)e-\(3\) for LLaMA-\(33\)B. For the experiments of ‘B + FlexRound’ with \(4\)-bit weight quantization on WikiText2 in Table ¿tbl:tab:llama95finetuned95lora95w4a16?, the batch size and the learning rate are set to \(4\) and \(5\)e-\(4\) for LLaMA-\(7\)B and LLaMA-\(13\)B, and \(2\) and \(2\)e-\(4\) for LLaMA-\(33\)B. For the experiments of ‘B + FlexRound’ with \(4\)-bit weight quantization on PTB in Table ¿tbl:tab:llama95finetuned95lora95w4a16?, the batch size and the learning rate are set to \(4\) and \(5\)e-\(4\) for LLaMA-\(7\)B and LLaMA-\(13\)B, and \(2\) and \(1\)e-\(3\) for LLaMA-\(33\)B. For the experiments of ‘B + FlexRound’ with \(3\)-bit weight quantization on WikiText2 in Table ¿tbl:tab:llama95finetuned95lora95w4a16?, the batch size and the learning rate are set to \(4\) and \(3\)e-\(4\) for LLaMA-\(7\)B and LLaMA-\(13\)B, and \(2\) and \(3\)e-\(4\) for LLaMA-\(33\)B. For the experiments of ‘B + FlexRound’ with \(3\)-bit weight quantization on PTB in Table ¿tbl:tab:llama95finetuned95lora95w4a16?, the batch size and the learning rate are set to \(4\) and \(7\)e-\(4\) for LLaMA-\(7\)B, \(4\) and \(6\)e-\(4\) for LLaMA-\(13\)B, and \(2\) and \(6\)e-\(4\) for LLaMA-\(33\)B. The probability of dropping activation quantization is set to \(0.5\) in the QDrop’s setting. The other experimental setups of ‘B + AdaRound’ and ‘Q + AdaRound’ follow [14] and [16], respectively.

References↩︎