1 Introduction↩︎

1.0.0.1 Motivation.

Inference on large language models (LLMs) incurs a significant compute toll for every token generated, which ultimately costs money and consumes environmental resources. These costs limit the proliferation of LLM use, especially on resource-constrained edge devices. They are also a significant barrier to furthering AI research and democratization. Therefore, improving LLM inference efficiency is a critical goal. Of all LLM efficiency techniques, quantization is by far the most successful, significantly reducing cost by reducing the data and model bit width.

1.0.0.2 Quantization.

Integer quantization involves mapping continuous values to a uniform integer grid, while minimizing error. Outliers in transformer weights, activations, and key-value data are a key challenge for quantization  [1]–[4]. The fundamental issue is that quantizing outliers to a grid leads to an unfortunate range-precision trade-off. We can either (1) capture the outliers by increasing the range, but lose valuable precision at the highest distribution density around zero, or (2) retain precision, but clip the outliers. Both options unfortunately impact model performance.

1.0.0.3 Transforms for aggressive quantization.

Prior work has explored operations, such as scaling or rotations, that can be added or applied to pretrained networks to smooth outliers without altering the overall model behaviour in the absence of quantization. For example, [5] take a single linear layer, \(\boldsymbol{\mathrm{W}}\), with input, \(\boldsymbol{\mathrm{X}}\), and apply a per-channel scaling \(\boldsymbol{\mathrm{T}}=\mathop{\mathrm{diag}}(\mathbf{s})\) to \(\boldsymbol{\mathrm{X}}\) before quantizing, to reduce outliers, applying the inverse scales to the linear weights. Without quantizers, \((\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{T}})(\boldsymbol{\mathrm{T}}^{-1}\boldsymbol{\mathrm{W}})=\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{W}}\), but with quantizers \(Q\), \(Q(\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{T}})Q(\boldsymbol{\mathrm{T}}^{-1}\boldsymbol{\mathrm{W}})\neq Q(\boldsymbol{\mathrm{X}})Q(\boldsymbol{\mathrm{W}})\). We refer to such operations as function-preserving transforms (FPTs), for which we desire the following properties:

1. Function-preservation. Without any quantization, inserting transform pairs should not change the output (up to computational errors). In practice, this means each FPT typically has an inverse operation.

2. Expressivity. Transforms with a continuous parametrization and more degrees of freedom are desirable. Continuous transforms can be optimized directly using gradient descent. Extra degrees of freedom offer more flexibility for reducing the quantization error.

3. Compute overhead. Depending on the FPT type and location, it may be possible to merge (or ‘fuse’) it into an existing operation in a pretrained model. Non-mergeable FPTs represent a new op in the computational graph, and incur additional overhead, as well as requiring software and/or hardware support.

Importantly, linear FPTs (which includes popular channel scalers, rotations, Hadamard transforms, and Kronecker transforms) can be merged if they are placed directly before or after a linear layer. Mergeability of FPTs can thus be improved if an FPT commutes with an existing operation, such that we have a choice in where we apply the FPT. For example, a rotation on the residual can be merged into transformer block input and output layers, because it commutes with the RMSNorm [6]. To create maximally-expressive, mergeable transforms, we thus need an ad-hoc understanding of the model to be quantized, including an analysis of which FPTs could commute with existing ops in the model.

1.0.0.4 Contributions.

Our contributions are threefold:

1. We introduce FPTQuant: Function-Preserving Transforms for Quantization (Figure 1). FPTQuant includes three novel transformer FPTs that are designed to be both expressive and cheap.

2. We show FPTQuant enables static INT4 quantization with minimal overhead. This provides a SOTA speed-up of up to \(3.9\times\) over FP. FPTQuant requires no kernel-level changes.

3. We show FPTQuant has an excellent accuracy-speed trade-off—it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.

2 Related work↩︎

2.0.0.1 Quantization

Neural network quantization has been demonstrated as an effective technique for reducing the model size and improving computational efficiency [10], [11]. Quantization methods can generally be categorized into post-training quantization (PTQ) and quantization-aware training (QAT) families. PTQ algorithms take a pretrained high precision network and convert it directly into a fixed-point network without the need for the original training pipeline  [12]–[20]. These methods are data-free or only require a small calibration dataset, and are generally fast and easy to use. Quantization-aware training (QAT) methods  [21]–[24] simulate quantization during training, allowing the model to find more optimal solutions compared to PTQ. However, they generally require longer training times, increased memory usage, need for labeled data and hyperparameter tuning.

2.0.0.2 LLM quantization

The excessive training cost and memory usage of traditional QAT methods make them less suitable for quantizing modern LLMs. Notably, ParetoQ [25] is the only work we are aware of that scales QAT to billions of tokens.

Post-training quantization of LLMs is a challenging task due to presence of strong numerical outliers in weights and activations [1]–[4], [26]. Various strategies have been explored at tackling these difficulties. These include employing second-order information to mitigate the quantization error [27]; emphasizing the importance of so-called “salient” weights that correspond to high-magnitude activations [28]–[30]; separating outliers and use mixed-precision [31]–[33]. Some of the other LLM PTQ methods include [34]–[37]. Note that many of these PTQ techniques focus primarily on weight quantization and memory size reduction.

2.0.0.3 Function-preserving transformations

[18] explored the idea of FPTs for CNN quantization, observing that ReLU and per-channel scaling commute, which allows scaling of weights across different layers. In the context of LLMs, [5] observe that activation quantization is harder than weight quantization due to more outliers. They propose migrating problematic outliers from the activations to the weights, using an online per-channel scaling factor for activations going into linear layers. [38] add a shift to the scaling, and use a grid search to find a scaler that minimizes the mean-squared error per linear layer. [39] extend this by including scaling vectors for queries and keys, and using gradient descent to minimize the error per transformer block.

[37] considered transforms that mix channels, albeit only for weight quantization. QuaRot [8] shows randomized Hadamard transforms (RHTs) are effective at reducing outliers. SpinQuant [9] shows that different RHTs perform very differently, yet they cannot be optimized. They extend QuaRot by adding two unconstrained rotation matrices, which are trained to minimize the standard causal LM loss. Critically, these rotation matrices are placed such that they can be merged into weights post-training, negating inference cost. [40] use online rotations consisting of fixed channel permutations and block diagonal rotations. OSTQuant [7] combined scaling vectors and rotations. Recently, FlatQuant [41] introduced matrix multiplications with a Kronecker product of two smaller matrices. This provides a transform that is both optimizable, and theoretically cheap to compute. In Appendix 6 we summarize the associated costs for various transforms and an in-depth comparison of transforms in prior work.

3 Method↩︎

3.1 Transforms↩︎

We argue equivariances and independencies in pretrained models are key to developing better FPTs, and should be explicitly exploited. Where a candidate FPT is equivariant w.r.t. pretrained model operations, we have the freedom to choose whether to apply it before, or after said operation. This can also influence whether the operation is mergeable. For example, [6] used the equivariance \(\mathop{\mathrm{RMSNorm}}(\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{M}})=\mathop{\mathrm{RMSNorm}}(\boldsymbol{\mathrm{X}})\boldsymbol{\mathrm{M}}\) for orthogonal \(\boldsymbol{\mathrm{M}}\), to apply a rotation matrix to the residual of LLMs, merging the transform and its inverse into the linear layers of each transformer block. This is a powerful transform, yet it incurs no compute overhead. Understanding equivariances and independencies in networks is thus essential for finding optimal trade-offs between expressivity ([des:expressivity]) and inference cost [des:cost]. In this section, we will discuss three equivariances, and how these offer three novel transforms.

3.1.1 Pre-RoPE transform (mergeable)↩︎

Reducing the bit width of KV cache and queries can significantly reduce memory footprint and computational cost of attention, especially with longer context windows. Unfortunately, we cannot naively merge transforms into the query and key projection weights, because modern LLMs use RoPE positional encodings [42] (see Appendix 8). We introduce a pair of pre-RoPE transforms \((\boldsymbol{\mathrm{T}}_k\), \(\bar{\boldsymbol{\mathrm{T}}}_k)\), where \(\boldsymbol{\mathrm{T}}_k\) is applied to keys and \(\bar{\boldsymbol{\mathrm{T}}}_k\) can be interpreted as an inverse of \(\boldsymbol{\mathrm{T}}_k\), applied to the queries. The transforms consist of scaled \(2\times2\) rotation matrices, and applying these to the query and key weights Pre-RoPE, the attention output remains unchanged. For simplicity we first assume a single attention head. Denoting \(i,j\in\mathbb{N}\) as the token indices and RoPE applied to queries and keys as function \(f:\mathbb{R}^d\times \mathbb{N}\to\mathbb{R}^d\) with \(f(\mathbf{x},i)=\mathbf{x}\boldsymbol{\mathrm{R}}^{d_{head}}_{\Theta,i}\) (see details Appendix 8), the following holds:

See Appendix 8 for the proof. In practice, for multi-head attention and grouped-query attention, we can choose an independent transform for each key head. Assuming there are \(H\) key heads and \(mH\) query heads for some \(m,H\in \mathbb{N}\) (\(m=1\) for standard multihead attention), this means we have \(H\) independent transforms as above. For the more typical grouped query-attention (\(m>1\)), each key head is attended to by multiple query heads, hence we need to repeat the corresponding \(\boldsymbol{\mathrm{T}}_k\) transform across these heads. Generally, we can thus write: \[\begin{align} \label{eq:prerope95grouped95query} \mathbf{s}^{(h)}&\in \mathbb{R}^d, \boldsymbol{\mathrm{R}}^{(h)}_n\in O(2),\quad \forall h,n \\ \boldsymbol{\mathrm{T}}^{(h)}_k &= \mathop{\mathrm{diag}}( \mathbf{s}^{(h)})\mathop{\mathrm{diag}}(\{\boldsymbol{\mathrm{R}}_n^{(h)}\}_{n=1}^N), \\ \boldsymbol{\mathrm{T}}_k &= \mathop{\mathrm{diag}}(\{\boldsymbol{\mathrm{T}}_k^{(h)}\}_{h=1}^H)\\ \bar{\boldsymbol{\mathrm{T}}}_k &= \mathop{\mathrm{diag}}(\underbrace{\bar{\boldsymbol{\mathrm{T}}}^{(1)}_{k}, ...,\bar{\boldsymbol{\mathrm{T}}}^{(1)}_{k}}_{m\times},\bar{\boldsymbol{\mathrm{T}}}_k^{(2)},...,\bar{\boldsymbol{\mathrm{T}}}_k^{(H)}), \end{align}\tag{1}\]

3.1.1.1 Applies to:

any model with RoPE embeddings directly after \(\boldsymbol{\mathrm{W}}_k,\boldsymbol{\mathrm{W}}_q\) (e.g. all Llama models). For models with no positional embedding (NoPE), transform \(\boldsymbol{\mathrm{T}}_k\) is valid, but it could be more expressive—it can be chosen freely per head with invertibility as the only constraint. For models with an RMSNorm between \(\boldsymbol{\mathrm{W}}_k,\boldsymbol{\mathrm{W}}_q\) and RoPE, \(\boldsymbol{\mathrm{T}}_k\) would be valid if and only if the RMSNorm does not also have a channel scaling vector, since the latter would not commute with the \(\boldsymbol{\mathrm{T}}_k\) rotations.

3.1.2 Multihead value transform (mergeable)↩︎

Note that the attention probabilities \(\boldsymbol{\mathrm{A}}\) are of shape \((B,mH, l^1,l^2)\) and the values are of size \((B, mH, l^2,d)\). The batched matmul (BMM) multiplies these per sample, head, and token, and sum this over \(l^2\). Note \(d\) plays no role in this BMM, consequently we are free to apply any invertible transform to the \(d\) axis—in particular, for a single head and any invertible matrix \(\boldsymbol{\mathrm{T}}\), the attention block output does not change upon merging \(\boldsymbol{\mathrm{T}}\) as follows: \((\boldsymbol{\mathrm{A}}(\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{W}}_v))\boldsymbol{\mathrm{W}}_o=(\boldsymbol{\mathrm{A}}(\boldsymbol{\mathrm{X}}(\boldsymbol{\mathrm{W}}_v \boldsymbol{\mathrm{T}})))(\boldsymbol{\mathrm{T}}^{-1}\boldsymbol{\mathrm{W}}_o)\). Note that the different heads in the values are independent, hence we can apply a different transform to each attention head. Newer models use grouped-query attention, which requires a bit of bookkeeping: we need to repeat the inverses per key head, across the corresponding softmax heads. Assuming there are again \(H\) value heads (repeated to \(mH\) heads) and \(mH\) query heads, we can choose any invertible \(\boldsymbol{\mathrm{T}}_v^{(h)} \in \mathbb{R}^{d\times d}\), and set: \[\begin{align} \boldsymbol{\mathrm{T}}_v &= \mathop{\mathrm{diag}}(\{\boldsymbol{\mathrm{T}}_v^{(h)}\}_{h=1}^H), \\ \bar{\boldsymbol{\mathrm{T}}}_v &= \mathop{\mathrm{diag}}(\underbrace{(\boldsymbol{\mathrm{T}}_v^{(1)})^{-1}, ...,(\boldsymbol{\mathrm{T}}_v^{(1)})^{-1}}_{m\times},(\boldsymbol{\mathrm{T}}_v^{(2)})^{-1},...,(\boldsymbol{\mathrm{T}}_v^{(H)})^{-1}), \nonumber \end{align}\] which are merged into respectively \(\boldsymbol{\mathrm{W}}_v\) and \(\boldsymbol{\mathrm{W}}_o\) weights.

3.1.2.1 Applies to:

all standard attention layers.

3.1.3 Pseudodynamic residual scaling↩︎

In modern transformer blocks, the residual remains unnormalized—i.e. the LayerNorm or RMSNorm that we apply to the input of the attention and FFN blocks, is not applied to the residual. In practice this means each token of the residual can have a vastly different scale and is difficult to quantize. Even if we do not quantize the residual, this implies that changing the residual representations of tokens \(i\) and \(j\) may require vastly different scales of the output of the attention and FFN blocks if the norm of \(i\)’s residual different than that of \(j\)’s. This may for example explain why the output of the SwiGLU in the FFN (i.e. input of \(\boldsymbol{\mathrm{W}}_d\)) has serious outliers, see [26] and Appendix 10, and that subsequent blocks can have similar outlier patterns, see Figure 2 (a,b).

Quantization could thus be improved if only the residual was normalized. Fortunately, this can be achieved at virtually no cost, without changing the output of the pretrained network. Moreover, we can apply a similar scaler inside the transformer blocks; this can reduce outliers for intra-block outlier patterns (Figure 2c).

3.1.3.1 Step 1: move RMSNorm.

Let us index all the blocks in the transformer with \(n=1,...,N\), where we index the attention and MLP blocks separately (i.e. typically, \(N\) equals two times the number of LLM transformer blocks). Let \(\boldsymbol{\mathrm{X}}_n\) denote the residual that bypasses a block, \(\boldsymbol{\mathrm{Y}}_n\) the output of a block, and \(\boldsymbol{\mathrm{Z}}_n=\boldsymbol{\mathrm{X}}_n+\boldsymbol{\mathrm{Y}}_n\). Note, normally \(\boldsymbol{\mathrm{X}}_{n+1}=\boldsymbol{\mathrm{Z}}_n\), and the transformer’s final output is \(\boldsymbol{\mathrm{Z}}_N\). We move the RMSNorm, such that it is applied to the residual too. Let us use \(\tilde{\boldsymbol{\mathrm{X}}}_n\) to denote the new residuals. Moving the RMSNorm implies that the residuals are now scaled by a matrix \(\boldsymbol{\mathrm{S}}_n=\mathbf{1}\oslash||\boldsymbol{\mathrm{X}}_n||_R\) of shape (batch, sequence length), where \(\oslash\) denotes an element-wise division and \(||\cdot||_R\) denotes the root-mean-square along the last dimension, \(||\cdot||_R:\mathbf{x}\mapsto \frac{1}{\sqrt{d}}||\mathbf{x}||_2\). In other words, \(\tilde{\boldsymbol{\mathrm{X}}}_n=\boldsymbol{\mathrm{S}}_n \odot \boldsymbol{\mathrm{X}}_n\), with \(\odot\) denoting the element-wise multiplication along the dimensions of \(\boldsymbol{\mathrm{S}}_n\).

3.1.3.2 Step 2: rescale outputs feeding back into residuals.

We do not want to change the network’s final output. To ensure this, we need to make sure that anything that feeds back into the residual is rescaled to the new normalized representation. We rescale the outputs \(\boldsymbol{\mathrm{Y}}_n\) using the same scales, i.e. ensure: \[\label{eq:y} \tilde{\boldsymbol{\mathrm{Y}}}_n=\boldsymbol{\mathrm{S}}_n\odot \boldsymbol{\mathrm{Y}}_n,\tag{2}\] which then gives \(\tilde{\boldsymbol{\mathrm{Z}}}_n=\tilde{\boldsymbol{\mathrm{X}}}_n+\tilde{\boldsymbol{\mathrm{Y}}}_n=\boldsymbol{\mathrm{S}}_n \odot \boldsymbol{\mathrm{Z}}_n\).

Note that (i) matrix multiplication, (ii) linear layers without bias¹, and (iii) BMM all commute with a scaler on the batch/sequence dimension. Consequently, we have a choice where we apply the scale, see Figure 1. Importantly, this means we can apply the rescaling far into the attention and MLP blocks, which we find reduces quantization error within these blocks.

3.1.3.3 Computing \(\boldsymbol{\mathrm{S}}_n\).

Note that for \(n>1\), \(\boldsymbol{\mathrm{S}}_n=\boldsymbol{1}\oslash||\boldsymbol{\mathrm{X}}_n||_R =\boldsymbol{1}\oslash||\boldsymbol{\mathrm{Z}}_{n-1}||_R= \boldsymbol{1}\oslash||\tilde{\boldsymbol{\mathrm{Z}}}_{n-1} \oslash \boldsymbol{\mathrm{S}}_{n-1}||_R=\boldsymbol{\mathrm{S}}_{n-1}\oslash||\tilde{\boldsymbol{\mathrm{Z}}}_{n-1}||_R\). The right-hand side means we can compute \(\boldsymbol{\mathrm{S}}_n\) based on \(\tilde{\boldsymbol{\mathrm{Z}}}_{n-1}\), instead of needing to rescale the residual first back to \(\boldsymbol{\mathrm{Z}}_{n-1}\) explicitly. We get the recursive relationship: \[\begin{align} \boldsymbol{\mathrm{S}}_0&={\boldsymbol{1}} \text{ and }\tilde{\boldsymbol{\mathrm{Z}}}_{0}=\boldsymbol{\mathrm{X}}_1 \\ \boldsymbol{\mathrm{S}}_n &= \boldsymbol{\mathrm{S}}_{n-1} \oslash||\tilde{\boldsymbol{\mathrm{Z}}}_{n-1}||_R \qquad n=1,...,N \end{align}\]

3.1.3.4 Step 3: rescale transformer output (in practice not needed).

Note that \(\tilde{\boldsymbol{\mathrm{Z}}}_N=\boldsymbol{\mathrm{S}}_n \odot \boldsymbol{\mathrm{Z}}_N\). To ensure we get the same output as the original network, we should divide the very last output by \(\boldsymbol{\mathrm{S}}_n\). In practice we do not need to: the transformer is followed by the LM head, which starts with an RMSNorm and hence removes the norm automatically.

3.1.3.5 Applies to:

all transformers with RMSNorms before their attention and FFN blocks (i.e. all modern transformers).

3.1.4 Other transforms↩︎

In addition to these new transforms, FPTQuant uses a rotation matrix \(\boldsymbol{\mathrm{T}}_r\) for rotating the residuals, since this is completely mergeable and effective at reducing activation quantization error [9]. Additionally, the notoriously bad activation quantization error at the down projection input (see Appendix 10 and Table 3 in [9]) warrants an online transform here; we use a Hadamard transform as in [8], [37], because it is cheap (Table 5). We further use a per-channel scaler transform \(\boldsymbol{\mathrm{T}}_u\) that we merge into \(\boldsymbol{\mathrm{W}}_u\) and \(\boldsymbol{\mathrm{W}}_d\), similar to [7], which effectively rescales the channels before the Hadamard transform mixes them. An illustration of all our transforms applied to a typical transformer block is shown in Figure 1.

3.2 Optimization↩︎

3.2.1 Local optimization↩︎

To reduce the worst outliers, we optimize all transforms first locally and independently—this improves subsequent end-to-end training (Appendix 11.2.1). We minimize the \(L_p\) norm of each transform’s merged weights and use gradient descent. For example, letting \(\mathcal{W}_{in}=\{\boldsymbol{\mathrm{W}}^i_q,\boldsymbol{\mathrm{W}}^i_k,\boldsymbol{\mathrm{W}}^i_v,\boldsymbol{\mathrm{W}}^i_{u},\boldsymbol{\mathrm{W}}^i_{g}\}\) and \(\mathcal{W}_{out}=\{\boldsymbol{\mathrm{W}}^i_{o},\boldsymbol{\mathrm{W}}^i_{d}\}\), for the residual rotation we optimize:

\[\label{eq:local95optimization95r1} \underset{\boldsymbol{\mathrm{T}}_r}{\min} \sum_{i=1}^{\#layers}\Big[\sum_{\boldsymbol{\mathrm{W}}\in \mathcal{W}_{in}}||\boldsymbol{\mathrm{T}}_r^{-1}\boldsymbol{\mathrm{W}}||_p + \sum_{\boldsymbol{\mathrm{W}}\in \mathcal{W}_{out}}||\boldsymbol{\mathrm{W}}\boldsymbol{\mathrm{T}}_r||_p\Big],\tag{3}\] whilst for the PreRoPE transforms \(\boldsymbol{\mathrm{T}}^i_k\) of layer \(i\), parameterized by \(\Phi^i\), we just minimize: \[\underset{\Phi^i}{\min} ||\boldsymbol{\mathrm{W}}^i_q\bar{\boldsymbol{\mathrm{T}}}^i_k||_p + ||\boldsymbol{\mathrm{W}}^i_k\boldsymbol{\mathrm{T}}^i_k||_p.\] Since \(\boldsymbol{\mathrm{T}}_r\) affects all linear layers, we optimize it first (Eq 3 ). Locally optimized transforms are merged into the weights, after which the next transform is optimized and so forth. We set \(p=4\), following [43], who showed \(L_4\) is good for determining the quantization grid. Local optimization cost is negligible compared to end-to-end training (Appendix 16.1).

3.2.2 End-to-end optimization↩︎

We follow [44] and use student-teacher training for reducing the quantization error further, training the student \(f_\Phi\) (quantized model with transforms) to approximate the teacher \(f\) (unquantized FP model). The original model’s weights are frozen and shared with the student, hence the student-teacher framework has no additional memory footprint. We use Jensen-Shannon Divergence loss: \[\min_\Phi \mathbb{E}_X[JSD[f(\boldsymbol{\mathrm{X}}),f_{\Phi}(\boldsymbol{\mathrm{X}})],\] where \(\Phi\) includes both the transformation and the quantization grid parameters. It is essential we include the latter—the grid cannot adapt to the transformed input otherwise. In Appendix 18 we show that FPTQuant’s parametrization is stable, i.e. that even with noisy training updates the function-preserving property ([des:identity]) holds.

The end-to-end student-teacher approach deviates from SpinQuant and FlatQuant. SpinQuant uses the LLM’s original next-token prediction loss. Compared to next-token prediction, student-teacher training: 1) provides more signal (i.e., for each data point and sequence element, a full vector of probabilities, vs. a single label), and in turn this 2) decreases overfitting. This is an important reason to avoid next-token prediction loss: although we are working with transforms that in the absence of quantization do not change the model output, the combination of the large number of parameters \(|\Phi|\) and the quantization non-linearities (i.e. rounding), actually provide the transformed and quantized model with enough capacity to overfit—see Appendix 11.2.2. FlatQuant optimizes the mean squared error (MSE) per transformer block. This is not directly applicable for transforms that may affect multiple blocks at once, for example a rotation applied to the residual and merged into all linears, as used here and by [8], [9].

4 Experiments↩︎

4.0.0.1 Evaluation.

We choose a range of models and settings to evaluate FPTQuant. We use Llama-2 7B/13B [45] and Llama-3 8B [46] to allow direct comparison to reported results from QuaRot, SpinQuant and FlatQuant. We add to this Llama-3.2 3B instruct—a newer and smaller model that is popular for edge devices. Finally, we test other model families and bigger sizes including Ministral 8B [47] and Qwen-2.5 32B [48] . We evaluate on Wikitext-2 [49], and use LM-harness to evaluate the same Common Sense Reasoning tasks used in FlatQuant: PIQA [50], WinoGrande [51], HellaSwag [52], ARC-e and ARC-c [53], and LAMBADA [54]. In Appendix 13 we also include results for reasoning tasks (5-shot MMLU and GSM8K).

4.0.0.2 Baselines.

We compare FPTQuant against the original floating point model (FP), PTQ using rounding-to-nearest (RTN), RTN with optimizing the quantization ranges (RTN-opt), rotation-based QuaRot, SpinQuant, OSTQuant, and the more expensive but state-of-the-art FlatQuant.

4.0.0.3 Training Setup.

For fair comparison, we use the same training and compute budget for all methods—1024 steps with batch size 16 and sequence length 2048. We train on Wikitext-2. We found that end-to-end student-teacher training significantly improves generalization over next-token prediction (see Appendix 11.2.2), hence choose to use this for the RTN-opt, SpinQuant and QuaRot baselines too. For each method and quantization setup we select hyper-parameters based on the perplexity on the Wikitext-2 validation set.

4.0.0.4 Quantization Setup.

Transforms help with low-bit activation quantization, hence we study different activation quantization settings—covering realistic deployment settings (Sec. 4.2) and dynamic activation quantization (Sec. 4.6). We initialize quantization ranges based on \(L_p\) norm of the error, and further improve the range during training (i.e. use learnable weight/activation clipping) for all methods except “RTN". In Appendix 14 we report results for GPTQ weight quantization.

Further details about hyperparameters and implementation are in Appendix 9.

4.1 FPTQuant is fast↩︎

4.1.0.1 Setup.

We evaluate the prefill² runtime performance of our method and compare against other methods using FTPs. We implement FPTQuant, SpinQuant, FlatQuant, and INT4 baseline using PyTorch CUDA/12.1 and using INT4 CUTLASS kernels from QuaRot repository³. Note that QuaRot and OSTQuant are about as fast as SpinQuant at inference time. QuaRot is slightly slower, since it has an extra Hadamard transform applied to the head dimension (before \(\boldsymbol{\mathrm{W}}_o\)); OSTQuant is marginally slower, since it uses online smoothing vectors after the RMSNorm (SpinQuant/FPTQuant merge these into linear layers). For all methods we assume static INT4 quantization. All the measurements are conducted on NVIDIA RTX 3080 Ti. We provide all our experiments on a single transformer block as the whole model does not fit on a single GPU for big enough model size and/or the batch size. We repeat each measurement 1000 times and report the mean speedup relative to FP16 baseline. More details and additional results with using dynamic quantization are in Appendix 15.

4.1.0.2 Results.

Figure 3 shows the prefill speedup of FPTQuant across different batch sizes and model sizes. For most configurations, we get \(2.8\)–\(3.9\times\) speedup over the FP16 implementation, which is significantly faster than prior reported speedups of QuaRot and FlatQuant. The speedup is consistently increasing with model size, as the computation becomes the main bottleneck. FPTQuant is on par or faster than SpinQuant and consistently faster than FlatQuant, with a relative speedup of 15–29%. FPTQuant is also faster to train, see Appendix 16.1. In all cases FPTQuant is within a 5–6% to the INT4 upper bound.

4.2 FPTQuant excels at hard deployment settings↩︎

4.2.0.1 Motivation.

There is a large decision space when choosing which activations to quantize. Prior works [8], [9], [41] focus on dynamic quantization of linear inputs and KV cache. This deviates from LLM deployment in practice, which typically (i) has better support for static activation quantization (see Appendix 7 for details); and (ii) quantizes more intermediate activations for better speed and memory footprint [55], [56]. In this experiment, we evaluate SpinQuant, FlatQuant, and FPTQuant for different static activation quantization settings on Llama-3.2 3B instruct. We observe similar behaviour for Llama-3 8B and Qwen-2.5 7B and zero-shot performance (Appendix 12).

4.2.0.2 Results.

We observe (Table 1) that FPTQuant performs comparably to baselines for quantization settings with only linear inputs and KV cache quantize (i). It excels at the most challenging setting (iii), in which all activations within the attention and MLP block are quantized. FPTQuant slightly underperforms baselines when queries and keys are quantized to 4 bit (ii), since the Pre-RoPE transform has less capacity to reduce quantizers here than the non-mergeable transforms of SpinQuant (\(R_3\)) or FlatQuant (\(P_h\)).

4.3 Main results↩︎

4.3.0.1 Setup.

In the previous section we saw that FPTQuant outperforms baselines comfortably for the most realistic quantization settings. We extend our evaluation to more models and multiple bit widths for static quantization, focussing on the setting that FPTQuant did relatively worst: (i) only Linears+KV. In Appendix 13 we include standard deviations for a subset of these results, and include reasoning metrics MMLU and GSM8K.

4.3.0.2 Results.

See Table 2. Similar to earlier results, FPTQuant almost always outperforms QuaRot and SpinQuant. OSTQuant generally performs poorly—this is due to OSTQuant not being strictly function-preserving, and having stability problems (Appendix 18). In most cases FPTQuant shows competitive performance to the significantly slower FlatQuant. However, we do note that for the very challenging setup of W4A4KV4 the gap can be larger, especially for zero-shot accuracy. Note that FlatQuant with static quantization can be unstable in the optimization, e.g. Llama-2 W4A8KV8 is surprisingly bad. We explore this instability further in Section 4.5 and Appendix 18, where we do a sensitivity analysis of the transforms. Also note that for W4A8KV8, we outperform FlatQuant, because the mergeable FPTQuant transforms are better at reducing weight quantization error—at W4A8KV8, this is relatively more important, since A8 activation quantization is easier.

6pt

4.4 Ablation of proposed transforms↩︎

4.4.0.1 Setup.

We explore the value of each transform. Let us take Llama-3.2 3B-it and the same quantization setup as before, Linears+KV, at W4A4KV4. We subselect different transforms and repeat the experiment for three seeds. We include more ablations in Appendix 11.1, including comparisons of individual transforms to “similar" transforms in literature.

4.4.0.2 Results.

We find (Table 3) that each of the three transforms \(\boldsymbol{\mathrm{S}}_n\), \(\boldsymbol{\mathrm{T}}_k\), \(\boldsymbol{\mathrm{T}}_v\) helps reduce the perplexity and improves the 0-shot CSR and 5-shot MMLU.

4.5 Sensitivity analysis↩︎

4.5.0.1 Sensitivity analysis.

We conduct a sensitivity analysis to show the function-preserving properties of different transforms without quantization on Llama-3.2 3B instruct. We measure the Wikitext perplexity after adding i.i.d. Gaussian noise \(N(0,\sigma^2)\) to all transform parameters. This also gives insight into training stability—when the output of the model is stable w.r.t. the parametrization of transforms, noisier gradient updates are less likely to lead to unstable training.

4.5.0.2 Results.

Figure 4 shows that SpinQuant and FPTQuant are completely stable w.r.t. noise. FlatQuant is theoretically function-preserving, but suffers from floating point issues from roughly \(\sigma\geq 0.3\) (see also their Appendix . OSTQuant is not function-preserving, and shows serious degradation for even \(\sigma=0.3\). We provide more details and results for other models in Appendix 18.

4.6 Dynamic quantization↩︎

4.6.0.1 Setup.

We repeat the previous experiment with W4A4KV4 in a dynamic quantization setting. This is identical to the FlatQuant setup, from which we report baseline results. For all methods, we use round-to-nearest for weight quantization, with learnable weight and activation clipping—see Appendix 14 for GPTQ results.

4.6.0.2 Results.

We observe (Table 4) that FPTQuant is consistently on par or better than all baselines except FlatQuant. However, FPTQuant is up to 29% faster than FlatQuant.

5 Discussion↩︎

5.0.0.1 FPTQuant.

When choosing FPTs, there is a trade-off between expressivity ([des:expressivity]) and cost ([des:cost]): more expressive transforms can help reduce quantization error, but incur overhead. By understanding commutation properties of existing operations within the transformer, we have designed most of FPTQuant’s transforms to be both expressive, yet mergeable into existing weights. In many settings, the FPTs used by FPTQuant provide a good trade-off between accuracy and speed. For some settings, one may prefer to combine FPTQuant with more expressive, non-mergeable transforms. For example, in Table 11 we show that adding the non-mergeable \(P_h\) transform (FlatQuant) to FPTQuant yields another good accuracy-speed trade-off, with better KV cache compression but incurring some overhead. Choosing which FPTs to choose is largely dependent on the model, quantization setting, and resource constraints. In Appendix 17 we provide some guidelines for practitioners who want to use FPTs for quantizing their own models.

5.0.0.2 Limitations and future work.

We evaluated FPTQuant on LLMs from different generations and with different sizes. While challenges and outlier patterns are often similar across different models [2], [3], [26], it cannot be guaranteed that our insights and gains equally translate to all LLMs. Mixture-of-expert models (MoEs) could also be explored. FPTQuant is applicable to MoEs with only minor modifications—the rotation \(\boldsymbol{\mathrm{T}}_r\) needs to be merged into the router input weights and all expert output weights, and \(\boldsymbol{\mathrm{S}}_n,\boldsymbol{\mathrm{T}}_r,\boldsymbol{\mathrm{T}}_u\) can be applied to each expert separately. Future work could alo extend evaluations to FP microscaling formats—there is some evidence that FPTs can help MXFP4 [57], but research is still limited.

Another limitation is integer quantization itself. Current Nvidia hardware has limited support for low-bit integer, which hinders accurate benchmarking. In the short term, this also means any integer quantization work is primarily relevant for edge compute, where most computation happens in integer due to energy and chip area constraints. Due to integer’s power advantage over FP and ongoing work to reduce quantization error at extreme low bits, we can expect more integer support on future (inference) servers.

Impact Statement↩︎

We think FPTQuant has significant positive societal impact. FPTQuant empowers the use of smaller bit widths, which reduces computational, energy, and environmental impact of LLMs. Reduced LLM cost and memory footprint could make LLMs more accessible to economically-disadvantaged populations, and could improve inference on edge devices (e.g. smartphones).

6 Detailed transforms comparison↩︎

In Table 5 we include the representation and theoretical cost of existing transforms. In Table 6 we review existing works, the transforms they use, and their placements.

\(^\dagger\) Authors claim channel-wise scaling of queries and keys can be merged, which does not hold for non-additive positional encodings (e.g. RoPE). \(^\ddagger\)We also use SpinQuant’s mergeable \(R_1\) rotation, and non-mergeable HT at mm. \(^*\)OSTQuant also proposed a scaler before RoPE, but this does not commute with RoPE and is thus not function-preserving.

7 Discussion on static vs dynamic quantization↩︎

Traditionally quantization literature and fixed-point accelerators always used static activation scaling factors. However, most LLM quantization literature almost silently started assuming dynamic scaling factors for activations. We call out the distinction here, since in practice it has a big impact on both inference token rate and even which platforms are supported.

7.0.0.1 Static quantization

fixes the maximum anticipated range of each quantized tensor ahead of inference time. The quantization grid is determined based on a small calibration dataset, before fixing the scale factors. At runtime, we need only apply these floating-point scale factors following integer matrix multiplication [11]. However, despite the ubiquity of static quantization, all prior work to date using FPTs for LLM quantization that we have surveyed (Section 2.0.0.3) assumes dynamic quantization.

7.0.0.2 Dynamic quantization (DQ)

foregoes the calibration step and instead computes scale factors dynamically at runtime for each token independently. This means we can set a large grid for tokens with outliers, while keeping the grid small for tokens without outliers. While this obviously is a huge boon for model performance, it unfortunately introduces a non-trivial compute cost.

7.0.0.3 DQ compute overhead.

At inference time, DQ requires the minimum and maximum activation values to be computed and reduced over the last dimension of the whole activation tensor, for each token. The resulting scale factors must then be broadcast and applied to each value. This reduce-broadcast operation can be relatively fast on a CPU, which operates on small chunks of data at a time, such that the binary tree required for the reduction is manageable. However, GPUs and NPUs typically process large tensors at once using custom hardware, and thus the reduction and broadcast tree operations are deep and slow relative to the high throughput MAC operations themselves.

7.0.0.4 Lack of support for DQ.

DQ is currently not natively supported on many popular hardware and software stacks. For example, popular quantization packages such as Nvidia TensorRT [58] and PyTorch AO [59] do not support DQ. Edge hardware platforms also lack support for DQ on their accelerators, including Qualcomm SnapDragon [60] and Nvidia Deep Learning Accelerator (DLA) [61].

8 Proof Theorem 1↩︎

8.0.0.1 RoPE background.

RoPE’s [42] aim is to modify the queries and keys, such that the output of the query-key multiplication is dependent on their relative positions. RoPE achieves this by multiplying queries and keys with a time-dependent rotation matrix, i.e. RoPE is a function \(f:\mathbb{R}^d\times \mathbb{N}\to\mathbb{R}^d\) with \(f(\mathbf{x},i)=\mathbf{x}\boldsymbol{\mathrm{R}}^{d_{head}}_{\Theta,i}\), where \(i\) denotes the token index, \(\Theta\) the RoPE parameters, and \(d_{head}\) the head dimension. Matrix \(\boldsymbol{\mathrm{R}}^{d_{head}}_{\Theta,i}\) is a block-diagonal matrix with \(N=d_{head}/2\) blocks. Each block \(n\) has size \(2\times 2\) and denotes a rotation of angle \(i\theta_n\) of two dimensions. Denoting a 2-dimensional rotation of angle \(\theta\) by \(\boldsymbol{\mathrm{R}}^{(2)}_{\theta}\), we can thus write \(\boldsymbol{\mathrm{R}}^{d_{head}}_{\Theta,i}=\mathop{\mathrm{diag}}((\boldsymbol{\mathrm{R}}_{i\theta_n})_{n=1}^N)\). As desired, the product between embedded keys and queries depends only on their relative, not absolute, position: \(\langle f(\mathbf{q}_i,i),f(\mathbf{k}_j,j)\rangle=\mathbf{q}_i \boldsymbol{\mathrm{R}}^d_{\Theta,i-j}\mathbf{k}_j^\intercal\). We develop transforms that we can apply to queries and keys, yet do not alter the output of the attention softmax. We design these to commute with RoPE’s \(\boldsymbol{\mathrm{R}}^d_{\Theta,i}\) for all \(i\), so that they can be applied before RoPE and merged into \(\boldsymbol{\mathrm{W}}_q\) and \(\boldsymbol{\mathrm{W}}_k\).

Theorem 1 Let \(N=d_{head}/2\), and \(\boldsymbol{\mathrm{R}}_n\in O(2)\) and \(s_n\in \mathbb{R}\), for \(n=1,...,N\). Define \(\boldsymbol{\mathrm{T}}_k = \mathop{\mathrm{diag}}(\mathbf{s})\mathop{\mathrm{diag}}(\{\boldsymbol{\mathrm{R}}_n\}_{n=1}^N)\) and \(\bar{T}_k=\mathop{\mathrm{diag}}(\mathbf{s}^{-1})\mathop{\mathrm{diag}}(\{\boldsymbol{\mathrm{R}}_n\}_{n=1}^N)\). Given query and key weights \((\boldsymbol{\mathrm{W}}_q, \boldsymbol{\mathrm{W}}_k)\in\mathbb{R}^{d_{in}\times d_{head}}\), define \(\tilde{\boldsymbol{\mathrm{W}}}_q=\boldsymbol{\mathrm{W}}_q \bar{\boldsymbol{\mathrm{T}}}_k\) and \(\tilde{\boldsymbol{\mathrm{W}}}_k=\boldsymbol{\mathrm{W}}_k \boldsymbol{\mathrm{T}}_k\). Now it holds: \[\langle f(\mathbf{x}_i \tilde{\boldsymbol{\mathrm{W}}}_q,i),f(\mathbf{x}_j \tilde{\boldsymbol{\mathrm{W}}}_k,j)\rangle=\langle f(\mathbf{x}_i \boldsymbol{\mathrm{W}}_q,i),f(\mathbf{x}_j \boldsymbol{\mathrm{W}}_k,j)\rangle\]

9 Experimental details↩︎

9.0.0.1 General set-up.

For all experiments and methods, we use batch size 16 with 2048 sequence length for training. We train on Wikitext-2, for 1024 steps with cosine learning rate scheduler, 10% warm-up, and learning rate based on validation PPL. For Qwen-2.5 32B, we use sequence length 512 and 512 steps. For W4A4KV4 Qwen-2.5 32B, we observed larger sensitivity to training, and run each method with multiple learning rates and number of steps (128,512,1024)—the best validation PPL was gotten for 128 steps for SpinQuant, 512 for RTN-OPT, QuaRot and OSTQuant, and 1024 for FPTQuant and FlatQuant. In all experiments, we use learnable weight and activation clipping, i.e. the quantization scale and offset are parameters that are updated. We have found significant advantage to optimizing the quantization grid end-to-end, as can be seen by the relatively good RTN-opt baseline in Table 2. For Wikitext perplexity, we evaluate on sequence length 4096—except for Llama-2 7B/13B, for which 2048 is the maximum. We use 2048 sequence length in Table 4, to allow fair comparison with results from FlatQuant.

9.0.0.2 Baselines.

This work focusses on FPTs. To really understand the value of the FPTs, and not just better and more costly training, we have chosen to use the same training set-up for all methods. This includes optimizing the quantization grid end-to-end. We have found that this significantly improved the performance of some of the baselines—in particular QuaRot and RTN, which do not use any optimization themselves. The only exception is the dynamic quantization experiment (Table 4): to make direct comparison possible, we use the set-up and numbers from FlatQuant, which does not optimize baselines.

FPT Parametrization. We use torch.nn.utils.parametrizations.orthogonal to parametrize orthogonal matrices with Cayley parametrization. Some FPTs use matrix inversions, e.g. our \(\boldsymbol{\mathrm{T}}_v\) and FlatQuant’s \(P_h\). To avoid computing the inverse during training, it is possible to parametrize these matrices using a singular value decomposition instead, consisting of a diagonal matrix and two orthogonal matrices. In practice, we have found that the added computation of the orthogonality parametrization (which internally computes inverses in any case), leads to slower training and worse results. To avoid potential instability problems with a direct inverse, we choose to keep all transforms in double precision.

Quantizer range setting. Although we learn the quantization grids, a good initial quantization grid improves training. We initialize the quantization grid during a range setting stage. For all experiments and method, we pass 64 sequences through the unquantized network and choose a grid that minimizes the \(L_p\) norm of the difference between the unquantized and would-be-quantized values. Note that \(L_\infty\) corresponds to minmax range setting, which is popular due to its simplicity. In practice, however, we have found that \(p=3\) is better than either minmax, \(L_4\) or \(L_2\). We choose \(L_3\) range setting for all experiments and also for baselines.

Dynamic quantization experiment. For the dynamic quantization experiment, we use the baseline numbers as reported in FlatQuant, except for OSTQuant which is not compared against in FlatQuant. Since OSTQuant only reports results with GPTQ in their own paper, we used their codebase and provided commands ⁵ to generate results for both RTN and GPTQ. For OSTQuant and FPTQuant, we repeat each experiment for three seeds and report the median perplexity and zero-shot accuracy.

9.0.0.3 Hadamard non-powers of 2

FPTQuant, SpinQuant[9], and QuaRot [8] use Hadamard transforms. Hadamard transforms are simple to define for powers of 2, namely \(H_{2^d} = \bigotimes_{i=1}^d \tiny{\begin{bmatrix} +1 & +1 \\ +1 & -1 \end{bmatrix}}\). For some non-powers of 2, there are Hadamard transforms, but these are not implemented in popular packages like fast-hadamard-transform [62]. The simplest approach, and default behaviour in fast-hadamard-transform, is to pad with zeros and discard added dimensions after applying the Hadamard transform. This is not correct for FPTs: the added rows are necessary for mapping the transformed activations back to the original values, so setting these rows to zero instead, will yield a different output.

To avoid problems with non-powers of 2, we take a block-wise Hadamard transform: we split dimensions into \(K\) groups that are each a power of 2, and apply a standard Hadamard to each. The residual and FFN hidden dimensions \(d\) in LLMs are typically \(2^{n}\times K\) with \(K\) small—the largest we have found is \(K=43\) for the FFN hidden dimension in Llama-2 7B. The grouped Hadamard can be parallelized efficiently by reshaping the channel dimension into two dimensions of sizes \((K,2^n)\), applying the Hadamard, and reshaping back. Using a grouped Hadamard reduces the mixing to within groups, but we have found no evidence of a reduced ability to spread outliers due to this—probably because group sizes are still always \(256\) (for Llama-2’s FFN hidden layer) or larger.

10 Quantization error per quantizer↩︎

In this section, we study the quantization sensitivity of individual weight and activation quantizers.

10.0.0.1 Setup

We apply INT4 RTN quantization, without optimization, to a single quantizer location (see Table [tbl:tbl:A95act95quantizers] for notation) at a time, and report the WikiText-2 test perplexity. We follow the same protocol for the range setting as in the main setup (Appendix 9). For activation quantization study, we repeat the experiment three times with different seeds (that affect the random selection of sequences for range estimation) and report the mean value.

3pt

3pt

10.0.0.2 Observations

In Table 7, we can see that each weight quantizer location adds about 0.1 perplexity, on average, while the down projection stands out from the rest a bit more. We can also see that the perplexity drop from quantizing all weights is roughly the sum of drops of individual weight quantizers, meaning that the weight quantization noise is approximately additive.

From Table 8, however, we observe that activation quantization is significantly more challenging, where often a single activation quantizer completely ruins the model performance. Specifically, among the most problematic locations consistently for all models are down projection input/output (mm, d), and residuals (ra, rm). Typically, those locations have the strongest outliers, which makes them difficult to quantize with uniform affine quantization scheme.

11 Additional ablation studies↩︎

We introduce FPT \(\boldsymbol{\mathrm{T}}_v\) which is an in-place replacement for \(R_2\) (SpinQuant) and \(P_v\) (FlatQuant). We also propose \(\boldsymbol{\mathrm{T}}_k\), which has a similar aim as \(R_3\) (SpinQuant) and \(P_h\) (FlatQuant), but is mergeable. At last, we introduce \(\boldsymbol{\mathrm{T}}_u\), which is an addition to QuaRot/SpinQuant’s Hadamard transform before the down projection. In this Appendix we ablate these FPTs.

11.1 Transform ablations↩︎

11.1.0.1 \(\boldsymbol{\mathrm{T}}_v\).

We introduce FPT \(\boldsymbol{\mathrm{T}}_v\), which is both mergeable, but also very expressive—we can choose and optimize any invertible \(d_{head}\times d_{head}\) matrix for each attention head, giving in total \(H\times d_{head}\times d_{head}\) degrees of freedom. This is much stronger than SpinQuant’s \(R_2\) [9], which optimizes a single orthogonal matrix across all value heads (about \(d_{head}^2/2\) degrees of freedom). It is also stronger than FlatQuant’s \(P_v\), who propose two options for parametrizing \(P_v\), either a Kronecker or full matrix (see Table 5), but in both cases not chosen per head (max \(d_{head}\times d_{head}\) degrees of freedom).

We ablate the value of \(\boldsymbol{\mathrm{T}}_v\) compared to \(P_v\) (full matrix) and \(R_2\). To isolate the effect of these FPTs, we quantize only weights, V-cache, and input to the out projection layer (W4A4). We use the same training set-up as in the main experiments.

We observe (Table 9) that \(\boldsymbol{\mathrm{T}}_v\) performs consistently better across models, in particular significantly outperforming SpinQuant’s \(R_2\). Since all these options have the same inference cost—0, since they are mergeable—we believe \(\boldsymbol{\mathrm{T}}_v\) should be a preferred choice.

11.1.0.2 \(\boldsymbol{\mathrm{T}}_k\).

We conduct a similar ablation for \(\boldsymbol{\mathrm{T}}_k\). \(\boldsymbol{\mathrm{T}}_k\) is merged into \(\boldsymbol{\mathrm{W}}_k\) and \(\boldsymbol{\mathrm{W}}_q\), and can thus help with key and query quantization. This is similar to \(R_3\) and \(P_h\) from respectively SpinQuant and FlatQuant, although these transforms are applied online after the RoPE operator, and thus incur overhead. However, these baselines FPTs are less restricted as a result, and can thus ensure more mixing across channels.

We run a similar experiment as before, only quantizing weights, queries, and keys. We find (Table 10) that for 4-bit quantization of queries and keys, FPTQuant underperforms baselines due to the more restrictive FPT and less mixing across channels. At W4A8, we find \(\boldsymbol{\mathrm{T}}_k\) performs on par with baseline FPTs. This experiment clearly shows the expressivity and cost trade-off, [des:expressivity] vs [des:cost]. In some cases, especially when aggressive query-key quantization is beneficial, the overhead of \(R_3\) or \(P_h\) may weigh up against their higher cost. In Table 11 we show that adding \(P_h\) indeed narrows the gap to FlatQuant on the hardest (W4A4KV4) quantization settings.

11.1.0.3 \(\boldsymbol{\mathrm{T}}_u\).

The activations before the down projection layer have large outliers. A Hadamard transform at this location has been shown to massively reduce the quantization error [8], [9], as it mixes outliers across channels and hence whitens the activation distribution. Whitening is more effective if variables (in this case, channels) have a similar scale. Our scaling transform \(\boldsymbol{\mathrm{T}}_u\) achieves exactly this, whilst being completely mergeable.

In this ablation, we test the performance of a Hadamard transform \(\boldsymbol{\mathrm{T}}_d\) with and without \(\boldsymbol{\mathrm{T}}_u\). We use a randomized Hadamard transform for \(\boldsymbol{\mathrm{T}}_d\), as [9] find that even 1 and -1 scales can perform better than non-randomized. Intuitively, \(\boldsymbol{\mathrm{T}}_u\) has large benefits over using randomized Hadamard transforms: the randomized Hadamard discrete binary vector is not easy to optimize, does not allow proper scaling down of high-variance channels, and has been shown to exhibit large variance w.r.t. initialization [9]. We only quantize the down projection input and weights (W4A4), but leave all other activations unquantized. We train for 512 steps with batch size 8 and sequence length 2048 optimizing quantization grid and \(\boldsymbol{\mathrm{T}}_u\) scalers, and run for three seeds.

In Table 12 we observe that adding \(\boldsymbol{\mathrm{T}}_u\) has a consistently significant positive effect on quantization error. Like [9], we find that the randomized Hadamard transform has large variance, yet we never observe it does better than when \(\boldsymbol{\mathrm{T}}_u\) is added. The largest benefit is observed for Llama-2 7B, which has significant outliers in activation before the down projection, which \(\boldsymbol{\mathrm{T}}_u\) can scale down.

11.2 Optimization↩︎

11.2.1 Local optimization↩︎

In Section 3.2.1 we proposed a simple data-free and cheap local optimization strategy. Here, we ablate the value of this for overall training stability and speed. We train Llama-3.2 3B-it with FPTQuant end-to-end, with and without first locally optimizing for 200 steps (see Eq. 3 ). We repeat the experiment for \([0,32,128,256,512]\) number of end-to-end training steps. As before, we use batch size 16 and sequence length 2048, and 10% warm-up steps.

We observe (Figure 5) that local optimization significantly improves pre-training performance. More importantly, the better initialization advantage persists during end-to-end training, partly due to a more stable training process. With larger number of end-to-end steps, local optimization becomes less beneficial. Locally optimizing all transforms sequentially for 200 steps each takes only about 9 minutes (equivalent in wall time to about 20 end-to-end training steps), and this could be reduced further by parallelizing. Consequently, we find that local optimization is a simple approach to make end-to-end training faster and more efficient.

11.2.1.1 Choosing \(p\).

During local optimization, we minimize the \(L_p\) of merged weights. In Table 13 we ablate different values of \(p\) for the same setting as before, evaluating the performance before and after end-to-end training. We find that no \(L_p\) performs significantly better than another—though not using local optimization ("No opt") does significantly worse on average and has a large variance. As before (Figure 5), this shows that local optimization improves training stability, though the choice of \(p\) is less important.

11.2.2 Student-teacher training.↩︎

We compare the value of end-to-end training in a student teacher fashion (E2E[ST]), versus the original next-token prediction loss (E2E[label]) used in e.g. SpinQuant [9]. We take Llama-3.2 3B instruct and use the same set-up as before—training on Wikitext with sequence length 2048, 1024 training steps, and batch size 16.

We observe (Table 14) that next-token prediction leads to consistently better Wikitext perplexity. This makes sense: the loss for next token prediction is highly similar to the loss of Wikitext perplexity, and since we train and evaluate on (different splits of) Wikitext, the next-token prediction loss fine-tunes the FPT weights and quantization grid to directly minimize this loss. However, we also observe that for FPTs with learnable transforms and hence more capacity (SpinQuant, FPTQuant), the next-token prediction leads to significantly worse 0-shot performance, which indicates that the next-token prediction loss generalizes poorly to tasks that are different than the training set. In other words, the next-token prediction loss allows the model to overfit to the target task. Student-teacher training does not allow the same level of overfitting, since the output is fitted to match the whole unquantized output vector. This also has the added value that a whole vector of probabilities (student-teacher training) provides more signal than a one-hot label (next-token prediction).

It may seem counterintuitive that FPTs can lead to such overfitting, since they are designed to preserve the model function ([des:identity]). However, note that most FPTs have a large number of trainable parameters (Table 5), which together with a learnable quantization grid, including activation clipping, entails a large capacity to change the function post-quantization. For example, next-token prediction could relatively easily decrease the loss by increasing the probability of words that are typical Wikitext lingo (which could be achieved through simple clipping of non-typical tokens). Student-teacher loss would not benefit from this, since even for untypical tokens, it needs to match the output probability.

FPTs are appealing because they do not alter the model’s function significantly and do not require significant training. The tendency of next-token prediction to overfit to the training task is undesirable to this end; overfitting alters the model function significantly, and to avoid it we would need to train for longer with more tasks. Consequently, we discourage researchers from using next-token prediction for training FPTs, unless they desire to fine-tune to a specific training set.

12 Quantization settings extended↩︎

We extend Table 1 to include 0-shot performance and extra models. Table 15 includes Llama-3.2 3B instruct and Llama-3 8B, as well as Qwen-2.5 7B instruct [48]. The latter deteriorates significantly for all transforms at 4-bit activations, due to more challenging activation distributions—see Table 8. Consequently, we use W4A4KV4 for the Llama models, but W4A8KV8 for Qwen.

0.8mm

13 Reasoning performance and standard deviations↩︎

To give insight into the stability of methods and significance of results, we repeat the experiment of Table 2 for Llama-3.2 3B instruct for multiple seeds and estimate standard deviation. We also include reasoning metrics 5-shot MMLU and GSM8K. This gives Table 16

Overall, we see that in line with the main paper’s results, FPTQuant outperforms baselines SpinQuant and QuaRot almost consistently. Especially on the low bitwidth W4A4KV4, FPTQuant improves over SpinQuant/QuaRot very significantly. The more expensive FlatQuant performs comparably to FPTQuant for Wiki and 0-shot, but outperforms FPTQuant on the more sensitive reasoning tasks.

14 Dynamic quantization with GPTQ↩︎

We repeat the dynamic quantization experiment (Section 4.6 with GPTQ weight quantization. See Table 17. We find that FPTQuant and FlatQuant are not helped consistently by GPTQ, likely because their transforms are already trained to reduce weight quantization error for the RTN results. We leave it to future work to improve the results with GPTQ weight quantization.

15 Detailed benchmarking results↩︎

In this section, we provide additional details on runtime performance setup and evaluation of our method using dynamic quantization in comparison to other methods using FTPs.

15.0.0.1 Setup

We implement FPTQuant, SpinQuant, FlatQuant, and INT4 baselines (using static and dynamic quantization) using PyTorch CUDA/12.1 and using INT4 CUTLASS kernels from QuaRot repository⁶. All the measurements are conducted on NVIDIA RTX 3080 Ti. We provide all our experiments on a single transformer block as the whole model does not fit on a single GPU for big enough model size and/or the batch size. We repeat each measurement 1000 times and report the mean speedup relative to FP16 baseline.

Specifically, we use CUTLASS kernels for quantization/de-quantization and linear layers. Because there is no native INT4 support on Nvidia hardware yet, we use INT8 storage, where each entry represents a pair of INT4 numbers (“double-packed” representation). The kernel for a linear layer, for instance, takes two packed tensors representing weights and activations and computes the matmul assuming INT32 accumulator. Note that query-key and softmax-value BMMs, which are crucial part of the computation, as well as elementwise multiplication in SwiGLU are not quantized in our simulations, and instead are kept in FP16.

15.0.0.2 Dynamic INT4 runtime

Figure 6 shows the prefill speedup of FPTQuant across different batch sizes and model sizes, assuming dynamic INT4 quantization. For most configurations, we still get a solid \(2.4\times\) – \(3.8\times\) speedup over the FP16 implementation. The speedup is again consistently increasing with model size and batch size, as the computation becomes the main bottleneck. FPTQuant is on par or faster than SpinQuant and consistently faster than FlatQuant, with a relative speedup of 11-21%. Similar to static case, FPTQuant is once again within a 3-6% to the INT4 upper bound.

16 Compute resources↩︎

16.1 Training cost↩︎

Although the FPTQuant transforms are mergeable (except Hadamard transform \(\boldsymbol{\mathrm{T}}_d\)), we need to consider their training cost.

We detail the training times of the runs from Table 2. For all methods, we trained with a batch size of 4, sequence length 2048, and gradient checkpointing per transformer block, which allowed us to run on a single A100 GPU. We time total training and average over \(1024 \times 4\) steps (i.e., 1024 training steps with gradient accumulation of 4), see Table 18.

The results are unsurprising. RTN-opt only optimizes the quantization parameters and is trivially the fastest. QuaRot achieves almost the same time—it only adds static Hadamard Transforms to RTN-opt, which incurs virtually no cost.

SpinQuant is significantly more expensive than QuaRot, because it optimizes the relatively high-dimensional rotation matrix \(\boldsymbol{\mathrm{T}}_r\) (\(R_1\) in their paper). This is more expensive than a standard matrix multiplication at training time, since it requires internally parametrizing the rotation matrix using either Cayley or matrix exponentials (see torch.nn.utils.parametrizations.orthogonal). On average, FPTQuant is slightly slower than SpinQuant. This makes sense, considering FPTQuant includes more optimizable transforms in addition to rotation transform \(\boldsymbol{\mathrm{T}}_r\). Note that even the most expensive training of FPTQuant (Llama-2 7B for 4096 steps) takes less than 1 single GPU day. Most expensive is FlatQuant, which includes multiple non-mergeable, trainable transforms, including multiple explicit reshapes of activations.

The relative training time of FPTQuant can decrease further with a bit of optimization when sequence length, batch size, or gradient accumulation increase—since all new FPTQuant transforms are mergeable (except the cheap dynamic per-token scaler), they can be merged into the weights prior to passing data and thus do not scale with input size. In contrast, FlatQuant transforms almost always have a forward transform applied online at both training and inference time, and thus scale linearly with the input batch size and sequence length.

Note: local optimization (Section 3.2.1) of FPTQuant is negligible; until convergence takes around 8 minutes for Llama-2 7B (1<% of training).

16.2 Training cost of inverse \(\mathbf{T}_v\)↩︎

We may wonder about the cost of the inverse of transform \(\boldsymbol{\mathrm{T}}_v\). Surprisingly, the inverse of \(\boldsymbol{\mathrm{T}}_v\) is cheap to compute during training. This is partly because of the dimension \(d_{\text{head}} = 128\) (for tested models), and partly because it can be computed in parallel across the heads. In Table 19 we time the inverse forward and backward passes, compared to SpinQuant’s rotation (parametrized in torch.nn.utils.parametrizations.orthogonal). We find that an inverse is cheaper to compute and backpropagate through than parametrized rotations.

16.2.0.1 SVD for inverse.

Large head dimensions can also be supported efficiently by using a singular value decomposition (SVD) to parametrize \(\boldsymbol{\mathrm{T}}_v\) during training. For one head \(h\), we parametrize: \[\boldsymbol{\mathrm{T}}_v^h = \mathbf{U} \, \text{diag}(\mathbf{s}) \, \mathbf{V}, \quad \text{with } \mathbf{s} \text{ a vector and } \mathbf{U}, \mathbf{V} \in O(d_{\text{head}})\] This requires rotation reparametrizations and more memory, but the inverse is simpler: \[\boldsymbol{\mathrm{T}}_v^h = \mathbf{V}^T \, \text{diag}(\mathbf{s}^{-1}) \, \mathbf{U}^T\] In our experiments, however, the SVD parametrization was slower and not more accurate than the direct inverse (see Table 19), even for very large head dimensions. This is probably due to the need for two orthogonal matrix parametrizations. Hence, in all the paper’s experiments we use the direct inverse.

16.3 Total compute cost for paper↩︎

All the experiments were executed on a single Nvidia A100 GPU equipped with 80GB of VRAM. Models of sizes 3B, 7B and 8B needed respectively around 9.1, 14.5, and 16.5 hours of training with FPTQuant, assuming the main setup with \(1024\) training steps, sequence length \(2048\), and a total batch size of \(16 = 4\) (per-device batch size) \(\times 4\) (gradient accumulation). For obtaining all the results in the paper, including the ablations, we needed 69.8 GPU days (A100). Including preliminary experiments that did not make it in the final paper and hyperparameter tuning we estimate the total compute costs of this research to approximately 386 GPU days.

17 Guide to choosing FPTs↩︎

When choosing FPTs, there is a trade-off between expressivity ([des:expressivity]) and cost ( [des:cost]). With FPTQuant, we have aimed to find maximally expressive FPTs that are mergeable or very cheap. FlatQuant is a strong baseline that regularly outperforms FPTQuant, although this incurs a cost. Fortunately, in practice we can choose on a case-by-case basis which FPTs to include. Here we provide a high-level guide to adding FPTs to your own model.

1. Explore. Evaluate quantization error per quantizer placement (e.g. Appendix 10)

2. Choose transforms. Based on step 1, choose which FPTs to add:

   1. Attention and FFN input. \(R_1\) (SpinQuant) and \(P_a,P_d\) (FlatQuant) are similar transforms. The first is shared across all layers of the model, whilst FlatQuant’s are not. However, an orthogonal matrix \(R_1\) has about \(d^2/2\) degrees of freedom, whereas each of FlatQuant’s Kronecker transforms only has about \(2d\) degrees of freedom.⁷ Additionally, \(R_1\) is mergeable, whereas \(P_a\) and \(P_d\) are not. As a result of this, \(R_1\) should have preference unless a per-layer independent FPT like \(P_a,P_d\) is warranted—e.g. if some layers have much higher quantization error than others.

   2. Keys and queries. Depending on how difficult queries and keys are to quantize, one can choose \(\boldsymbol{\mathrm{T}}_k\), \(R_3\) (SpinQuant), or \(P_h\) (FlatQuant), in increasing order of expense and power (see Appendix 11.1)

   3. Values. The \(\boldsymbol{\mathrm{T}}_v\) transform is more expressive than baselines and as a result better at reducing quantization error (Appendix 11). Since it is mergeable and hence free, this should always be used for improving value and out projection input

   4. Down projection input. For many networks, these activations are the trickiest to quantize (Appendix 10), which usually warrants an online transform (e.g. Hadamard). If a Hadamard is used, adding the mergeable \(\boldsymbol{\mathrm{T}}_u\) improves quantization further (Appendix 11.1).

   5. Residual A dynamic residual scaler \(S\) can aid quantization if the residual has large outliers in particular tokens. There are multiple possible placements for \(S\) (Section 3.1.3), e.g. on the softmax output and after SwiGLU.

3. Initialize FPTs. Initialize transforms, e.g. as a Welsh-Hadamard matrix or identity.

4. Locally optimize FPTs. Locally optimizing transforms improves performance and reduces training time, whilst incurring very little cost (Appendix 11.2.1).

5. Set quantization range. Set the initial quantization grid, e.g. using \(L_3\) minimization (Appendix 9). It is important to only set the grid now, so that initialized FPTs can be taken into account when choosing this grid.

6. Train end-to-end. Train the FPTs and quantization grid end-to-end, with the unquantized outputs as target.

18 Function-preservation and sensitivity analysis to noisy training↩︎

The function-preserving property (desideratum P1) of FPTs is useful because it reduces the capacity to change the pretrained model’s output, and consequently can avoid overfitting to calibration data. We have conducted a sensitivity analysis to show the function-preserving properties of different transforms without quantization. This also gives insight into training stability—if the output of the model is stable w.r.t. the parametrization of transforms, it means noisier gradient updates are less likely to lead to unstable training.

We take the initialized transforms, and simulate noisy training dynamics by perturbing the parameters; we add i.i.d. Gaussian noise with standard deviation \(\sigma\in \{0, 0.1, 0.3, 1.0, 3.0\}\) to each transform parameter. Naturally, we ensure the parametrizations remain correct—i.e. that an orthogonal matrix remains orthogonal (using torch.nn.utils.parametrization). We do not add quantizers, since we want to test desideratum [des:identity]. We run it for three model sizes of Llama-3.2/3, of 1B, 3B, and 8B parameters, and for 5 seeds. See Table 20.

SpinQuant and FPTQuant are very stable—even when we completely randomize the transform parameters, we are ensured that the function-preserving properties are in fact, preserved, and that output is stable. FlatQuant is theoretically function-preserving, but is less stable: their approach consists of 6 transforms per transformer block, which each have 1 or 2 online matrix inversions (or each consisting of SVD decompositions, consisting of multiple matrix multiplications). The latter can result in floating point precision issues which destroy the function-preservation, roughly observed from \(\sigma=0.3\) (for Llama-3.2 3B-it) and completely destroying the model performance at \(\sigma=1\). We have found this is not an issue during training as long as a small learning rate is chosen (i.e. the noise is small and the optimizer can correct errors in later steps).

OSTQuant is not function-preserving; it uses smoothing transforms that do not cancel each other out. For example, their \(S_{qk}\) transform does not commute with RoPE, and hence the query and key transforms do not cancel out (i.e. no function-preservation). We see this in the results. Smoothing vectors are initialized as identities, hence without noise the model works as expected (yields identical output to the original full precision network). When the transform weights are updated even with relatively small noise (\(\sigma=0.3\)), the model is no longer function-preserving and deviates significantly from the original model. This also means that the model has capacity to overfit the training data.

References↩︎