1 Introduction↩︎

Post-training quantization (PTQ) is one of the most practical ways to reduce the memory and compute cost of large language model (LLM) inference. Its main challenge, however, lies in low-bit activation quantization. Unlike weights, LLM activations often contain a small number of dominant coordinates that enlarge the quantization range and force round-to-nearest quantization to map many normal values to the same few levels. This mismatch becomes particularly severe in 4-bit settings, where limited quantization levels leave little room to preserve both rare extremes and dense central values.

Recent PTQ methods increasingly address this problem through activation transformations before quantization. SmoothQuant [1] migrates activation difficulty into weights through diagonal scaling, while QuaRot [2] and SpinQuant [3] use orthogonal rotations to redistribute activation energy; other methods further introduce more flexible affine transformations or reconstruction objectives [4], [5]. Although these methods differ in form, they share the same practical role: they change the activation distribution seen by the quantizer. Yet most of them are motivated by suppressing outliers, balancing channels, or reducing reconstruction error, rather than by defining what transformed distribution a low-bit quantizer can represent well. Consequently, as illustrated in Figure 1, they may reduce visible peaks without fully improving discretizability, or preserve small numerical error while still collapsing distributional resolution. The central question, therefore, is not only how to transform activations, but what transformed activation distribution is actually quantization-friendly.

We address this gap by recasting activation transformation as quantizer-facing distribution design. From an information-theoretic perspective, we analyze how quantization error depends on the activation distribution after transformation. Our theoretical and empirical study shows that lower quantization error is associated with two complementary properties: a smaller numerical range and greater dispersion within that range. This result reframes the role of activation transformation. Rather than merely suppressing outliers or minimizing heuristic reconstruction losses, a PTQ method should explicitly shape activations into distributions that are easier for a low-bit quantizer to preserve. Since LLM activations are typically bell-shaped and often contain outliers [6], they are naturally misaligned with this target, which explains why low-bit activation quantization remains difficult.

Guided by this principle, we introduce InfoQuant, a train-free PTQ method that learns orthogonal transformations to produce more quantization-friendly activation distributions. Its core component is Peak Suppression Orthogonal Transformation(PSOT), which applies an activation-wise peak suppression objective to reduce the numerical range while increasing normalized dispersion. We further introduce adaptive outlier-token selection to improve optimization robustness, and learn activation clipping parameters to refine the final quantization range after the distribution has been reshaped. Although InfoQuant is designed around activation optimization, it remains compatible with standard weight quantization pipelines. Overall, our contributions can be summarized as follows:

• We introduce an information-theoretic framework for understanding activation quantization error and show, both theoretically and empirically, that quantization-friendly distributions should have a smaller numerical range and greater dispersion.

• We introduce InfoQuant, a hardware-efficient and train-free PTQ method centered on learned orthogonal activation shaping, together with adaptive outlier-token selection and learnable activation clipping for robust calibration.

• We demonstrate that activation-distribution optimization yields strong empirical and practical gains. In the W4A4KV4 setting, LLaMA-2 (7B, 13B, 70B) and LLaMA-3 (8B, 70B) retain an average of \(97\%\) of their original performance, and the 70B model can be quantized using only \(24\)GB of GPU memory.

2 Related Work↩︎

2.0.0.1 Post Training Quantization for LLMs.

PTQ is an efficient and widely used approach for compressing LLMs. Due to the flatness and uniform distribution of LLM weights, weight-only quantization typically results in minimal performance degradation. GPTQ [7] uses Hessian-based error compensation to enable high compression with low accuracy loss. AWQ [8] and OWO [9] further improve performance by mitigating the effects of activation outliers. QuIP [10] and QuIP# [11] apply random Hadamard transforms for incoherent processing and employ vector quantization on weights, achieving better performance. In contrast, activation quantization remains more challenging due to the presence of rare but extreme outliers [1], [12], which can disproportionately affect accuracy.

2.0.0.2 Transformation-based Methods.

These methods more effectively redistribute activation outliers across channels. Channel scaling [1] shifts part of this burden to weights, OmniQuant [13] and LRQuant [14] optimize scaling parameters via MSE minimization. However, recent work [15] shows channel scaling alone fails under 4-bit settings, leading to notable degradation. AffineQuant [4] learns affine transformations to precondition activations. However, due to the significant overhead of full-size matrix multiplication, AffineQuant can only apply affine transformations to a small fraction of linear layers. FlatQuant [5] reduces this cost via Kronecker decomposition, applying affine transformation to every linear layer. Leveraging computational invariance [16], orthogonal transforms can be applied to weights and between-block activations without extra inference overhead. QuaRot [2] uses randomized Hadamard transforms to remove outliers. SpinQuant [3] further optimizes learnable orthogonal matrices on the Stiefel manifold with task loss (e.g., cross-entropy) to find stable transformations. OSTQuant [17] combines channel scaling with orthogonal transforms and uses end-to-end distillation from original outputs to boost quantization. Kurtail [18] facilitates quantization by controlling the kurtosis to make the distribution more uniform. BASE-Q [19] introduces an additional bias term to balance the mean values of different channels after rotation.

3 Motivation↩︎

3.1 Quantization Preliminaries↩︎

Quantization maps high-precision values to a set of discrete levels. The process is detailed as follows: \[\mathcal{Q}(\mathbf{X}) = \text{clamp}\left( \left\lfloor \frac{\mathbf{X}}{s} \right\rceil + z,\;0,\;2^N - 1 \right)\] Here, quantization step size is denoted by \(s = \frac{\mathbf{X}_{\max} - \mathbf{X}_{\min}}{2^N - 1}\), and \(z = -\left\lfloor \frac{\mathbf{X}_{\min}}{s} \right\rceil\) is the corresponding zero-point, \(\left\lfloor \cdot \right\rceil\) denotes the rounding operation, and \(N\) represents the target bit-width. Given a floating-point tensor \(\mathbf{X}\), the quantization function \(\mathcal{Q}(\cdot)\) produces its integer-valued representation. Quantization error primarily arises from the rounding operation, which collapses all values within a single interval of size \(s\) into the same discrete level.

3.2 A Distributional View of Quantization Error↩︎

Recent PTQ methods often optimize activation transformations with MSE-based objectives [5], [13], [14]. While MSE is a useful measure of numerical distortion, it does not fully capture the distributional mismatch introduced by low-bit quantization. This limitation is especially important for activation quantization under round-to-nearest (RTN), where many values may incur only small pointwise errors yet still be mapped to a small number of discrete levels. In such cases, the quantized activations can remain close in value to the original ones while losing substantial distributional resolution, which is not well reflected by MSE alone.

Prior work [6] has shown that activation distributions in LLMs are typically bell-shaped (e.g., Gaussian or Laplace). When such activations are quantized with a low-bit uniform RTN quantizer, a few large values can determine the quantization range, forcing most normal values to collapse toward levels near the mean (Figure 2, bottom). The resulting error is therefore not only a matter of local rounding distortion, but also of how poorly the available quantization levels match the underlying activation distribution. Although non-uniform quantizers can in principle better adapt to dense regions, they usually introduce additional hardware complexity and are less attractive in practical low-bit deployment. These observations motivate an analytical metric that reflects both numerical deviation and quantization-induced distribution shift. To address this, we use a smoothed KL divergence as an analytical lens for the distributional distortion caused by low-bit quantization. Let \(\mathbf{x}\) denote an activation token, and let each entry be denoted by a scalar \(x \in \mathbf{x}\) with distribution \(P(x)\). We consider a centered finite-bit clamped quantizer \[\hat{x}=Q_{s,c}(x)=\operatorname{clip}\!\left(s\left\lfloor \frac{x}{s}\right\rceil,\,-c,\,c\right),\] where \(s\) is the quantization step size and \(c\) is the clipping scale. Directly comparing \(P(x)\) with the quantized distribution is ill-posed as a density-to-density KL, because quantization turns a continuous density into probability masses on discrete centroids. We therefore spread each centroid into a narrow continuous kernel. Specifically, let \(q_i\) denote a quantization centroid, let \(I_i=\{x\mid Q_{s,c}(x)=q_i\}\) be its quantization cell, and define the corresponding probability mass \(p_i=\int_{I_i}P(x)\,dx\). The quantized distribution is relaxed as \[Q_\theta^{(c)}(x)\approx \sum_i p_i \cdot \delta(x-q_i;\theta),\] where \(\delta(x;\theta)=\frac{1}{2\theta}\exp\!\left(-\frac{|x|}{\theta}\right)\) is a Laplace kernel that approaches a Dirac delta as \(\theta\to 0\). Under the standard separation assumption \(\theta\ll s\), the smoothed KL objective admits the approximation: \[\begin{align} &D_{\mathrm{KL}}\left(P \,\|\, Q_\theta^{(c)}\right) = \int P(x)\, \log\!\left(\frac{P(x)}{Q_\theta^{(c)}(x)} \right) dx \nonumber \\ &\approx -H(P) + H(\{p_i\}) + \log(2\theta) + \frac{1}{\theta}\, \mathcal{E}_{\mathrm{clip}}, \end{align}\] where \(H(P)\) denotes the entropy of the original distribution, \(H(\{p_i\})\) is the entropy of the quantized probability masses, and \(\mathcal{E}_{\mathrm{clip}}=\mathbb{E}|x-Q_{s,c}(x)|\) is the expected absolute error of the finite-bit clamped quantizer. This decomposition exposes the key mechanism: after smoothing the discrete outputs, the distributional KL surrogate contains a direct quantization-error term. Thus, a transformation that makes activations easier to quantize should not only suppress extreme values, but also reduce the normalized error induced by the finite set of quantization levels.

Since activations typically have bounded yet varying ranges, we normalize the clamped quantization error by the standard deviation \(\sigma\): \[\begin{align} \mathcal{E}_{\mathrm{clip}}' &= \frac{1}{\sigma}\,\mathbb{E}|x-Q_{s,c}(x)| \\ &\le \frac{s}{2\sigma} + \frac{1}{\sigma}\,\mathbb{E}\!\left[(|x|-c)_+\right]. \end{align}\] The two terms reveal the trade-off that ordinary outlier suppression does not fully describe. The first term is the in-range rounding error, which decreases when the step size is small relative to the activation spread. The second term is the clipping-tail error, which measures the mass left outside the finite range. Let \(\lambda=s/\sigma\) and \(\kappa=c/\sigma\). Then \[\mathcal{E}_{\mathrm{clip}}' \le \frac{\lambda}{2} + \tau_P(\kappa),\] where \(\tau_P(\kappa)=\mathbb{E}\!\left[(|Y|-\kappa)_+\right]\) for the normalized variable \(Y=x/\sigma\). For common bell-shaped activation distributions such as Gaussian and Laplace, \(\tau_P\) admits closed forms, and for a \(B\)-bit quantizer with \(c=Ms\) and \(M=2^{B-1}-1\), the resulting bound decreases as \(\kappa\) decreases in the tail-controlled regime relevant to calibrated PTQ.² Since \(\kappa=c/\sigma=1/b_n\) and \(\lambda=s/\sigma=\bar{s}/b_n\) with \(b_n=\sigma/c\), this analysis turns the vague goal of “making activations smoother” into a concrete distributional target: reduce the clipped numerical range while keeping the normalized activation values well dispersed inside that range. Empirically, as shown in the top of Figure 2, KL divergence decreases consistently with smaller \(s\) and larger \(b_n\). Together, the analysis and observation suggest a simple design principle for low-bit activation quantization: a good transformation should compress the effective range and spread useful activation mass across more available quantization levels.

4 Method↩︎

InfoQuant is a train-free PTQ framework that reshapes activations into distributions better matched to low-bit quantizers. It consists of three components: Peak Suppression Orthogonal Transformation(PSOT) learns an orthogonal activation transformation, adaptive outlier-token selection (ASOT) emphasizes informative calibration tokens, and learnable activation clipping (LAC) refines the final quantization interval.

4.1 Peak Suppression Orthogonal Transformation↩︎

PSOT learns a quantizer-facing orthogonal transformation by directly penalizing peak-dominated activation tokens. For each target activation stream, we optimize a block-diagonal orthogonal rotation on calibration activations and initialize it from a Hadamard transform, preserving the efficient rotation-based deployment path used by existing PTQ systems. Let \(\mathbf{x}\in\mathbb{R}^{d}\) denote one activation token, and let \(\mathbf{R}\in\mathbb{R}^{d\times d}\) be a learnable orthogonal matrix with \(\mathbf{R}^{\top}\mathbf{R}=\mathbf{I}\). We remove the coordinate-wise mean after rotation using the centering projector \(\mathbf{P}_{\perp}=\mathbf{I}-\frac{1}{d}\mathbf{1}\mathbf{1}^{\top}\), and define \(\boldsymbol{\pi}_T(\mathbf{y})=\operatorname{softmax}(|\mathbf{y}|/T)\): \[\begin{align} \mathbf{y}(\mathbf{x};\mathbf{R}) &= \mathbf{x}\mathbf{R}\mathbf{P}_{\perp}, \nonumber \\ \ell_{\mathrm{ps}}(\mathbf{x};\mathbf{R}) &= \left\| \boldsymbol{\pi}_T(\mathbf{y}) \odot \mathbf{y} \right\|_2 , \label{eq:ommin} \end{align}\tag{1}\] where \(T\) is the temperature and \(\odot\) denotes element-wise multiplication. The softmax weights concentrate the objective on high-magnitude coordinates, so minimizing \(\ell_{\mathrm{ps}}\) suppresses the coordinates that dominate the quantization range.

This peak-suppression objective promotes the two distributional properties identified by the KL analysis. Since \(\mathbf{R}\) is orthogonal, the rotation preserves token energy before centering. Reducing the largest centered coordinate therefore pushes the remaining energy to spread across more dimensions, which lowers the clipping scale and increases the max-normalized dispersion \(b_n\). As illustrated in Figure 1, the learned rotation yields a narrower range and a more dispersed normalized distribution than both the original activations and a fixed Hadamard rotation.³

4.2 Adaptive Outlier-Token Selection↩︎

PSOT obtains the strongest learning signal from tokens with quantization-sensitive outlier structure. Uniformly optimizing all calibration tokens can dilute this signal because many tokens have weak or noisy peaks. We therefore use ASOT to select reliable outlier tokens and reweight them during rotation optimization.

Let \(\mathbf{X}^{(r)}\in\mathbb{R}^{n_r\times d}\) be the activation matrix of the \(r\)-th calibration sample, where \(\mathbf{x}^{(r)}_i\) is its \(i\)-th token. For a threshold coefficient \(k\), we define the selected token-index set as \[\begin{align} o_i^{(r)} &= \left\| \frac{\mathbf{x}^{(r)}_i-\mu\mathbf{1}}{\sigma} \right\|_{\infty}, \nonumber\\ \mathcal{T}^{(r)}(k) &= \left\{ i \mid o_i^{(r)} > k \right\}, \label{eq:token95set} \end{align}\tag{2}\] where \(\mu\) and \(\sigma\) are estimated from the corresponding calibration activations. This criterion keeps tokens that contain at least one statistically extreme coordinate.

The threshold should select sparse outlier tokens without overfitting to fixed sequence positions. Fixed-position peaks can reflect prompt layout or calibration artifacts, whereas sample-dependent outliers provide a more useful signal for learning a rotation that generalizes across inputs. Following the observation that activation outliers are more closely tied to token identity than absolute sequence position [20], [21], we measure whether the selected positions vary across \(m\) calibration samples: \[\begin{align} \eta_m(k) = 1 - \frac{\frac{1}{m}\sum_{r=1}^{m}|\mathcal{T}^{(r)}(k)|}{\left|\bigcup_{r=1}^{m}\mathcal{T}^{(r)}(k)\right|}. \label{eq:eta} \end{align}\tag{3}\] where a larger \(\eta_m(k)\) indicates that the selected outlier positions are less tied to fixed sequence locations. We choose the smallest threshold at which the inconsistency curve stabilizes while the selected tokens remain sparse: \[\begin{align} k^{\star} = \min \left\{ k\in\mathcal{K} \,\middle|\, \begin{aligned} &|\nabla\eta_m(k)| < \delta,\\ &\frac{1}{m}\sum_{r=1}^{m}|\mathcal{T}^{(r)}(k)| < \tau |\mathcal{I}| \end{aligned} \right\}, \label{selek} \end{align}\tag{4}\] where \(\mathcal{K}\) is the ordered threshold grid, \(\mathcal{I}\) is the token-index universe, \(\delta\) controls the stabilization tolerance, and \(\tau\) limits the selected-token ratio. With the selected sets \(\mathcal{T}^{(r)}(k^\star)\), the final rotation objective is \[\begin{align} \min_{\mathbf{R}^{\top}\mathbf{R}=\mathbf{I}} &\sum_{r=1}^{m}\sum_{i\in\mathcal{I}} w_i^{(r)}\,\ell_{\mathrm{ps}}(\mathbf{x}^{(r)}_i;\mathbf{R}), \nonumber\\ w_i^{(r)} &= \begin{cases} \gamma, & i\in\mathcal{T}^{(r)}(k^\star),\\ 1, & \text{otherwise}. \end{cases} \label{eq:gamma} \end{align}\tag{5}\] where \(\gamma>1\) emphasizes outlier tokens while retaining normal tokens as regularizing calibration samples. All ASOT hyperparameters are selected by grid search on the calibration set, and the effect of \(\gamma\) is studied in Table 4.

4.3 Learnable Activation Clipping↩︎

After PSOT reshapes the activation distribution, we apply learnable activation clipping as a final calibration step. Following prior work [5], [14], two bounded parameters \(\alpha\) and \(\beta\) refine the activation step size: \[\begin{align} s(\alpha,\beta) = \frac{\alpha X_{\max}-\beta X_{\min}}{2^N-1}, \label{eq:lac95step} \end{align}\tag{6}\] where \(N\) is the bit-width, and \(X_{\max}\) and \(X_{\min}\) are the observed activation bounds. The clipping parameters are optimized by matching the quantized block output to the full-precision block output: \[\begin{align} \min_{\alpha,\beta} \left\| \mathcal{F}_{l}(\mathbf{X}_{l}) - \widehat{\mathcal{F}}_{l}(\mathbf{X}_{l};\alpha,\beta) \right\|_2^2, \label{eq:lac95obj} \end{align}\tag{7}\] where \(\mathcal{F}_{l}\) denotes the full-precision Transformer block and \(\widehat{\mathcal{F}}_{l}\) denotes the same block evaluated with activation quantization under Eq. 6 .

5 Experiments↩︎

5.0.0.1 Models and Datasets.

We evaluate whether activation-distribution optimization transfers across model families, scales, and evaluation metrics. The main comparison covers LLaMA-2 (7B–70B) [22] and LLaMA-3 (8B–70B) [23]; additional Qwen2.5 (14B/32B) [24] results are reported in Appendix 10.0.0.1. We report WikiText2 perplexity (PPL) [25] as a sensitive language-modeling metric and use nine zero-shot tasks from lm-evaluation-harness (version 0.4.7) [26] to check whether lower quantization error translates to task-level behavior. The zero-shot suite includes BoolQ [27], HellaSwag [28], LAMBADA (OpenAI) [29], OpenBookQA (OBQA) [30], PIQA [31], SIQA [32], WinoGrande [33], ARC-Easy, and ARC-Challenge [34].

5.0.0.2 Baselines.

We compare with representative quantization methods that stress different parts of the design space: weight reconstruction methods GPTQ [7] and AWQ [8], rotation-based methods QuaRot [2] and SpinQuant [3], and recent low-bit LLM quantizers Kurtail [18], BASE-Q [19], and OSTQuant [17]. This comparison is useful because InfoQuant changes the activation distribution before quantization, whereas several baselines mainly reconstruct weights, use fixed rotations, or rely on stronger supervision; in particular, the distilled OSTQuant results are included as a strong supervised reference.

5.0.0.3 Implementation Details.

The calibration process uses 128 samples from WikiText-2, each with a sequence length of 2048. Activations are quantized using per-token asymmetric quantization, while weights are quantized using asymmetric per-channel quantization with GPTQ [7], applying a group size of 128 for key-value matrices. During the PSOT phase, we optimize block-diagonal orthogonal matrices initialized with Hadamard matrices via Cayley SGD [35]. The ASOT hyperparameters are selected by grid search on the calibration set, and the final search space and chosen values are reported in Appendix 8 and Appendix 9. We report two variants to separate accuracy and deployment considerations. InfoQuant uses block-diagonal rotations to reduce transformation overhead, while InfoQuant* uses a full global orthogonal rotation similar to SpinQuant. More implementation details are provided in Appendix 8.

5.1 Main Results↩︎

5.1.0.1 Quantization Performance.

Table [MainResults] shows that the value of activation-distribution optimization becomes visible when activations are quantized. In the weight-only 4-16-16 setting, InfoQuant preserves \(98.7\%\)–\(99.5\%\) of the floating-point zero-shot accuracy across the evaluated LLaMA models, but the gap among strong rotation-based methods is relatively small. This pattern is informative: when activations remain in high precision, reshaping them is not the dominant bottleneck. The setting mainly verifies that the learned rotation does not damage the weight-only quantization path.

Once activations are quantized, the table reveals different behavior. Under 4-4-16, InfoQuant improves the average zero-shot accuracy over SpinQuant by \(2.91\) points across the five LLaMA settings, with larger gains on LLaMA-2 7B (\(+5.47\)) and LLaMA-3 70B (\(+3.72\)). Under W4A4KV4, where weights, activations, and KV cache are all quantized to 4 bits, InfoQuant preserves \(96.9\%\) of floating-point accuracy on average and improves over SpinQuant by \(1.70\) points. These results support the main design intuition of the paper: fixed or generic rotations are often sufficient to avoid catastrophic outliers, but low-bit activation quantization benefits from learning a distribution that uses the available quantization levels more evenly.

The comparison with OSTQuant also clarifies the boundary of the method. InfoQuant outperforms the distillation-based OSTQuant baseline on LLaMA-3 8B and LLaMA-2 13B, while OSTQuant remains competitive on several 70B entries. This mixed pattern is useful rather than merely negative: it suggests that distribution shaping can recover a large part of the activation-quantization loss without full-precision supervision, but supervision and scale-specific calibration may still help in some large-model regimes. Full per-task results are reported in Appendix 10.0.0.2.

5.1.0.2 Speedup and Memory Savings.

We evaluate inference efficiency using the W4A4 kernel from [2]. Table 1 reports prefill speedup and decoding memory savings relative to FP16 on a single Transformer block, using batch size 4 on an NVIDIA RTX 4090. On LLaMA-2 70B with sequence length 2048, InfoQuant* achieves a \(2.61\times\) prefill speedup and \(3.25\times\) memory saving, while InfoQuant achieves a \(2.46\times\) prefill speedup and \(2.91\times\) memory saving. The difference between InfoQuant and InfoQuant* exposes the main deployment trade-off: full rotations can be slightly faster in this implementation path, whereas block-diagonal rotations provide a more flexible per-layer optimization structure.

We further compare quantization-time memory overhead and inference-time transformation FLOPs on LLaMA-3 70B. As shown in Figure 4, InfoQuant* requires less than 24GB of GPU memory during quantization and introduces low additional transformation cost. The broader insight is that activation shaping should not be evaluated only by accuracy: a transformation that must be repeatedly applied online can erase part of the benefit of low-bit inference. FlatQuant, for example, performs multiple dynamic activation transformations during inference, which increases transform FLOPs and depends on specialized kernels. InfoQuant instead keeps the rotation-based deployment path lightweight, making the method more compatible with existing low-bit kernels and consumer-grade quantization hardware.

5.2 Ablation Study↩︎

5.2.0.1 Module-wise Impact.

Table 2 isolates how each component contributes to InfoQuant under the W4A4KV4 setting. Replacing a fixed Hadamard rotation with the learned PSOT rotation reduces WikiText2 perplexity from \(10.90\) to \(8.84\) on LLaMA-3 8B and from \(8.99\) to \(7.03\) on LLaMA-2 7B. This is the largest single change in the ablation, which suggests that the core gain comes from learning where activation energy should be redistributed, not from simply adding more calibration stages.

The later components produce smaller but more diagnostic changes. Adding GPTQ further reduces perplexity to \(7.53\) and \(6.01\), indicating that activation-oriented rotation remains compatible with standard weight reconstruction. ASOT gives a modest improvement, which is consistent with its role as a signal reweighting mechanism rather than a separate quantizer. LAC then gives the best perplexity on both models (\(7.16\) and \(5.89\)). The ordering of these gains is important: first reshape the distribution, then reconstruct weights, then refine which calibration tokens and clipping ranges deserve attention.

5.2.0.2 More Ablations.

Additional ablations in Appendix 9 study the weighting factor \(\gamma\), the temperature \(T\), initialization robustness, clipping-ratio sensitivity, and the block size of block-diagonal orthogonal matrices. These experiments provide practical guidance for using InfoQuant: objective reweighting and temperature control the stability of peak suppression, while block size and clipping range determine how much distribution-shaping flexibility can be traded for lower inference cost and easier calibration.

6 Conclusion↩︎

This work studies low-bit LLM quantization from the perspective of quantizer-facing activation distribution design. By analyzing the distributional error introduced by quantization, we show that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this principle, InfoQuant reshapes activations with Peak Suppression Orthogonal Transformation, emphasizes informative calibration tokens with ASOT, and refines the quantization interval with learnable activation clipping. Our results suggest that activation quantization should be evaluated by both outlier reduction and effective use of available low-bit levels. A natural next step is to extend this analysis beyond bell-shaped activation assumptions and develop transformations for broader distributional regimes.

Limitations↩︎

The findings of this paper should be interpreted within the scope of the evaluated settings. We validate InfoQuant on a limited set of model families, tasks, and quantization configurations, and do not claim that the same gains will automatically transfer to substantially different architectures, activation regimes, or deployment scenarios without additional study. Our method is motivated by activation patterns that have been widely observed in prior studies and are also present in the models evaluated in this paper. While these observations are sufficient to support the improvements reported here, broader validation would still be useful to determine how consistently the same behavior holds across other model families and quantization settings. From a practical perspective, the method still relies on calibration data and implementation choices such as transformation and clipping settings. Although the approach is intended for practical post-training quantization, its effectiveness may therefore vary when calibration conditions differ substantially from those used in evaluation.

Appendix Overview↩︎

• Section 7: Theory proofs.

• Section 8: Additional implementation details.

• Section 9: More ablations.

• Section 10.0.0.2: Full results.

• Section 11: Visualization results.

7 Theory Proofs↩︎

7.1 Proof of Theorem 1↩︎

For the finite-bit clamped quantizer, the interior cells are \[\begin{align} I_i&=\left[\left(i-\frac{1}{2}\right)s,\left(i+\frac{1}{2}\right)s\right),\\ &\qquad |i|<M, \end{align}\] while the boundary cells absorb the clipped tails: \[\begin{align} I_M &= \left[c-\frac{s}{2},\infty\right),\\ I_{-M} &= \left(-\infty,-c+\frac{s}{2}\right]. \end{align}\] For \(x\in I_i\), the quantizer outputs the centroid \(q_i=is\). Under the kernel-separation assumption \(\theta\ll s\), neighboring kernels contribute negligibly around each cell, and we may approximate \[\begin{align} Q_\theta^{(c)}(x) &\approx p_i\,\delta(x-q_i;\theta)\\ &= \frac{p_i}{2\theta}\exp\!\left(-\frac{|x-q_i|}{\theta}\right),\qquad x\in I_i. \end{align}\] Substituting this expression into the KL divergence gives \[\begin{align} D_{\mathrm{KL}}(P\|Q_\theta^{(c)}) &= \sum_{i=-M}^{M}\int_{I_i} P(x) \log\!\left(\frac{P(x)}{Q_\theta^{(c)}(x)}\right)dx \nonumber \\ &\approx \sum_{i=-M}^{M}\int_{I_i} P(x)\log P(x)\,dx \nonumber \\ &\quad - \sum_{i=-M}^{M}\log(p_i)\int_{I_i}P(x)\,dx \nonumber \\ &\quad + \frac{1}{\theta}\sum_{i=-M}^{M}\int_{I_i}|x-q_i|P(x)\,dx \nonumber \\ &\quad + \log(2\theta). \end{align}\] Using \[\begin{align} \sum_i\int_{I_i}P(x)\log P(x)\,dx &= -H(P),\\ \sum_i \log(p_i)\int_{I_i}P(x)\,dx &= \sum_i p_i\log p_i\\ &= -H(\{p_i\}), \end{align}\] we obtain \[\begin{align} D_{\mathrm{KL}}(P\|Q_\theta^{(c)}) &\approx -H(P)+H(\{p_i\})+\log(2\theta) \nonumber \\ &\quad + \frac{1}{\theta}\mathcal{E}_{\mathrm{clip}}, \end{align}\] where \[\begin{align} \mathcal{E}_{\mathrm{clip}} &=\sum_{i=-M}^{M}\int_{I_i}|x-q_i|P(x)\,dx\\ &=\mathbb{E}|X-Q_{s,c}(X)|. \end{align}\]

To upper bound the practical clamped-quantization error, define the clipping projection \(\Pi_c(x)=\operatorname{clip}(x,-c,c)\). Then \[\begin{align} |x-Q_{s,c}(x)| &\le |x-\Pi_c(x)| \nonumber \\ &\quad + |\Pi_c(x)-Q_{s,c}(x)| \nonumber \\ &\le (|x|-c)_+ + \frac{s}{2}. \end{align}\] The second term is the standard in-range round-to-nearest error, while the first term is the clipping-tail error. Taking expectations and dividing by \(\sigma\) gives \[\begin{align} \mathcal{E}_{\mathrm{clip}}' &:= \frac{1}{\sigma}\mathbb{E}|X-Q_{s,c}(X)| \nonumber \\ &\le \frac{s}{2\sigma} + \frac{1}{\sigma}\mathbb{E}\!\left[(|X|-c)_+\right]. \end{align}\] With the normalized variable \(Y=X/\sigma\), \(\lambda=s/\sigma\), and \(\kappa=c/\sigma\), this becomes \[\begin{align} \mathcal{E}_{\mathrm{clip}}' &\le \frac{\lambda}{2} + \tau_P(\kappa), \nonumber \\ \tau_P(\kappa) &:= \mathbb{E}\!\left[(|Y|-\kappa)_+\right]. \end{align}\] Because \(c=Ms\) for a fixed \(B\)-bit quantizer, we have \(\kappa=M\lambda\) and therefore \[F_{P,B}(\kappa)=\frac{\kappa}{2M}+\tau_P(\kappa).\]

7.1.0.1 Gaussian activations.

If \(X\sim\mathcal{N}(0,\sigma^2)\), then \(Y\sim\mathcal{N}(0,1)\) with density \(\phi(y)=\frac{1}{\sqrt{2\pi}}e^{-y^2/2}\) and survival function \(\bar{\Phi}(y)=1-\Phi(y)\). The clipping-tail term is \[\begin{align} \tau_G(\kappa) &= 2\int_{\kappa}^{\infty}(y-\kappa)\phi(y)\,dy \nonumber \\ &= 2\big(\phi(\kappa)-\kappa\bar{\Phi}(\kappa)\big). \end{align}\] Hence \[\begin{align} F_{G,B}(\kappa) &=\frac{\kappa}{2M}+2\big(\phi(\kappa)-\kappa\bar{\Phi}(\kappa)\big), \end{align}\] and \[F_{G,B}'(\kappa)=\frac{1}{2M}-2\bar{\Phi}(\kappa).\] Therefore \(F_{G,B}\) is increasing whenever \[\kappa \ge \kappa_G^\star := \Phi^{-1}\!\left(1-\frac{1}{4M}\right).\] In this tail-controlled regime, decreasing \(\kappa=c/\sigma\) reduces the bound.

7.1.0.2 Laplace activations.

If \(X\sim\mathrm{Laplace}(0,b)\), then \(\sigma=\sqrt{2}\,b\). We keep the same normalized error definition as above, \[\mathcal{E}_{\mathrm{clip}}'=\frac{1}{\sigma}\mathbb{E}|X-Q_{s,c}(X)|,\] so the Laplace case is parameterized by the scale \(b\) only through the identity \(\sigma=\sqrt{2}b\). With the normalized variable \(Y=X/\sigma\), the density becomes \[f_Y(y)=\frac{1}{\sqrt{2}}e^{-\sqrt{2}|y|}.\] The clipping-tail term becomes \[\begin{align} \tau_L(\kappa) &= 2\int_{\kappa}^{\infty}(y-\kappa)\frac{1}{\sqrt{2}}e^{-\sqrt{2}y}\,dy \nonumber \\ &= \frac{1}{\sqrt{2}}e^{-\sqrt{2}\kappa}. \end{align}\] Hence \[\begin{align} F_{L,B}(\kappa)&=\frac{\kappa}{2M}+\frac{1}{\sqrt{2}}e^{-\sqrt{2}\kappa},\\ F_{L,B}'(\kappa)&=\frac{1}{2M}-e^{-\sqrt{2}\kappa}. \end{align}\] Therefore \(F_{L,B}\) is increasing whenever \[\kappa \ge \kappa_L^\star := \frac{\log(2M)}{\sqrt{2}}.\] Again, in this regime, decreasing \(\kappa\) tightens the bound. Equivalently, one may express the same condition in terms of \(b\) via \(\kappa=c/(\sqrt{2}b)\), but the normalized error itself remains defined with respect to \(\sigma\).

Finally, the normalized dispersion metric used in the main text is \[b_n=\frac{\sigma}{c}=\frac{1}{\kappa}, \qquad \lambda=\frac{s}{\sigma}=\frac{s/c}{\sigma/c}=\frac{\bar{s}}{b_n}.\] Therefore, for a fixed bit-width and a controlled clipping tail, a smaller clipped range relative to the activation spread and a larger max-normalized dispersion \(b_n\) both correspond to a smaller \(\kappa\) and a tighter error bound.

7.2 Why \(b_n\) Tends to Increase Under PSOT↩︎

Based on the analysis in the motivation section, the normalized surrogate error is controlled by the ratio between the quantization step size and the distributional spread. Since practical activation quantization uses a clipped numerical range, we normalize both quantities by the clipping scale. This yields the dispersion metric \(b_n\) used in the main text.

Suppose we apply a rotation to an activation token \(\mathbf{t}\) with \(d\) dimensions using an orthogonal matrix \(\mathbf{A}\), and then center it as in Eq. 1 , yielding \(\mathbf{t}' = \mathbf{t}\mathbf{A} - \mathbb{E}[\mathbf{t}\mathbf{A}]\). The optimization objective aims to reduce \(\|\mathbf{t}'\|_{\infty}\), which lowers the clipping scale that determines the quantization step size. Since \(\mathbf{A}\) is orthogonal, the Euclidean norm remains invariant under rotation, and centering only removes the mean component: \[\begin{align} \|\mathbf{t}'\|_2 \le \|\mathbf{t}\mathbf{A}\|_2 = \|\mathbf{t}\|_2 \end{align}\] For a centered token, the range-normalized dispersion satisfies \[\begin{align} b_n(\mathbf{t}') = \sqrt{\frac{1}{d}} \frac{\|\mathbf{t}'\|_2}{\|\mathbf{t}'\|_\infty}. \end{align}\] Comparing the transformed and original tokens gives \[\begin{align} \frac{b_n(\mathbf{t}')}{b_n(\mathbf{t})} = \frac{\|\mathbf{t}'\|_2}{\|\mathbf{t}\|_2} \cdot \frac{\|\mathbf{t}\|_\infty}{\|\mathbf{t}'\|_\infty}. \end{align}\] Therefore, when PSOT suppresses the peak value while preserving most of the centered token energy, the decrease in \(\|\mathbf{t}'\|_\infty\) can dominate the mild change in \(\|\mathbf{t}'\|_2\), causing \(b_n(\mathbf{t}')\) to increase. This is a conditional mechanism rather than a universal guarantee, but it explains why peak suppression empirically tends to produce both a smaller effective clipping range and a more dispersed normalized distribution.

8 Additional Implementation Details↩︎

8.0.0.1 Additional Setup.

To improve efficiency, block diagonal matrices are used, partitioned into two blocks for 7B–13B models and four blocks for the 70B model. The PSOT temperature is set to \(T=2\), with a batch size of \(4\) and an initial learning rate of \(2\), linearly decayed over 15 epochs. For ASOT, we use \(m=10\) calibration samples and select the hyperparameters by grid search on the calibration set; the final setting uses \(\delta=0.02\), \(\tau=0.4\), and \(\gamma=30\). In the subsequent LAC phase, the initial clipping ratio is set to \(0.95\), constrained within the interval \([0.5,1]\). We use the AdamW optimizer with an initial learning rate of \(0.05\), applying a cosine annealing decay schedule over 5 epochs with a batch size of \(4\).

8.0.0.2 Computational Graph.

To better accommodate the varying degrees of discrepancy across activation at different layers, our proposed InfoQuant method employs PSOT to optimize the distribution of each layer’s activation individually. This per-layer optimization yields improved quantization performance. However, to maintain computational consistency in the presence of residual connections, an additional matrix multiplication is required specifically at these connection points, as illustrated in Figure 6. To mitigate the overhead introduced by this operation, we adopt block-diagonal orthogonal matrices as the rotation matrices, which significantly reduce inference cost and enhance the efficiency of the PSOT process. For fair comparison, the InfoQuant* employs a global rotation matrix, as shown in Figure 5. This approach eliminates the need for additional computation at residual connections and results in a computation graph equivalent to that of SpinQuant [3].

8.0.0.3 Runtime and Memory Overhead.

Table 3 presents the optimization runtime and memory consumption for a single Transformer layer of the LLaMA-2 models (7B, 13B, and 70B), measured on an NVIDIA RTX 4090 GPU. Remarkably, our method enables quantization of the 70B model using just 24GB of GPU memory, demonstrating compatibility with consumer-grade hardware. Note that GPUs with larger memory capacities can reduce optimization time by enabling greater parallelism, potentially achieving multi-fold speedups.

9 More Ablations↩︎

9.0.0.1 Influence of Weighting Factor \(\gamma\).

The weighting factor \(\gamma\) controls how strongly ASOT emphasizes outlier tokens in Eq. 5 . Table 4 shows that a moderate emphasis works best: \(\gamma=30\) achieves the lowest perplexity, while both weak emphasis (\(\gamma=1\)) and exclusive focus on outlier tokens (\(\gamma=+\infty\)) are worse. This trend suggests that outlier tokens provide high-value learning signals for peak suppression, but normal tokens are still necessary to preserve the overall activation distribution. In other words, ASOT should rebalance the optimization objective rather than collapse it into an outlier-only objective.

9.0.0.2 Influence of Temperature \(T\) in PSOT.

The temperature \(T\) controls how selectively PSOT suppresses high-magnitude activation coordinates. As shown in Table 5, \(T=2\) achieves the best perplexity, while both sharper weighting (\(T=0.3\)) and flatter weighting (\(T=8\)) degrade performance. This pattern reveals a useful design trade-off. If \(T\) is too small, the objective concentrates on only a few extreme coordinates and may overfit the calibration activations. If \(T\) is too large, the softmax weights become nearly uniform and the objective loses its ability to target peaks. A moderate temperature therefore provides the best balance between peak suppression and distribution-level stability.

9.0.0.3 Block Sizes of Block-diagonal Orthogonal Matrices.

To enhance computational and memory efficiency during activation rotation and inference, we adopt block-diagonal orthogonal matrices. However, this structure inherently limits interaction to within individual blocks, thereby restricting global information aggregation. As shown in Table 6, increasing the number of blocks, which corresponds to a reduction in block size, results in a decline in quantization performance. This observation aligns with findings from [36], which attributes the degradation to inter-block shifts in activation means that hinder quantization efficiency. We further hypothesize that full orthogonal matrices enable more effective redistribution of activation values due to the cumulative contribution of their unit vectors across higher dimensions. In contrast, block-diagonal matrices reduce this capability, thereby diminishing the range compression essential for effective quantization. To balance these competing considerations, we select an intermediate block size, as detailed in the Implementation details.

9.0.0.4 Robustness of Initialization \(\mathbf{A}\).

To investigate the robustness of the proposed PSOT algorithm with respect to orthogonal initialization strategies, we conduct an ablation study comparing Hadamard and randomly generated orthogonal matrices. For the latter, we start from uniformly sampled random matrices and apply QR decomposition to ensure orthogonality. As shown in Table 7, we evaluate both initialization methods on LLaMA-2 13B and LLaMA-3 8B models. While previous work [2] reported a notable performance gap in favor of Hadamard-based initialization, our findings demonstrate that, after optimization via PSOT, both initialization schemes yield similarly stable and effective results. This suggests that PSOT is robust to the choice of orthogonal basis at initialization.

9.0.0.5 Clipping Ratio Ablation.

The orthogonal matrix obtained by PSOT reduces quantization error through rotation, but this benefit comes at the cost of increased sensitivity to clipping thresholds. As shown in Table 8. Small changes in the clipping thresholds (\(\alpha,\beta\)) can lead to significant variations in the model’s performance, highlighting a trade-off between reduced quantization error and the difficulty of fine-tuning the clipping parameters for optimal results.

0.5pt

10 More Results↩︎

10.0.0.1 Results for Qwen family

For a more comprehensive evaluation, we further test our method on the Qwen-2.5 models in Table [QWENResults]. Experimental results show that under 4-4-4 quantization, our method maintains superior performance on both the 14B and 32B models, highlighting its robustness and scalability.

10.0.0.2 Full Results

In Table 9, we report the complete InfoQuant results for the experimental section. We compare WikiText2 perplexity and accuracy on nine zero-shot tasks using the lm-evaluation-harness (version 0.4.7)[26], including BoolQ[27], HellaSwag [28], LAMBADA (OpenAI)[29], OpenBookQA (OBQA)[30], PIQA [31], SIQA [32], WinoGrande [33], ARC-Easy, and ARC-Challenge [34].

11 Visualization results↩︎

Figure 7 and Figure 8 shows the activation distribution of different layers in LLaMA-2-7B and LLaMA-3-8B.

References↩︎

---

1. Code is available at: github.com/LLIKKE/InfoQuant.↩︎

2. Proofs and the Gaussian/Laplace closed forms can be found in Appendix 7.↩︎

3. Appendix 7 gives the corresponding formal analysis.↩︎