1 Introduction↩︎

Few-bit quantization relies heavily on uniform integer grids uniquely defined by an alphabet, a scale factor, and (optionally) a zero point. A small but often overlooked detail is that the standard signed \(q\)-bit alphabet \(\mathcal{A}_q = \{-2^{q-1}, \ldots, 2^{q-1}-1\}\) is not symmetric: it contains one more negative value than positive. Yet, by convention, the standard symmetric quantizer takes the scale \(s\) to be positive [1]–[3], assigning the extra representable value to the negative tail of the data and thereby possibly clipping large positive values. We show in this work that this can become a material source of quantization error at few-bit precision.

Asymmetric quantization introduces a zero point to align the grid with the data range, but this flexibility is well-known to carry an inference cost [2], [4], [5]. Kernels must account for the offset, and the corresponding metadata must be stored and loaded. For example, on Llama3 8B, we find that a symmetric 4-bit format uses 9% less memory and provides \(2.21\times\) higher prefill and \(2.03\times\) higher decode throughput than its asymmetric counterpart as measured on an AMD EPYC “Turin” CPU (Table 1). This motivates a narrower question: can we recover the useful endpoint alignment of an asymmetric grid while keeping the inference path operationally symmetric?

We answer this question with signed symmetric quantization, where we choose the sign of the scale to align the dominant outlier in the data with the extra negative representable value in the signed integer alphabet. The resulting sign rule is closed-form, data-free, and metadata-free when scales are already stored as signed real values, as in llama.cpp. Our analyses show that signed symmetric quantization recovers a restricted asymmetric alignment while preserving the symmetric deployment path. In particular, on Llama3 8B at 2 bits, it reduces WikiText2 perplexity from \(103.1\) to \(17.8\) and improves few-shot accuracy recovery by \(7.9\) percentage points (Table 3), while retaining \(1.24\times\) prefill, \(1.08\times\) decode, and \(14\%\) memory advantages over asymmetric quantization (Table 1).

1.0.0.1 Contributions.

Our contributions are four-fold. First, we define signed symmetric quantization and introduce a principled, lightweight sign selection rule that preserves the dominant outlier (Section 3). Second, we prove a worst-case \(\ell_2\) error bound that yields conditional bound-optimality of our proposed sign selection rule (Theorem [thm:worst-case-bound] and Corollary [cor:optimal-sign]). Third, we prove that a scale sign flip is analytically equivalent to a unit zero-point shift on the same signed alphabet (Theorem [thm:signed-asymmetric]). Fourth, we evaluate models in the Qwen3, Qwen3.5, and Llama3 families, showing that signed symmetric quantization improves over the conventional strictly positive symmetric grid, especially at low bit widths (Tables 2, 3, and 4), while preserving the symmetric deployment profile (Table 1).

1.0.0.2 Notation.

Let \(w \in \mathbb{R}^{d}\) denote a \(d\)-dimensional weight vector, with elements \(w_j \in \mathbb{R}\) for \(j \in [d]\), where \([d] = \{1, \ldots, d \}\). A quantizer \(\mathcal{Q}\) maps real values to a discrete \(q\)-bit alphabet \(\mathcal{A}_q \subset s \mathbb{Z}\) with a step size (or resolution) of \(s \in \mathbb{R}\), where \(\vert \mathcal{A}_q \vert = 2^q\). In particular, the standard integer quantizer is \[\mathcal{Q}(w) = s \cdot \left( \text{clip}\Big(\left\lfloor \frac{w}{s} + z \right\rceil; \mathcal{A}_q \Big) - z \right), \label{eq:quantizer}\tag{1}\] where \(s\) is the non-zero scaling factor, \(z \in \mathbb{R}\) is the zero point, \(\lfloor \cdot \rceil\) is the round-to-nearest (RTN) operator, and \(\text{clip}(x;\mathcal{A}_q) := \max ( \min(x, \max \mathcal{A}_q), \min \mathcal{A}_q)\). Different integer quantization schemes (e.g., asymmetric or symmetric) are distinguished by how \(s, z,\) and \(\mathcal{A}_q\) are chosen, and we refer to unique selections of \(s, z,\) and \(\mathcal{A}_q\) as a grid.

2 Background and Related Work↩︎

Quantization seeks to map a full-precision tensor onto a discrete, low-precision grid while minimizing reconstruction error, often in the form \(\|w - \mathcal{Q}(\tilde{w})\|_2^2\). Prior work attacks this objective along two complementary axes: (i) for a fixed grid, search for the assignment \(\tilde{w}\) that minimizes the error; (ii) for a fixed \(\tilde{w}\), search for the grid that minimizes the error. The two are not mutually exclusive, as we demonstrate empirically in Section 5.

2.0.0.1 Optimizing \(\tilde{w}\) for a fixed grid.

Frantar et al. [6] introduce GPTQ, a greedy sequential rounding scheme that uses approximate second-order information from calibration data to update weights on a fixed grid, correcting for local quantization error. More recently, Zhang et al. [7] introduce Qronos, which generalizes this idea to also account for quantization error propagated from previously quantized layers. A complementary line of work transforms the weight tensor before rounding: Xiao et al. [8] rescale channels to migrate quantization difficulty from activations to weights, Ashkboos et al. [9] apply Hadamard rotations to suppress outliers, and Liu et al. [10] extend this with learned rotations. This family of algorithms is not the focus of our work, but we show that signed symmetric quantization composes constructively with it (Section 5).

2.0.0.2 Optimizing the grid for a fixed \(\tilde{w}\).

Esser et al. [11] learn scaling factors via gradient descent, extended by Bhalgat et al. [12] to learnable zero points. In LLM quantization, however, the scale is typically derived from data range using heuristics [2], [5], [7], or by greedily minimizing local quantization error [6]. Recent works instead optimize the scale to minimize local output error without explicitly constraining its sign, yet in practice restrict evaluation to positive solutions [13], [14]. In any case, the scale is assumed (implicitly or explicitly) to be strictly positive; this convention is widespread but not a theoretical requirement. While some implementations differ on whether to use the full representation range of the format [3] or restrict to a symmetric subset (i.e., narrow range) [1], [3], this choice determines whether the extra negative level is used, not which tail receives it. To our knowledge, no prior work treats the sign of the scale factor as an explicit degree of freedom. Our contribution therefore sits in this second strategy: we provide the first theoretical analysis and systematic evaluation of signed scale factors for few-bit integer quantization.

3 Signed Symmetric Quantization↩︎

Signed symmetric quantization exploits the native asymmetry of the signed \(q\)-bit alphabet: it has one more negative value than positive. Standard symmetric quantization fixes the scale to be positive by convention, which always assigns this extra representable value to the negative tail, possibly clipping large positive values. This is often negligible at higher precision, but this clipping can dominate quantization error as the bit width \(q\) is reduced, as later made explicit in Theorem [thm:worst-case-bound].

Asymmetric quantization shifts the grid with a zero point, improving range alignment but changing the deployed format: kernels must handle the offset and store its metadata. We instead ask whether the signed alphabet’s native asymmetry can be matched to the data while keeping \(z = 0\). Signed symmetric quantization does exactly this, as we detail below.

3.0.0.1 Asymmetric quantizer.

Asymmetric quantization allows both the scale and zero point to be free parameters (i.e., \(s,z \in \mathbb{R}\)). A common choice is the min-max grid, \[s = \frac{\max(w) - \min(w)}{2^q - 1}, \qquad z = -\frac{\min(w)}{s}, \label{eq:minmax}\tag{2}\] on the unsigned alphabet \(\mathcal{A}_q := \{0, \ldots, 2^q - 1\}\).

3.0.0.2 Symmetric quantizer.

Symmetric quantization fixes the zero point to zero (i.e., \(z = 0\)) and commonly defines the scale as the maximum absolute value (or absmax), \[s = \frac{\|w\|_\infty}{2^{q-1}}, \label{eq:absmax}\tag{3}\] on the signed alphabet \(\mathcal{A}_q := \{-2^{q-1}, \ldots, 2^{q-1} - 1\}\). We refer to this as the absmax grid.

These grid choices imply different inference costs [2], [4], [15]. We can observe this distinction directly in GGUF models executed with llama.cpp [1] on an AMD EPYC “Turin” CPU. At 4-bit precision, Q4_1 defines an asymmetric quantizer per group of 32 elements, storing a 16-bit scale, a 16-bit zero point, and 32 4-bit weights per group; Q4_0 is the corresponding symmetric format, storing only a scale per group of 32 4-bit weights. We implement analogous 2-bit formats, Q2_1 and Q2_0, to further characterize the low-bit regime; these are experimental formats introduced for this study, with implementation details and further discussion deferred to Appendix 8. Table 1 shows that the symmetric formats use significantly less memory on larger models (up to \(9\%\) at Q4 and up to \(15\%\) at Q2) and consistently improve prefill throughput (\(1.10\)–\(2.45\times\) at Q4 and \(1.02\)–\(1.39\times\) at Q2), with decode throughput mostly following the same trend. This is the price of the flexibility of the asymmetric format, and it motivates a quantizer that retains the runtime profile of a symmetric format while (partially) recovering the endpoint alignment of the asymmetric format.

3.0.0.3 Signed symmetric quantizer.

We define signed symmetric quantization as the family of uniform integer quantizers on the signed alphabet \(\mathcal{A}_q = \{-2^{q-1}, \ldots, 2^{q-1}-1\}\) with zero point \(z = 0\) and non-zero scale (i.e., \(s \neq 0\)). Given scale magnitude \(\alpha > 0\) and sign \(\gamma \in \{-1, +1\}\), let \(s := \gamma\alpha\) and \[\mathcal{Q}_\gamma(w) := \mathcal{Q}(w;\, \gamma\alpha,\, 0) = \mathcal{Q}(w;\, s,\, 0). \label{eq:signed95sym95quant}\tag{4}\] Note that the standard symmetric quantizer is a degenerate case of the signed symmetric quantizer with \(\gamma = +1\). Furthermore, allowing \(\gamma = -1\) does not change the grid resolution; it only determines which tail receives the extra representable endpoint in the signed alphabet.

3.0.0.4 The signed absmax grid.

Within this family, we introduce the signed absmax grid. Let \(M := \|w\|_\infty > 0\) and \(\alpha := M / 2^{q-1}\). Choose \[\gamma^\star = -\operatorname{sign}(w_{i^\star}), \qquad i^\star \in \mathop{\mathrm{arg\,max}}_{i \in [d]} |w_i|, \label{eq:sign95rule}\tag{5}\] where \(i^\star\) indexes the largest-magnitude coordinate, with deterministic tie-breaking. Intuitively, this sign choice maps the largest representable value in \(\mathcal{A}_q\) to the dominant outlier in \(w\). In particular, \[\mathcal{Q}_{\gamma^\star}(w_{i^\star}) \;=\; \gamma^\star\alpha \cdot (-2^{q-1}) \;=\; -\gamma^\star M \;=\; \operatorname{sign}(w_{i^\star})\, M \;=\; w_{i^\star}.\] Hence, the dominant outlier is represented exactly. The opposite sign choice would instead place the extra endpoint on the other tail, potentially clipping the largest-magnitude coordinate. Indeed, in Section 4, we prove that our sign choice conditionally minimizes worst-case quantization error (Corollary [cor:optimal-sign]); a condition that holds for 88–99% of evaluated LLM weight groups (Remark 1).

This is the key distinction from min-max asymmetric quantization. Signed absmax does not attempt to cover the entire interval \([\min w, \max w]\). Instead, it uses the asymmetry already present in the signed alphabet to place the exact endpoint on the tail containing the dominant outlier, while retaining the runtime profile measured above for symmetric formats. Section 4 analyzes this choice, and Section 5 evaluates how the proposed endpoint alignment translates into model-level accuracy.

4 Theoretical Analysis↩︎

In Section 3, we defined the signed symmetric quantization family and argued informally that the signed absmax grid reduces clipping error by aligning the representable endpoint with the dominant outlier. We now formalize this intuition.

We first define the clipped set (Definition 1), then decompose the \(\ell_2\) error to isolate the two roles of the scale: (1) its magnitude governs grid resolution and rounding error, while (2) its sign governs grid polarity and clipping error (Lemma [lem:error-separability]). We bound the \(\ell_2\) error in terms of the clipped set (Theorem [thm:worst-case-bound]), then prove that the signed absmax grid minimizes this bound under a mild cardinality condition (Corollary [cor:optimal-sign]). We empirically verify this condition on real LLM weights and show that reducing the clipped set cardinality is an effective surrogate for reducing \(\ell_2\) quantization error (Remark 1). We close by showing that the signed symmetric quantization family is a non-trivial special case of the asymmetric quantization family (Theorem [thm:signed-asymmetric]).

Throughout our analysis, we assume weight vector \(w \in \mathbb{R}^d\) is quantized to the \(q\)-bit signed integer alphabet \(\mathcal{A}_q := \{-2^{q-1}, \ldots, 2^{q-1}-1\}\) under round-to-nearest with step size \(\alpha := M / 2^{q-1}\), where \(M := \|w\|_\infty > 0\). We also assume \(w_i/\alpha \notin \mathbb{Z} + 1/2\) for all \(i \in [d]\) (a measure-zero condition under any continuous weight distribution) so that \(\lfloor -v \rceil = - \lfloor v \rceil\) for \(v \in \mathbb{R}\). For brevity, all proofs are deferred to Appendix 7.

Lemma [lem:error-separability] separates the two roles of the scale: (1) its magnitude \(\alpha\) determines the grid resolution and rounding error \(E(w)\), and (2) its sign \(\gamma\) determines the grid polarity and clipping penalty \(\Delta_\gamma(w)\). The following theorem converts this exact expression to a deterministic worst-case bound.

Theorem [thm:worst-case-bound] turns the decomposition identified in Lemma [lem:error-separability] into a worst-case objective for sign selection. Given \(w\), Lemma [lem:error-separability] shows that the sign of the scale affects the \(\ell_2\) error only through clipping. Theorem [thm:worst-case-bound] then upper bounds the total squared error by a quantity whose only \(\gamma\)-dependent term is the clipped fraction \(\rho_\gamma = |C_\gamma|/d\). Therefore, under the deterministic worst-case bound, sign selection reduces to minimizing the size of the clipped set.

Importantly, as outliers are well-known to dominate few-bit quantization error [9], [10], worst-case error bounds are effective optimization targets for post-training quantization [16]. The following corollary and remark build on this intuition to justify the signed absmax grid (Equation 5 ), both analytically and empirically: Corollary [cor:optimal-sign] characterizes when the proposed sign choice minimizes the worst-case bound, and Remark 1 shows that this condition holds for 88–99% of evaluated LLM weight groups, with a cardinality margin that is highly predictive of realized \(\ell_2\) gains.

Corollary [cor:optimal-sign] links the practical sign rule from Section 3 to the worst-case analysis in Theorem [thm:worst-case-bound]: whenever \(|C_{\gamma^\star}| \le |C_{-\gamma^\star}|\), the choice \(\gamma^\star\) minimizes Equation 8 . This yields a directly interpretable optimization criterion: aligning the extra representable endpoint with the dominant outlier should not increase the number of clipped coordinates. It also establishes the size of the clipped set \(|C_\gamma|\) as a surrogate for the exact sign-dependent clipping penalty \(\Delta_\gamma(w)\) in Lemma [lem:error-separability]. Both relationships are directly measurable on LLM weight groups, allowing us to evaluate not only how often the condition holds, but also how well the cardinality margin predicts realized \(\ell_2\) error reduction.

We note that our choice \(\gamma^\star = -\operatorname{sign}(w_{i^\star})\) also carries two structural properties that hold unconditionally, independent of whether the cardinality condition is met. First, as detailed in Section 3, it represents the dominant outlier exactly: \(\mathcal{Q}_{\gamma^\star}(w_{i^\star}) = w_{i^\star}\), since \(|w_{i^\star}| = M\) coincides with the representable endpoint placed on its side. Second, it confines clipping to the opposite tail: \(C_{\gamma^\star} \subseteq \{i \in [d] : \gamma^\star w_i > 0\}\). These properties help explain why the sign rule remains effective in practice even on the small fraction of groups where the condition fails (Figure 1): preserving the dominant outlier exactly and confining clipping to the opposite tail make \(\gamma^\star\) a natural choice when a clear dominant tail is present. Tables 3 and 4 corroborate this at the model level: the largest gains appear at 2 bits, precisely where the cardinality condition holds least often (Figure 1).

Interestingly, asymmetric quantization pursues the same goal of aligning the grid endpoint with the dominant outlier, but does so by introducing a zero point that carries a non-trivial inference cost [2], [4], [15] (Table 1). The next theorem shows that the signed symmetric quantizer is identical to a unit zero point shift on the same signed integer alphabet, showing that signed symmetric quantization is analytically a non-trivial instance of the family of asymmetric quantizers.

Theorem [thm:signed-asymmetric] shows that, at a shared step size \(\alpha\), negating the scale under \(z = 0\) is identical to setting \(z = -1\) with a strictly positive scale on the same signed integer alphabet. Signed symmetric quantization therefore admits two equivalent views: (1) operationally a symmetric quantizer (\(z = 0\)) that incurs no zero point overhead at inference, and (2) analytically an asymmetric quantizer with a unit zero point shift (\(z = -1\)) that absorbs the shift into the sign of the scale.

It is worth noting the relationship to the standard min-max asymmetric grid (Equation 2 ), which is not addressed by Theorem [thm:signed-asymmetric]: the min-max grid uses an unsigned alphabet and a scale derived from the full data range, i.e., \(\max w - \min w\), rather than from \(\|w\|_\infty\). Both schemes preserve the dominant outlier exactly, but the min-max grid additionally preserves the opposite tail by covering \([\min w, \max w]\) exactly, eliminating clipping error and leaving rounding as the only source of error (Lemma [lem:error-separability]). Furthermore, its step size depends on the full data range, so its relationship with \(\alpha\) varies with the distribution of \(w\). An analytical comparison between the signed symmetric and min-max asymmetric grids is therefore non-trivial. However, we compare them empirically in Section 5.

5 Experimental Results↩︎

Sections 3 and 4 show that the sign of the scale is a useful degree of freedom: by choosing the sign appropriately, the dominant outlier in the data can be mapped to the extra negative value in the signed integer alphabet rather than being clipped. We now evaluate whether this local choice of grid polarity yields benefits for end-to-end model accuracy.

We organize our evaluation in two parts. First, we isolate the sign rule by removing post-training quantization machinery (no rotations, error correction, or activation quantization) and show that the signed absmax grid improves over the conventional strictly positive absmax grid (Section 4). Second, we use Llama3 8B as a representative case study to show that this improvement survives composition with state-of-the-art transformations and rounding methods.

Models and datasets. Our experiments span three model families: Qwen3 [17], Qwen3.5 [18], and Llama3 [19], all sourced as instruction fine-tuned checkpoints from Huggingface [20] and used without modification. For calibration, we sample 128 random sequences of 2048 tokens from the WikiText2 [21] training split and report perplexity on its test split. To evaluate downstream reasoning, we report zero-shot accuracy via LightEval [22] on ARC (Challenge and Easy) [23], HellaSwag [24], PIQA [25], and WinoGrande [26]. We further report few-shot accuracy on MMLU-Redux (MMLU-R) [27] and GSM8K [28]. We quantize all models using the Brevitas quantization library [3].

5.0.0.1 Sign rule in isolation.

We first isolate the effect of the sign rule by direct casting all linear projection weights with group size 32, matching the llama.cpp block convention. This experiment uses no rotations, no error correction methods (e.g., Qronos), no activation quantization, and no calibration or optimization. Thus, the only difference between the absmax and signed rows in Table 2 is whether the scale is constrained to be strictly positive or allowed to choose its sign per group.

The results show that our sign rule (Equation 5 ) gives a consistent improvement over the standard absmax grid: signed absmax improves WikiText2 perplexity in 23 of the 24 configurations in Table 2, with gains concentrated at lower precision. This is consistent with the \(2^{-2q}\) dependence in the error bound of Theorem [thm:worst-case-bound], Equation 8 . In particular, the largest reductions occur on Qwen3-1.7B (\(81.6\) to \(25.9\)) and Llama3-1B (\(32.3\) to \(21.9\)) when quantizing weights to 3 bits (\(q = 3\)).

The downstream zero-shot accuracy results, averaged over ARC (Challenge and Easy), HellaSwag, Winogrande, and PIQA, follow the same trend: signed absmax improves average zero-shot accuracy by \(0.24\) points at INT4 and \(1.11\) points at INT3. Across all 24 configurations, signed absmax improves 18, ties 2, and trails in 4; every regression is at most \(0.2\) points. These results support the central claim from Section 4: the sign of the scale is a small local choice but, at few-bit precision, it has measurable model-level consequences.

5.0.0.2 Composition with PTQ methods.

Direct casting isolates the sign rule, but practical quantization pipelines typically include transformations, error correction, and activation quantization. For example, in llama.cpp, few-bit weight formats are commonly paired with dynamic Q8_0 activation quantization at inference. We therefore evaluate Llama3 8B as a case study under a practical PTQ pipeline. We first insert normalized Hadamard rotations into rotation-invariant regions within the model [9] (Figure 2). Weight grids are then calculated on the rotated weights, while activation quantization grids are computed dynamically from the rotated activations under Q8_0 quantization. Qronos is then applied as the adaptive rounding step to reduce the mismatch between the quantized model and its full-precision counterpart [7]. We defer further algorithmic details to Appendix 9.

Table 3 presents results under three few-bit formats in extended GGUF notation: Q4_0 denotes the standard symmetric format, using the absmax grid from Equation 3 , Q4_0s denotes our signed symmetric variant, using the signed absmax grid from Equation 5 , and Q4_1 denotes the asymmetric format, using the min-max grid from Equation 2 . We use analogous notation for Q3 and Q2.²

At W4, all three formats are close in accuracy. Q4_1 and Q4_0s both achieve \(6.6\) WikiText2 perplexity, while Q4_0 gives \(6.7\). Downstream recovery is similarly saturated: \(98.9\%\) for Q4_1, \(98.6\%\) for Q4_0s, and \(98.7\%\) for Q4_0. In this regime, the choice between symmetric and asymmetric formats is dominated less by accuracy than by deployment cost. Since Q4_0s uses the same runtime path as Q4_0, it inherits the memory and throughput advantages of the symmetric format measured in Table 1. On Llama3 8B at W4, this corresponds to a \(2.21\times\) prefill and \(2.03\times\) decode throughput advantage over Q4_1, while Q4_0s trails Q4_1 by only \(0.3\) recovery points.

At lower bit widths, the sign choice remains valuable but does not fully eliminate the advantage of an asymmetric zero point. At W3, signed symmetric improves the conventional symmetric baseline from \(7.7\) to \(7.2\) perplexity and from \(89.0\%\) to \(94.6\%\) recovery, closing the recovery gap to Q3_1 (which achieves \(7.0\) perplexity and \(95.9\%\) recovery) from \(6.9\) to \(1.3\) percentage points. At W2, the effect is even more pronounced in perplexity: Q2_0 collapses to \(103.1\), while Q2_0s reduces this to \(17.8\). The asymmetric Q2_1 format remains stronger at \(12.0\) perplexity, but signed symmetric closes the few-shot accuracy recovery gap from \(15.6\) to \(7.7\) percentage points.

5.0.0.3 Composition with learned rotations and grid parameters.

We next ask whether the same degree of freedom remains useful when the quantization pipeline is allowed to adapt from data. To this end, we jointly learn rotations and grid parameters on FineWeb [29] at the same insertion points as the fixed pipeline (Figure 2), using Table 3 as an initialization, then freeze them before applying Qronos [7] with dynamic Q8_0 activation quantization. Rotations are optimized on the Stiefel manifold using Cayley SGD as in SpinQuant [10]. For the grid parameters, Q4_1 learns the scale and zero-point offset jointly as in LSQ+ [12], while Q4_0 and Q4_0s follow LSQ [11] with the scale constrained to be strictly positive (\(s > 0\)) and non-zero (\(s \neq 0\)), respectively. We use analogous parameterizations for Q3 and Q2, with full optimization details deferred to Appendix 9.

Table 4 shows that signed symmetric formats continue to improve over their conventional symmetric counterparts after joint rotation and grid learning. At W4, Q4_0s slightly improves over Q4_0 in both WikiText2 perplexity, \(6.7\) to \(6.6\), and recovery, \(98.7\%\) to \(98.8\%\). The effect is larger at lower precision: Q3_0s improves over Q3_0 from \(7.3\) to \(7.1\) perplexity and from \(90.9\%\) to \(95.3\%\) recovery; Q2_0s reduces Q2_0 perplexity from \(103.8\) to \(12.9\) and improves recovery from \(42.7\%\) to \(56.1\%\). The comparison to the asymmetric format is mixed but informative: Q4_1 and Q3_1 maintain superior recovery at W4 and W3, while Q2_0s slightly exceeds Q2_1 in aggregate recovery at W2 (\(56.1\%\) versus \(54.3\%\)) despite worse perplexity (\(12.9\) versus \(12.2\)). This perplexity-versus-recovery inversion is consistent with prior observations that the two metrics can decouple at low precision [7], [16], and is analogous to overfitting: Zhang et al. [30] (Remark 3.8) decompose generalization error into a calibration reconstruction term and a parameter-proximity term \(|w - \mathcal{Q}(w)|\), which can be at odds.

Taken together, Tables 2–4 support a consistent interpretation. The sign of the scale is a local degree of freedom with measurable model-level effects when clipping is material. Direct casting isolates this effect: signed absmax consistently improves over the conventional strictly positive absmax grid. In practical PTQ pipelines, signed symmetric quantization decouples the accuracy and systems axes: it deploys through exactly the symmetric runtime path, while its accuracy moves toward that of the asymmetric format. On Llama3 8B, it reaches the asymmetric accuracy regime at 4 bits, closes most of the recovery gap at 3 bits, and substantially improves over Q2_0 at 2 bits, with jointly learned grids and rotations even exceeding Q2_1 in aggregate few-shot accuracy recovery despite worse perplexity. Thus, the sign remains a useful degree of freedom even when rotations and quantizer parameters are learned from data, at no extra inference cost relative to the conventional symmetric format.

6 Discussion and Conclusion↩︎

Standard symmetric quantization constrains the scale factor to be positive by convention. We showed that allowing signed scales exposes a zero-overhead degree of freedom. The signed absmax grid uses this freedom to place the extra signed-integer endpoint on the dominant-outlier tail. Our analysis separates the role of scale magnitude, which controls rounding error, from the role of scale sign, which controls clipping. Theoretically, this separation yields a worst-case \(\ell_2\) bound under which signed absmax is conditionally bound-optimal. Empirically, this degree of freedom consistently improves conventional symmetric quantization at few bits and composes with rotations, Qronos, learned grid parameters, and dynamic activation quantization.

6.0.0.1 Limitations and future work.

Our analysis focuses mainly on data-free, per-group weight quantization. This is intentionally narrow: it isolates the degree of freedom and yields a closed-form rule that already captures much of the benefit observed in our PTQ pipelines. Learned rotations and grid parameters can provide additional gains, suggesting that a fully data-aware treatment of signed scales is a promising extension rather than a prerequisite for the method. More broadly, future work should evaluate signed scales across additional model families, operator types such as convolutions, quantization granularities, and deployment backends. Our results nevertheless suggest that the sign of the scale should be treated as a quantization parameter rather than fixed by convention.

7 Proofs↩︎

Throughout this appendix, we adopt the assumptions stated in Section 4: the signed integer alphabet \(\mathcal{A}_q := \{-2^{q-1}, \ldots, 2^{q-1}-1\}\), and the no-midpoint condition \(w_i/\alpha \notin \mathbb{Z} + \tfrac{1}{2}\) for all \(i \in [d]\) (a measure-zero condition under any continuous weight distribution), under which rounding to nearest is unique and satisfies \(\lfloor -x\rceil = -\lfloor x\rceil\) whenever \(x \notin \mathbb{Z} + \tfrac{1}{2}\).

7.1 Proof of Lemma [lem:error-separability]↩︎

7.2 Proof of Theorem [thm:worst-case-bound]↩︎

7.3 Proof of Corollary [cor:optimal-sign]↩︎

7.4 Proof of Theorem [thm:signed-asymmetric]↩︎

8 Implementation Details for llama.cpp↩︎

This appendix expands on the inference measurements in Table 1. The goal is to quantify the systems cost of the deployed quantization format, independent of our sign selection rule (Section 3). In particular, the measurements compare symmetric and asymmetric GGUF formats with matched group size and bit width. Since signed symmetric quantization uses the same deployed format as the standard symmetric quantization, these measurements apply to both; the sign rule changes scale selection offline, not the runtime kernel path.

We consider two standard 4-bit GGUF formats: Q4_1 (asymmetric) and Q4_0 (symmetric). We also contribute two new 2-bit GGUF formats, Q2_1 (asymmetric) and Q2_0 (symmetric), enabling direct comparison of signed symmetric and asymmetric quantization at INT2 precision. We benchmark CPU inference throughput using the llama.cpp framework [1] on an AMD EPYC “Turin” 128-core processor with the ZenDNN(L) backend v1.0.0.

8.1 Formats and block layout↩︎

Consistent with the Q4_0 and Q4_1 standard, all formats use a block size of 32 weights. For Q2, each block packs 32 two-bit indices into 8 bytes, with four indices per byte. For Q4, each block packs 32 four-bit indices into 16 bytes, with two indices per byte.

Q4_0 and Q2_0 store a single scale \(s = \gamma\alpha\), which is strictly positive by convention but may be negative under our implementation. Q4_1 and Q2_1 also store a scale \(s\) and minimum \(m\).

8.2 Measurement setup↩︎

8.2.0.1 Hardware.

CPU experiments run on an AMD EPYC 9755 (Zen 5, “Turin”) system with 128 physical cores per socket (256 logical), two NUMA nodes, and 1.5 TB DDR5. The CPU supports AVX-512 (F/BW/DQ/VL), AVX-512VNNI, AVXVNNIINT8, AVX2, and FMA.

8.2.0.2 Software and builds.

All experiments use llama.cpp at commit b1be68e8 (master, ggml-org/llama.cpp), compiled with GCC 14.3.0 and CMake 4.2.1 on Ubuntu 22.04 (Linux 5.15) in Release mode with GGML_NATIVE=ON and GGML_OPENMP=ON, enabling -march=native and all ISA extensions supported by the target CPU. We evaluate two configurations from this same commit: (i) an unmodified upstream baseline build, supporting BF16, Q4_0, and Q4_1 with no custom kernels or code modifications, and (ii) a modified build that adds Q2_0/Q2_1 quantization support and AVX-512–optimized dot product kernels for all quantized types. In the baseline build, Q4_0/Q4_1 dot product kernels do not have explicit AVX-512 paths and execute via 256-bit AVX2 VNNI instructions, while other operations (e.g., attention, RMSNorm, SiLU, softmax) use existing 512-bit AVX-512 kernels. Models are converted from HuggingFace safetensors to BF16 GGUF and then quantized to Q4_0, Q4_1, Q2_0, and Q2_1. However, we note that one could similarly export Brevitas-quantized models that are compatible with the llama.cpp runtime.

8.2.0.3 Benchmarking and metrics.

Throughput is measured using llama-bench. We evaluate a 512-token prefill followed by 512-token autoregressive decode, with prefill batch size 512 and 128 threads. All CPU runs are pinned to a single NUMA node using numactl --cpunodebind=0 --membind=0, mapping one thread per physical core and avoiding cross-NUMA traffic. We report tokens/s for: (i) prefill (pp512), a compute-bound phase dominated by matrix multiplications, and (ii) decode (tg512), a memory-bandwidth-bound phase. Each experiment performs one warmup iteration followed by 20 timed repetitions; reported results are the mean over repetitions. Across all reported configurations, over 90% exhibit a coefficient of variation (standard deviation divided by mean) below 6%.

8.3 Kernel-level cost gap↩︎

The throughput comparison is between deployed formats, not between sign selection policies. Unsigned symmetric and signed symmetric quantization both deploy as a symmetric format with \(z = 0\); the only difference is the sign assigned to each per-group scale. When scales are stored as signed real values, the sign introduces no additional metadata. Therefore, Table 1 characterizes the runtime profile inherited by signed symmetric quantization from the symmetric family.

8.3.0.1 Unpacking overhead.

After loading 8 bytes of packed 2-bit indices from a Q2 format, the kernel must expand them to 32 individual bytes in element order. This requires broadcasting the 8-byte value to a 256-bit register, extracting four shift-and groups, and interleaving the results with two unpacklo_epi8 and one unpacklo_epi16, approximately 10 SIMD instructions per block. By comparison, the 4-bit nibble unpack (bytes_from_nibbles_32) needs only a 128-bit load, one shift, and one mask: approximately 3 instructions. This \(3\times\) difference in unpack cost is a fixed overhead shared by both Q2_0 and Q2_1, and it is the primary reason the symmetric-over-asymmetric throughput advantage is smaller at 2 bits than at 4 bits (Table 1). On the AVX-512 path, a dual-block unpack primitive loads two blocks’ qs fields into a single 128-bit register and produces a 512-bit result (64 bytes), halving the number of unpack calls per iteration.

8.3.0.2 Zero-point cost in quantized GEMM.

The asymmetric format also adds work in the GEMM itself [4], [5]. Consider one output element \(Y_{ik} = \sum_j X_{ij} W_{kj}\) with \(W\) quantized in groups of size \(g\), so that within group \(b\), \(W_{kj} \approx s_{kb}(W_{q,kj} - z_{kb})\). Expanding, \[Y_{ik} \;\approx\; \sum_b s_{kb} \underbrace{\sum_{j \in b} X_{ij}\, W_{q,kj}}_{\text{integer GEMM}} \;-\; \sum_b s_{kb} z_{kb} \underbrace{\sum_{j \in b} X_{ij}}_{\text{row-sum correction}}.\] Both formats compute the integer GEMM term. The asymmetric format (\(z \ne 0\)) additionally accumulates a per-block row-sum of \(X\) scaled by \(sz\) per output, while the symmetric format (\(z = 0\)) skips this term entirely.

9 Additional Experiment Details↩︎

This appendix provides implementation details for the experiments in Section 5. We separate the quantization graph from the quantization pipeline. The graph specifies where rotations and quantizers are inserted into the model, while the pipeline specifies how grid parameters, rotations, and rounding decisions are chosen.

Across all experiments, we quantize linear projection weights using group size 32, matching the GGUF block convention used by llama.cpp. Unless otherwise stated (i.e., Table 2), we evaluate models with dynamic Q8_0 activation quantization. Weight grids are calculated (or learned) per group, while activation grids are computed dynamically from the corresponding (potentially transformed) activations per-token during each forward pass.

9.1 Direct casting↩︎

The direct-casting experiments in Table 2 isolate the effect of the scale-sign rule. We quantize all linear projection weights groupwise with group size 32 and do not use rotations, error correction, learned grid parameters, calibration data, or activation quantization. The conventional absmax baseline uses the signed integer alphabet with zero point \(z = 0\), strictly positive scale \(s = \alpha > 0\), and \(\alpha = \|w\|_\infty / 2^{q-1}\) per group. The signed absmax variant uses the same alphabet and zero point, but sets \(s = \gamma\alpha\), where \(\gamma \in \{+1, -1\}\) is chosen by Equation 5 . All values are rounded to nearest and clipped to the target integer alphabet.

9.2 Algorithm details↩︎

The experiments in Table 3 evaluate whether the signed grid remains useful when composed with a practical PTQ pipeline. We first insert normalized Hadamard rotations into rotation-invariant regions of the model only where rotations can be fully merged, as shown in Figure 2. Note that the llama.cpp runtime assumes a fixed contract on the model (i.e., no changes to the graph architecture and therefore no online rotations can be added), which is a notable departure from QuaRot [9] and SpinQuant [10]. These rotations are merged before grid calculation, so weight grids are calculated based on the rotated weights. Activations are quantized dynamically with Q8_0, with activation quantization grids computed from the rotated activations. In llama.cpp [1], Q8_0 on activations assumes a “narrow range” quantizer, where activations are scaled to be symmetric such that \(\mathcal{A}_q = \{-K, \ldots, K\}\) for \(K = 2^{q-1} - 1\). We emulate this quantizer within Brevitas during calibration.

After the rotations and grids are fixed, we apply Qronos as the adaptive rounding method [7]. Qronos is run with the same transformed graph and dynamic activation-quantized execution path used for evaluation, so that the rounding decisions account for the mismatch between the quantized model and its full-precision counterpart under Q8_0 activation quantization.

For the studied formats, Q4_0 uses the conventional symmetric grid with \(z = 0\) and \(s > 0\). Q4_0s uses the same signed alphabet and \(z = 0\), but allows \(s = \gamma\alpha\) with \(\gamma \in \{+1, -1\}\). Q4_1 uses the corresponding asymmetric min-max grid. We use analogous definitions for Q3 and Q2.

For Qronos, we use 128 calibration sequences of length 2048. The covariance matrix is damped as in Qronos, with damping parameter \(\lambda = 10^{-4}\,\sigma_1\) where \(\sigma_1\) is the largest singular value of the covariance matrix. We quantize weights in descending order of the diagonal of the covariance matrix [31]. All Qronos hyperparameters are held fixed across Q4_1, Q4_0s, and Q4_0 at the same bit width.

9.3 Jointly learned rotations and grid parameters↩︎

Table 4 evaluates a data-driven pipeline in which rotations and grid parameters are learned jointly before applying Qronos. The goal is to test whether the signed scale remains useful when the quantization pipeline is allowed to adapt from data, rather than relying only on the closed-form signed absmax rule.

We initialize rotations and grid parameters from the fixed-rotation pipeline in Table 3. Starting from this initialization, we jointly optimize the rotations and grid parameters for 100 steps on FineWeb [29], using batches of 8 sequences. Rotations are optimized on the Stiefel manifold using Cayley SGD, following SpinQuant [10]. Quantized weights and activations are simulated during optimization using the straight-through estimator [32], [33].

The three grid families differ only in their parameter constraints. For Q4_1, we learn the scale and zero-point offset jointly, as in LSQ+ [12]. We follow LSQ [11] when learning scales for Q4_0 and Q4_0s, constraining the scale to be strictly positive (\(s > 0\)) for Q4_0 and non-zero (\(s \neq 0\)) for Q4_0s, with the latter initialized from the signed absmax grid. We use analogous parameterizations for Q3 and Q2. After the 100-step joint optimization, the learned rotations are merged and the grid parameters are frozen, and Qronos is applied using the same adaptive rounding setup described in Appendix 9.2.

9.4 Implementation notes↩︎

All quantization experiments are implemented using Brevitas [3] and run on an AMD Instinct MI325X GPU with 256 GB of GPU memory. The learned grid experiments use the same quantization graph as the fixed-rotation experiments; only the procedure for choosing rotations and grid parameters changes. In particular, Q4_0s and Q4_0 share the same deployed symmetric inference path in llama.cpp. The signed variant changes the sign of the stored scale but does not introduce a zero point or any additional runtime arithmetic.

References↩︎

---

1. Equal contribution. Correspondence to: {ian.colbert, eashan.dash}@amd.com.↩︎

2. Q4_0 and Q4_1 are standard GGUF formats; the Q3 and Q2 variants are introduced for this study (Appendix 8).↩︎