1 Introduction↩︎

In recent years, large language models (LLMs) built upon the transformer architecture have achieved state-of-the-art results on a broad spectrum of natural language processing tasks [1]. Their strong empirical performance primarily stems from their enormous parameter counts, which frequently reach tens of billions. For instance, the open pretrained transformer (OPT) family [2] provides variants scaling up to 66 billion parameters, while the LLaMA series [3] includes even larger versions such as LLaMA3-70B. Despite their effectiveness, these models impose an immense memory footprint and computational burden. For example, running a 70B model in full precision requires over 130 GB of memory, which poses serious challenges for deployment on resource-constrained mobile devices and restricts the widespread adoption of LLMs.

To address these limitations, numerous compression techniques [4]–[6] have been developed for LLMs, such as weight quantization [7], [8], network pruning [9], [10], knowledge distillation [11], [12], and low-rank factorization [13], [14]. Among them, binarization is particularly notable since it reduces each parameter to approximately one bit, achieving an extremely high level of storage efficiency [15]. The post-training quantization (PTQ) framework enables the conversion of full-precision models into low-bit counterparts without incurring the substantial cost of retraining. As a PTQ approach, binarization enables approximately 1-bit weight quantization. Recent approaches [16]–[18] within this framework, including PB-LLM [16] and BiLLM [17], aim to mitigate performance degradation by identifying parameters that are most influential to model accuracy and applying refined optimization strategies or selective higher-precision representations, thus striking a more favorable trade-off between compression efficiency and task performance. Additionally, ARB-LLM [18] introduces an alternating refinement strategy for binarization that iteratively adjusts the binary parameters to substantially decrease the quantization error.

Despite these advances, it remains challenging to directly extend existing PTQ methods to large vision-language models (LVLMs). Unlike LLMs, LVLMs integrate visual encoders with large language backbones, requiring precise cross-modal alignment between visual and textual representations. Under such settings, model parameters from different layers and modalities contribute unevenly to downstream multimodal tasks. However, most PTQ approaches focus solely on minimizing the global weight quantization error after post-training, overlooking the heterogeneous importance of weights across layers and modalities, as shown in Fig. 1 (a)(b). As a result, task-irrelevant encoding parameters may be mistakenly preserved, while modality-critical weights may be insufficiently optimized, leading to severe performance degradation under weight binarization.

To address the aforementioned challenges, we propose a novel Significance-Aware Binarization for Large Vision-Language Models (SAB-LVLM), which represents an early attempt at achieving binarization for LVLMs. Firstly, we separately feed textual and visual calibration data into LVLMs. Based on the corresponding hidden states, we construct Hessian matrices to evaluate the significance of each full-precision weight. Secondly, we employ a spatial significance map to distinguish weights activated under a single modality from those activated by inputs from different modalities. Furthermore, we perform modality-guided significance integration using a modality integration score, yielding the final significance-aware binarization map. Unlike existing PTQ approaches [19], [20], as presented in Fig. 1 (a), we inject the significance-aware binarization map into the binarization objective as an error reweighting term. Finally, we perform binarization fitting through alternating significance-weighted update scheme. As depicted in Fig. 1 (c), extensive experiments on representative LVLM benchmarks demonstrate that the proposed SAB-LVLM consistently outperforms existing binary PTQ approaches under an approximately 1-bit constraint, achieving superior performance on downstream tasks while preserving compression efficiency.

• We propose a novel Significance-Aware Binarization for Large Vision-Language Models (SAB-LVLM). To the best of our knowledge, this work represents an early exploration of weight binarization for LVLMs.

• We design a spatial significance map to identify weights activated under a single modality from those activated across modalities, and devise a modality-guided integration to obtain the significance-aware binarization map.

• We develop an alternating significance-weighted update scheme to perform binarization fitting. Experiments on representative LVLM benchmarks verify that our SAB-LVLM outperforms existing binary PTQ approaches.

2 Related Work↩︎

2.1 Post-Training Quantization↩︎

Post-training quantization (PTQ) achieves significant reductions in model storage overhead, GPU memory consumption, and inference costs by performing low-bit discretization directly on a pre-trained model without training [7], [20], [21]. Compared to quantization-aware training (QAT) [22], [23], PTQ eliminates the need for costly retraining processes, making it particularly well-suited for deploying large-parameter language models and vision-language models [24]. The primary objective of PTQ is to minimize quantization error without retraining [8], [25], [26]. BRECQ [21] advances PTQ to lower bits through block-level reconstruction. ZeroQuant [27] proposes an efficient and economical PTQ workflow for large-scale Transformers. GPTQ [4] employs layer-wise quantization using approximate second-order information, becoming a representative method for LLMs. SmoothQuant [7] enhances 8-bit quantization stability by smoothly migrating activation outliers to the weight side. Recent works [5], [28]–[30] push PTQ toward ultra-low-bit quantization, particularly binary approaches. PB-LLM [16] showed that naive binarization severely harms LLMs and that a small set of salient weights should be preserved at higher precision. BiLLM [17] systematically introduces 1-bit PTQ into LLMs through structured significant weight selection and binary residual approximation. ARB-LLM [18] proposed alternating refined binarization to progressively update binarization parameters and reduce the distribution gap between binarized and full-precision weights. However, existing binary PTQ methods primarily focus on LLMs [15], while relevant research on LVLMs remains scarce.

2.2 Large Vision-Language Models↩︎

Large Vision-Language Models (LVLMs) extend LLMs [1], [31]–[35] to visual understanding by coupling a vision encoder with a large language model through lightweight alignment modules. Early efforts [36]–[38] mainly focused on bridging a pretrained vision encoder and a pretrained LLM through lightweight cross-modal connectors and visual instruction tuning, so that language models could accept visual inputs and perform general image-conditioned reasoning. The Qwen family [32], [39], as powerful open-source LLMs, have been widely adopted as the backbone of LVLMs. BLIP-2 [36] uses Q-Former to connect frozen visual encoders and frozen LLMs. InstructBLIP [37] builds upon this by introducing instruction-aware visual queries. LLaVA [38] advances this approach toward more general multimodal dialogue and reasoning through large-scale visual instruction fine-tuning. In recent years, modern LVLMs [39], [40] have acquired powerful perception and reasoning capabilities through extensive training, and have achieved success in numerous downstream tasks, such as visual question answering (VQA) [41], multimodal dialogue [42], visual grounding [43], and embodied interaction [44]. These downstream tasks require both perceptual and linguistic reasoning abilities. However, this multimodal capability also incurs substantial deployment costs. To address this limitation, a common approach involves compressing models through paradigms such as distillation [12], PTQ [45], and QAT [46]. Among these, PTQ typically offers lower training costs and better plug-and-play capabilities, yet it still struggles to be deployed on edge devices. Binarization advances PTQ to the approximate 1-bit precision, thereby further reducing storage requirements and computational overhead.

3 Methodology↩︎

3.1 Preliminary and Overview↩︎

Preliminary: Binarization [17], [18] in large vision-language models (LVLMs) compresses continuous weights into binary values (e.g., \(\pm 1\)), thereby reducing storage requirements and computational overhead. Given a full-precision weight \(\mathbf{W} \in \mathbb{R}^{n \times m}\) in the original LVLMs, we define its binarization objective as: \[\begin{align} \mathcal{L}(\mathbf{W}, \widehat{\mathbf{W}}) = \|\mathbf{W}-\widehat{\mathbf{W}}\|_F^2, \label{eq:binarization95objective} \end{align}\tag{1}\] where \(n\) and \(m\) denote the row and column dimensions of \(\mathbf{W}\). \(\widehat{\mathbf{W}} \in \mathbb{R}^{n \times m}\) represents the low-bit weight, and it is approximated as \(1\)-bit in this paper. To perform binarization, we follow [18] and represent \(\widehat{\mathbf{W}}\) using a \(T\)-order binary expansion strategy: \[\begin{align} \widehat{\mathbf{W}} = \sum_{\epsilon=1}^{T} \big(\boldsymbol{\alpha}^{\epsilon}_{r} (\boldsymbol{\alpha}^{\epsilon}_{c})^\top\big) \odot \mathbf{B}^{\epsilon}, \label{eq:binary95expansion} \end{align}\tag{2}\] where \(\mathbf{B}^{\epsilon}\in\{+1,-1\}^{n\times m}\) is the binary basis at order \(\epsilon\), \(\boldsymbol{\alpha}^{\epsilon}_{r}\in\mathbb{R}^{n}\) and \(\boldsymbol{\alpha}^{\epsilon}_{c}\in\mathbb{R}^{m}\) denote the row-wise and column-wise scaling vectors at order \(\epsilon\). \(\odot\) indicates the Hadamard product. Generally, most existing binarization methods [15], [16], [45] focus solely on minimizing the weight quantization error \(\mathcal{L}(\mathbf{W}, \widehat{\mathbf{W}})\) in Eq. 1 after post-training. They overlook the fact that model weights from different layers and modalities contribute differently to downstream tasks. Evidently, this phenomenon may lead to the mistaken retention of task-irrelevant encoding parameters, resulting in significant performance degradation.

Overview: To address the above challenge, we propose a significance-aware binarization map \(\boldsymbol{\Lambda} \in\mathbb{R}^{n \times m}\) to evaluate the contribution of weights across layers and modalities during binarization. \(\boldsymbol{\Lambda}\) forces the optimization to preserve important weights based on their significance, enhancing the representational capacity of the binarized low-bit weights. Thus, we utilize \(\sqrt{\boldsymbol{\Lambda}}\) to reweight Eq. 1 : \[\begin{align} \mathcal{L}(\mathbf{W}, \widehat{\mathbf{W}}) = \|\sqrt{\boldsymbol{\Lambda}} \odot (\mathbf{W}-\widehat{\mathbf{W}}) \|^2_F. \label{eq:weighted95binarization95objective} \end{align}\tag{3}\] As shown in Fig. 2, we present the algorithmic pipeline of our model for obtaining \(\boldsymbol{\Lambda}\) during post-training. First, we evaluate the sensitivity of model weights to different modalities based on activation values for text and image inputs, respectively. We then characterize the modal preference of local weights by computing the sparsity of sensitivity of different modalities. Second, we further compute the global preference of the weights. Finally, by integrating local and global preference, we compute dynamic sensitivity sparsity factor \(\sqrt{\boldsymbol{\Lambda}}\) to perform weighting to the Eq. 1 and then optimize \(\mathbf{\widehat{W}}\) through an alternating iteration process.

3.2 Significance-Aware Binarization Map↩︎

As introduced in [47], [48], sensitivity plays an important role in determining weight saliency during binarization. Inspired by these works, we utilize the Hessian matrix to quantify the sensitivity of weights with respect to different modalities for the significance-aware binarization map \(\boldsymbol{\Lambda}\). Specifically, we first sample \(K\) pairs of samples \(\mathbf{X} = \{\mathbf{x}_k^t, \mathbf{x}_k^v\}_{k=1}^K\) from COCO 2017 [49] as calibration data, where \(\mathbf{x}_k^t\) and \(\mathbf{x}_k^v\) represent the textual and visual modalities of the \(k\)-th sample. Subsequently, we input the \(k\)-th calibration sample \(\{\mathbf{x}_k^t, \mathbf{x}_k^v\}\) into LVLM to obtain \(\{\mathbf{E}_k^t, \mathbf{E}_k^v\}\). Here, \(\mathbf{E}_k^t \in\mathbb{R}^{l_p\times m}\) and \(\mathbf{E}_k^v \in\mathbb{R}^{l_p\times m}\) denote the hidden states of textual and visual modalities, and \(l_p\) is the token length. After constructing the Hessian matrices \(\mathbf{H}_k^t = 2(\mathbf{E}_k^t)^\top \mathbf{E}_k^t \in \mathbb{R}^{m \times m}\) and \(\mathbf{H}_k^v = 2(\mathbf{E}_k^v)^\top \mathbf{E}_k^v \in \mathbb{R}^{m \times m}\) for the \(k\)-th sample, we aggregate them over all samples to obtain \(\mathbf{H}^t, \mathbf{H}^v \in \mathbb{R}^{m \times m}\): \[\begin{align} \mathbf{H}^t=\sum^K_{k=1}\mathbf{H}_k^t + \eta^t \mathbf{I},~ \mathbf{H}^v=\sum^K_{k=1}\mathbf{H}_k^v + \eta^v \mathbf{I}, \label{eq:Hessian95matrices} \end{align}\tag{4}\] where \(I\) is the identity matrix, and \(\{\eta^t, \eta^v\}\) denote the damping coefficients used to ensure numerical stability. In this paper, we set \(\eta^t=\mathrm{Tr}(\sum^K_{k=1}\mathbf{H}_k^t)/m\) and \(\eta^v=\mathrm{Tr}(\sum^K_{k=1}\mathbf{H}_k^v)/m\). Then, we employ Hessian matrices to compute sensitivity matrix \(\mathcal{S} \in\mathbb{R}^{n\times m\times z}\), and \(\mathcal{S}_{ij} \in\mathbb{R}^z\) is defined as the \((i,j)\)-th element of \(\mathcal{S}\): \[\begin{align} \mathcal{S}_{ij} = \mathrm{Concat}(\mathcal{S}_{ij}^t, \mathcal{S}_{ij}^v), \label{eq:sensitivity} \end{align}\tag{5}\] \[\begin{align} \mathcal{S}_{ij}^t = \mathbf{W}_{ij}^2/(\mathbf{H}^{t}_{jj})^{-2},~ \mathcal{S}_{ij}^v = \mathbf{W}_{ij}^2/(\mathbf{H}^{v}_{jj})^{-2}, \label{eq:sensitivity95component} \end{align}\tag{6}\]

where \(z\) is the number of modalities, and we set \(z=2\) for textual and visual modalities. \(\mathrm{Concat}(\cdot)\) denotes the concatenation operation. \(\mathbf{W}_{ij}\) is the \((i, j)\)-th element of the weight \(\mathbf{W}\), \(\mathbf{H}^{t}_{jj}\) and \(\mathbf{H}^{v}_{jj}\) are the \((j, j)\)-th entry of \(\mathbf{H}^{t}\) and \(\mathbf{H}^{v}\).

3.2.1 Spatial Significance Map.↩︎

To evaluate the significance of the full-precision weight \(\mathbf{W}\) at different spatial encoding locations, we propose to construct a spatial significance map \(\mathbf{\Gamma} \in\mathbb{R}^{n\times m}\) by measuring the sparsity of \(\mathcal{S}\). As a result, the \((i, j)\)-th element \(\mathbf{\Gamma}_{ij}\) of \(\mathbf{\Gamma}\) can be formulated as follows: \[\begin{align} \label{eq:hoyer} \mathbf{\Gamma}_{ij} &= \frac{\sqrt{z}-\frac{\|\mathcal{S}_{ij}\|_1}{\|\mathcal{S}_{ij}\|_2}}{\sqrt{z}-1}\in(0, 1),~ \end{align}\tag{7}\] where \(\|\mathcal{S}_{ij}\|_1 = \sum_{u=1}^{z} \left|\mathcal{S}^{u}_{ij}\right|\) and \(\|\mathcal{S}_{ij}\|_2= \left(\sum_{u=1}^{z} (\mathcal{S}^{u}_{ij})^2\right)^{1/2}\) denote the \(\ell_1\) and \(\ell_2\) norms of \(\mathcal{S}_{ij}\). As can be seen from Eq. 7 , when the \((i,j)\)-th weight \(\mathbf{W}_{ij}\) is activated only by inputs from a single modality, \(\|\mathcal{S}_{ij}\|_1/\|\mathcal{S}_{ij}\|_2 \to 1\), corresponding to \(\mathbf{\Gamma}_{ij} \to 1\). Otherwise, when the \((i,j)\)-th weight \(\mathbf{W}_{ij}\) is activated under inputs from different modalities, \(\|\mathcal{S}_{ij}\|_1/\|\mathcal{S}_{ij}\|_2 \to \sqrt{z}\), corresponding to \(\mathbf{\Gamma}_{ij} \to 0\). Therefore, \(\mathbf{\Gamma}_{ij}\) approaches \(1\) when \(\mathcal{S}_{ij}\) concentrates on a single modality (unimodality), and \(\mathbf{\Gamma}_{ij}\) approaches \(0\) when \(\mathcal{S}_{ij}\) spreads across modalities (multimodality).

3.2.2 Modality-Guided Significance Integration.↩︎

Although \(\mathbf{\Gamma}\) in Eq. 7 measures spatial significance of encoding weights, it inherently biases the binarization process toward single-modality information, thus neglecting the complementary interactions across multiple modalities. To address this challenge, we leverage a modality integration score \(r\in\mathbb{R}\) to perform modality-guided significance integration. In this paper, we employ the sensitivity matrix \(\mathcal{S}\) to compute \(r\): \[\begin{align} \label{eq:mask-uni} r = \frac{\sum_{i,j}(\mathcal{S}_{ij}^{t}+\mathcal{S}_{ij}^{v})\cdot\mathcal{M}_{ij}^{\text{uni}}}{\sum_{i,j}(\mathcal{S}_{ij}^{t}+\mathcal{S}_{ij}^{v})} \in(0,1), \; \end{align}\tag{8}\] \[\begin{align} \label{eq:mask} \mathcal{M}^{uni}_{ij}=\left\{\begin{matrix} 1, \; if \; \mathbf{\Gamma}_{ij}>\tau, \\ 0, \; otherwise, \end{matrix}\right. \end{align}\tag{9}\] where \(\tau\) is the threshold for controlling sparsity. If \(\mathbf{\Gamma}_{ij} > \tau\), we consider the \((i,j)\)-th element \(\mathbf{W}_{ij}\) tends to be activated under a single modality; otherwise, it tends to be activated under inputs from different modalities. Therefore, the significance-aware binarization map \(\boldsymbol{\Lambda}\in\mathbb{R}^{n\times m}\) is defined as follows: \[\begin{align} \label{eq:lambda} \boldsymbol{\Lambda} = \underbrace{r \cdot\mathbf{\Gamma}}_{{\text{\scriptsize Unimodal}} } \;+\; \underbrace{(1-r)\cdot(1-\mathbf{\Gamma})}_{{\text{\scriptsize Multimodal}}}. \end{align}\tag{10}\] Evidently, larger values of \(\boldsymbol{\Lambda}\) in Eq. 10 impose a greater penalty on the quantization error in Eq. 3 . The significance-aware binarization map \(\boldsymbol{\Lambda}\) explicitly implements an error-control mechanism: when the modality integration score \(r > 0.5\), full-precision weights activated by a single modality receive larger \(\boldsymbol{\Lambda}\) values; otherwise, weights activated across multiple modalities are emphasized.

Theorem 1 shows that for any two distinct elements \((i,j)\) and \((p,q)\), \(\boldsymbol{\Lambda}_{ij} > \boldsymbol{\Lambda}_{pq}\) implies \(e_{ij}^* < e_{pq}^*\). Moreover, the optimal error is inversely proportional to the squared significance weight, i.e., \(e_{ij}^* \propto \boldsymbol{\Lambda}_{ij}^{-1}\). From a theoretical perspective, this theorem demonstrates that full-precision weights corresponding to larger values in the significance-aware binarization map \(\boldsymbol{\Lambda}\) incur smaller quantization errors. Theorem 1 thus provides theoretical support for our claim that the proposed significance-aware binarization map \(\boldsymbol{\Lambda}\) guides the optimization process to preserve task-relevant weights according to their significance.

2pt

max width=

2pt

max width=

3.3 Alternating Significance-Weighted Update↩︎

Inspired by [4], [17], [18], we propose an alternating refinement procedure to optimize Eq. 3 based on a sensitivity-aware binarization map \(\boldsymbol{\Lambda}\). Given the sensitivity matrix \(\mathcal{S}\), we set \(T=2\) for key weights to achieve enhanced representation capabilities; otherwise, \(T=1\). Additionally, in Eq. 2 , we set the binary basis \(\mathbf{B}=\mathrm{sign}(\mathbf{W})\) and use \(\partial \mathcal{L}(\mathbf{W}, \widehat{\mathbf{W}})/\partial \alpha^{\epsilon}_r = 0\) to update row-wise scaling vector \(\alpha^{\epsilon}_{r} \in\mathbb{R}^n\) at order \(\epsilon\): \[\begin{align} \label{eq:alpha-r} \alpha^{\epsilon}_{r}[i] = \frac{\sum_{j=1}^{m}\boldsymbol{\Lambda}_{ij}\, \mathbf{\Phi}^{\epsilon}_{ij}\, \widetilde{\mathbf{W}}_{ij}}{\sum_{j=1}^{m}\boldsymbol{\Lambda}_{ij}\, (\mathbf{\Phi}^{\epsilon}_{ij})^2},\; \mathbf{\Phi}^{\epsilon}_{ij} = \alpha^{\epsilon}_{c}[j]\mathbf{B}^{\epsilon}_{ij} \mathcal{M}^g_{ij}, \end{align}\tag{11}\] where \(\widetilde{\mathbf{W}} = \mathbf{W} - \mathbf{\widehat{W}} \in \mathbb{R}^{n \times m}\) is the residual matrix, \(\alpha^{\epsilon}_{r}[i]\) is the scale value in the \(i\)-th row, and \(\alpha^{\epsilon}_{c}[j]\) denotes the scale value of the \(j\)-th column at order \(\epsilon\). \(\mathcal{M}^g\) represents the group mask generated by [4], and \(\mathcal{M}^g_{ij}\) indicates the \((i, j)\)-th element of \(\mathcal{M}^g\). Similar to \(\alpha^{\epsilon}_r\), we update the column-wise scaling vector \(\alpha^{\epsilon}_c\in\mathbb{R}^m\) at order \(\epsilon\) by setting \(\partial \mathcal{L}(\mathbf{W}, \widehat{\mathbf{W}})/\partial \alpha^{\epsilon}_c=0\): \[\begin{align} \label{eq:alpha-c} \alpha^{\epsilon}_{c}[j] = \frac{\sum_{i=1}^{n} \boldsymbol{\Lambda}_{ij}\mathbf{\Psi}^{\epsilon}_{ij}\, \widetilde{\mathbf{W}}_{ij}}{\sum_{i=1}^{n} \boldsymbol{\Lambda}_{ij}(\mathbf{\Psi}^{\epsilon}_{ij})^2}, \; \mathbf{\Psi}^{\epsilon}_{ij} = \alpha^{\epsilon}_{r}[i]\mathbf{B}^{\epsilon}_{ij}\mathcal{M}^g_{ij}, \end{align}\tag{12}\] The optimization pipeline of our proposed SAB-LVLM is shown in Algorithm 3.

4 Experiments↩︎

4.1 Setup↩︎

Implementation Details. All experiments were conducted on a single NVIDIA A100 GPU (80G) and evaluated using lmms-eval [50] for zero-shot testing. We set the block size to 128 and performed 15 iterations of alternating significance-weighted update. We sampled \(K\) examples from the COCO 2017 [49] dataset as calibration data. In this paper,we set \(K=128\). Additionally, as a PTQ method, SAB-LVLM did not apply any training or fine-tuning of the model.

Models. To comprehensively demonstrate the effectiveness of our approach, we evaluate the proposed SAB-LVLM across different LVLMs, including Qwen2.5-VL [33], [39], [51] and InternVL3.5 [40] families. Our evaluation spans a diverse range of model capacities, specifically targeting the 7B, 32B, and 72B variants of Qwen2.5-VL, alongside the 8B, 14B, and 38B variants of InternVL3.5.

Benchmarks. Furthermore, to rigorously assess the versatility of SAB-LVLM across various downstream tasks, we conduct extensive experiments on multiple benchmarks, including MMStar [52], DocVQA [41], TextVQA [53], Video-MME [54], and VSI-Bench [55]. MMStar [52] is a comprehensive benchmark for LVLMs, comprising 1,500 meticulously curated samples designed to assess model capabilities across six core competencies. DocVQA [41] is a document-visual question-answering dataset comprising 50,000 questions based on over 12,000 document images. It primarily evaluates ability to understand and reason about document content and layout structures. TextVQA [53] is a visual question-answering dataset that requires LVLMs to read and understand text within natural scene images, then answer questions by integrating visual context. Video-MME [54] is a comprehensive evaluation benchmark for assessing the video analysis and temporal understanding capabilities of LVLMs. VSI-Bench [55] is a video benchmark for evaluating the visual-spatial intelligence of LVLMs. It constructs over 5,000 question-answer pairs, focusing on assessing spatial relationship comprehension, metric estimation, and spatio-temporal reasoning.

2pt

max width=

2pt

max width=

5pt

max width=

Baselines. We compare with various low-bit methods [4], [16]–[18] across multiple datasets and across multiple LVLMs. GPTQ [4] maintains performance at lower bit widths through layer-by-layer weighting based on second-order information. Note that GPTQ [4] is evaluated at 3-bit quantization, and we report its results only for performance reference rather than as a 1-bit baseline. BiLLM [17] is a 1-bit PTQ method that reduces compression error by identifying significant weights and combining them with binary residual approximation. PB-LLM [16] is a partial binarization method that preserves a small number of significant weights as high-bit representations while binarizing the remaining weights, thereby maintaining the reasoning capabilities with extremely low bit. ARB-LLM [18] reduces distribution bias between full-precision weights and binary weights by alternately optimizing the binarization parameters.

4.2 Comparison Results↩︎

To comprehensively evaluate the performance of the proposed SAB-LVLM across various downstream tasks, as shown in Tabs. ¿tbl:tab:qwen2595quant95results?-¿tbl:tab:vsibench95detailed95qwen?, we conducted a thorough assessment of the proposed SAB-LVLM against baseline methods across five multimodal benchmarks and multiple LVLMs.

4pt

max width=

3pt

max width=

4pt

In Tab. ¿tbl:tab:qwen2595quant95results?, SAB-LVLM consistently outperforms all 1-bit baselines across all five benchmarks while maintaining an approximate 1-bit quantization depth. Notably, PB-LLM [16] and BiLLM [17] suffer severe performance degradation after binarization, indicating that naive partial binarization or reliance solely on residuals is insufficient for LVLM. In contrast, the proposed SAB-LVLM achieves stable gains across perception, document understanding, and video understanding tasks. In Tab. ¿tbl:tab:internvl35958b95main95results?, we present results across the InternVL3.5 family. The proposed SAB-LVLM significantly outperforms existing 1-bit baselines in most benchmarks, indicating our approach is independent of specific backbone networks. In Tabs. ¿tbl:tab:qwen2595mmstar95results? and  ¿tbl:tab:qinternvl3595mmstar95results?, we report the detailed results on the MMStar benchmark across Qwen2.5-VL and InternVL3.5 families to further examine which capabilities are preserved. Compared to the 1-bit baselines, the proposed SAB-LVLM better preserves coarse perception and reasoning capabilities. In Tab ¿tbl:tab:vsibench95detailed95qwen?, we present the detailed results of VSI-Bench on Qwen2.5-VL-32B-Instruct. SAB-LVLM outperforms baseline methods across multiple tasks and achieves comparable performance to full-precision models in object appearance order, object absolute distance, and route planning tasks. In Fig. 4, We present qualitative analysis on MMStar using QwenVL-32B-VL. The three questions in the figure respectively tested the perception, counting, and reasoning capabilities of LVLM. Compared to ARB-LLM [15], SAB-LVLM successfully answered all three questions.

4.3 Ablation Study↩︎

In this section, we analyze the proposed components, including the spatial significance map, modality-guided significance integration (MGSI), and different variants of SAB-LVLM. All experiments were conducted on the MMStar benchmark using Qwen2.5-VL-32B, and detailed performance metrics are reported.

4.3.1 Analysis of Spatial Significance Map.↩︎

As shown in Tab. ¿tbl:tab:ablation95tau95mmstar?, we evaluate the performance of SAB-LVLM on Qwen2.5-VL-32B using three different values of the threshold \(\tau\) on the MMStar benchmark. The results show that the spatial significance map is sensitive to the choice of \(\tau\). Among all settings, \(\tau=0.0003\) achieves the best overall performance, reaching 54.77 and improving the baseline by 6.29 points. In particular, this setting yields more evident gains on instance reasoning, logical reasoning, mathematics, and science & technology, indicating that an appropriate threshold helps preserve multimodal reasoning ability under extreme low-bit compression. When \(\tau\) is too small or too large, the distinction between weights activated under a single modal and those activated activated across modalities becomes less effective, which limits the benefit of significance-aware binarization map. These results indicate that partitioning spatial significance map using an appropriate threshold \(\tau\) is crucial for preserving the capabilities of LVLMs.

4.3.2 Analysis of Modality-Guided Significance Integration.↩︎

As shown in Tab. ¿tbl:tab:variant95r95mmstar?, we evaluate the performance of MMStar on Qwen2.5-VL-32B-Instrust with different variants of modality-guided significance integration. Using a fixed modality integration score \(r\) yields limited improvements. The best fixed setting (\(r=0.2\)) reaches 51.15, still below the MGSI (54.77). As shown in Fig. 5, we present the distribution of \(r\) among different layers, including self-attention output projection, down projection layers. Since \(r_\ell\) varies significantly across different layers, simply fixing \(r\) yields limited effectiveness. Therefore, adaptively estimating \(r\) for each layer is more effective than using a global constant. Moreover, using only a single-modality sensitivity map (only \(\mathcal{S}^t\) or only \(\mathcal{S}^v\)) still underperforms the joint MGSI (52.27/51.54 vs. 54.77), confirming that LVLM binarization requires jointly modeling both modalities rather than treating them independently.

4.3.3 Analysis of different variants of SAB-LVLM.↩︎

As shown in Tab. 1, we report MMStar results on Qwen2.5-VL-32B-Instruct under different variants of SAB-LVLM. “w/o SAB” denotes the baseline, which adopts the same alternating refinement strategy as [18]. “w/ SAB-R” applies row-wise significance reweighting in the alternating significance-weighted update, where all weights within the same row share an identical significance value, i.e., significance value of \(i\)-th row \(\boldsymbol{\Lambda}_{ij} =\frac{1}{n}\sum_{j=1}^nr \cdot\mathbf{\Gamma}_{ij}+(1-r)\cdot(1-\mathbf{\Gamma}_{ij})\). “w/ SAB-C” applies column-wise significance reweighting, where all weights within the same column share an identical significance value, i.e., significance value of \(j\)-th column \(\boldsymbol{\Lambda}_{ij} =\frac{1}{m}\sum_{i=1}^m r \cdot\mathbf{\Gamma}_{ij}+(1-r)\cdot(1-\mathbf{\Gamma}_{ij})\). Finally, “w/ SAB” uses the proposed significance map defined in Eq. 10 . As shown in Tab. 1, the complete spatial significance map (w/ SAB) achieved the highest average score.

5 Conclusion↩︎

In this paper, we introduce SAB-LVLM, a novel significance-aware binarization framework for large vision-language models, which serves as an early exploration of weight binarization for LVLMs. Specifically, we first construct a spatial significance map using multimodal calibration data to distinguish full-precision weights activated under a single modality from those activated across modalities. We then devise a modality-guided significance integration strategy to measure weight significance across layers and modalities, and further build a significance-aware binarization map. Based on this design, we employ an alternating significance-weighted update scheme to perform LVLM binarization under approximate 1-bit constraints. Extensive experiments on representative LVLM benchmarks demonstrate that SAB-LVLM consistently outperforms existing binarization methods across multiple downstream tasks. In the future work, we will evaluate the proposed SAB-LVLM on a broader set of LVLM architectures, more diverse multimodal tasks, or longer-context video understanding scenarios.

References↩︎