MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
January 01, 1970
Abstract
Vision-Language Models (VLMs) improve generalization and interpretability in autonomous driving but suffer from efficiency issues due to long visual token sequences, particularly in standard multi-view settings. Existing token pruning methods employ fixed pruning rate allocation and static importance metrics, ignoring dynamic inter-view importance differences and the evolving information importance during inference. Our analysis reveals that multi-view VLMs inherently encode task-related view priors in deeper layers and exhibit dynamic information requirements. Motivated by these findings, we propose MVPruner, a two-stage adaptive token pruning method that aligns pruning behavior with the model’s dynamic information requirements. The first stage allocates pruning budgets based on the information diversity of each view, and retains tokens with consistent contribution across stages, ensuring semantic representational capacity. The second stage allocates budgets and selects tokens guided by instruction text to guarantee task alignment. Experimental results on four benchmarks demonstrate the superior performance of our method. For example, DriveMM equipped with MVPruner achieves 87.3% reduction in FLOPs, 4.97× speedup in prefilling phase while retaining 98.5% accuracy on DriveLM benchmark.
1 sec:Introduction↩︎
Vision-Language Models (VLMs) [1]–[3] have shown promise for achieving interpretable and generalizable perception and decision-making in autonomous driving [4]–[9]. However, to ensure comprehensive environmental perception, autonomous driving systems typically rely on multi-view camera inputs [10]–[14]. The high-resolution multi-view images inevitably generate excessive visual token sequences, leading to substantial computational overhead and inference latency [15], [16], which severely hinders the deployment in latency-sensitive driving scenarios [17].
To improve inference efficiency, visual token pruning has emerged as an effective strategy for eliminating redundant information [18]–[20]. Although pruning techniques for VLMs have achieved preliminary progress, existing methods still face several critical challenges in multi-view autonomous driving scenarios: (1) Most existing methods are designed for single-view settings and apply uniform pruning ratios across views when handling multi-view inputs, ignoring the varying contributions of different views [15], [16], [21]. 1 illustrates this effect: in braking scenarios triggered by a red light ahead, removing rear views does not affect the decision, whereas pruning the front view containing the key red light cue leads to incorrect and unsafe behavior. Although recent work (, Prune2Drive [17]) allocates view-specific pruning budgets via offline hyperparameter search, such static strategies are sensitive to scene variations and cannot adapt to varying driving tasks and instructions. (2) Widely used token pruning methods rely on static importance evaluation metrics or single-stage pruning strategies, failing to account for the model’s dynamic information requirements and consequently resulting in suboptimal preservation of multi-view information.
To address these challenges, we first design a task-related view recognition experiment to quantitatively evaluate the ability of multi-view VLMs to recognize critical views across different layers. As shown in 2 (a), the accuracy is low in shallow layers, increases substantially in intermediate layers where it reaches its peak. This result indicates that VLMs inherently encode task-related view priors during deeper inference. Furthermore, we investigate the underlying mechanisms behind this layer-wise discrepancy. As shown in Figs. 2 (b) and (c), VLMs exhibit hierarchical attention patterns that shift from attending to broad visual context for global scene understanding to focusing on task-relevant local regions, and dynamic information requirements that transition from semantic diversity-driven processing to task relevance-driven processing.
The above findings provide principled guidance on which stages to prune, how to allocate pruning budgets, and which tokens to retain. Based on these insights, we propose MVPruner, a two-stage adaptive token pruning framework for multi-view VLMs that aligns budget allocation and token selection with the model’s dynamic information requirements. In the first stage, inspired by the model’s broad attention to all visual information in shallow layers, we design the Diversity-aware Ratio Allocation strategy that measures intra-view feature diversity to assess view importance and allocate larger budgets to information-rich views. Meanwhile, a Cross-stage Contribution-aware Token Selection mechanism is proposed to retain tokens that consistently contribute across different layers, satisfying the dynamic information requirements during inference. In the second stage, the textual instructions can effectively guide view-importance estimation in deeper layers. Therefore, we introduce the Instruction-aware Ratio Allocation strategy that measures semantic relevance between visual and instruction tokens, assigning larger budgets to task-related views. Finally, we select tokens guided by instruction text to enhance task-alignment capabilities. Experiments on the image-based multi-view autonomous driving benchmarks: DriveLMM-o1[14], DriveLM [13], and MAPLM [22], as well as video-based benchmark STSnu [23] demonstrate that our method can consistently accelerate inference while effectively maintaining reasoning accuracy, exhibiting strong generalization in autonomous driving. Our contribution can be summarized as follows:
(1) Our analysis reveals that multi-view VLMs allocate more pronounced attention to task-related views in deeper layers and exhibit dynamic information requirements, providing insights to guide method design.
(2) We propose MVPruner, which dynamically adjusts pruning ratios across different views in an online manner and adaptively retains important tokens according to the model’s dynamic information requirements.
(3) Experimental results demonstrate that our method achieves a superior efficiency–performance trade-off. With 10% of visual tokens retained, our method maintains accuracy of 98.0%, 98.5%, and 94.5%, outperforming the second-best method by 2.6%, 0.8% and 3.5%, achieving \(3.80\times\), \(4.97\times\) and \(3.95\times\) speedup in the prefilling stage on DriveLMM-o1, DriveLM and MAPLM, respectively.
2 sec:Related32Work↩︎
2.1 VLMs for Autonomous Driving↩︎
By jointly modeling visual inputs and language instructions within a shared embedding space, VLMs enhance reasoning, interpretability, and generalization in autonomous driving, supporting complex driving tasks [5], [24]–[28]. Early efforts including GPT-Driver [4], demonstrate that frozen VLMs can process driving tasks while simultaneously producing human-readable rationales. Recently, DriveLM [13] and DriveLMM-o1 [14] have respectively introduced a graph-based visual question answering dataset and a step-by-step reasoning dataset to enhance the driving-scene understanding of VLMs. DriveMM [29] designs a general VLM to process diverse data inputs and perform a broad spectrum of tasks. However, the long visual token sequences, especially for multi-view inputs, lead to significant inference latency and limit their practical applications.
2.2 Visual Token Pruning for VLMs↩︎
Recently, token pruning methods have attracted increasing attention as an effective acceleration strategy for VLMs [18], [30]. These methods are generally categorized into two groups [20]: (1) attention-based methods, which prune unimportant tokens via attention scores [15], [16], [31], [32]; (2) similarity-based methods [21], [33], [34], which either merge similar tokens or maximize diversity of retained tokens. However, they mainly focus on single-view inputs and overlook the varying redundancy across views. Although recent work, Prune2Drive [17], allocates pruning budgets across views through offline hyperparameter search, it incurs additional computational overhead and exhibits limited generalization capability. Moreover, these methods rely on static importance metrics or single-stage pruning, overlooking the model’s cross-layer dynamic information requirements.
3 sec:Method↩︎
3.1 Multi-view Vision-Language Models↩︎
A multi-view VLM [13], [14] typically processes a pair of inputs, denoted as \((V, T)\), where \(V\) is the multi-view image input and \(T\) is the text input which is composed of system text and instruction text. The visual input \(V\in\mathbb{R}^{P\times H \times W \times 3}\) is converted to visual tokens \(X^{v}\in\mathbb{R}^{P\times M \times D}\) using a vision encoder and a projector layer, where \(P\) is the number of views, \(M\) is the number of tokens within each view and \(D\) is the dimension of the feature. The text input is mapped to textual tokens \(X^{t}\in\mathbb{R}^{N \times D}\) using a text encoder, where \(N\) is the number of textual tokens. Then, they are fed to an LLM to generate the prediction.
3.2 Key Observations↩︎
In autonomous driving tasks, textual instructions often exhibit explicit spatial specificity (, “Would the object in the back-view camera be in the moving direction?”). These samples enable us to quantitatively analyze whether multi-view VLMs can inherently identify instruction-referenced views. To investigate this, we design a task-related view recognition experiment. Specifically, we evaluate two representative models, DriveMM [29] and DriveLMM-o1 [14], by extracting view-specific subsets from the DriveLM [13] and DriveLMM-o1 datasets. For each decoder layer, we compute the cross-modal attention scores between the instruction and visual tokens of each view. An identification is deemed correct if the view yielding the maximum score aligns with the instruction-referenced view. As shown in 2 (a), as the layers deepen, the accuracy gradually increases and reaches its peak in the middle layers. This finding indicates that cross-modal attention patterns in the deeper layers of VLMs effectively identify task-relevant views.
Furthermore, to examine layer-wise differences in recognition accuracy, we analyze the cross-layer evolution of attention patterns between visual and textual modalities. Specifically, we compute the entropy of text-to-vision attention at each layer to measure the distribution of attended visual information. As shown in 2 (b), entropy is initially high, decreases rapidly with fluctuations, and further declines in deep layers. This trend reflects a shift in the model’s attention patterns, consistent with the attention maps in 2 (c). In early layers, textual semantics interact with broad, global visual context to establish initial scene understanding and coarse cross-modal alignment. In intermediate layers, the model progressively concentrates on task-relevant visual regions. In deep layers, visual information is further aggregated and exhibits consistently high attention at fixed spatial locations within each view. This finding suggests that reasoning in VLMs exhibits dynamic information requirements that transition from semantic diversity-driven to task relevance-driven.
3.3 Framework Overview↩︎
Inspired by the above findings, we propose MVPruner, a two-stage pruning framework that transitions from structural summarization to semantic focusing, aligning with the model’s dynamic information requirements, as shown in 3. Specifically, in the first stage, the Diversity-aware Ratio Allocation strategy is employed in the shallow layer to assign pruning rates based on differences in information diversity across views. Meanwhile, the Cross-stage Contribution-aware Token Selection mechanism retains tokens that consistently contribute across different stages. In the second stage, the Instruction-aware Ratio Allocation strategy is applied in the deeper layer, where pruning rates are determined by the task relevance of each view. Finally, redundant tokens are further removed guided by instruction.
3.4 Diversity-aware Ratio Allocation↩︎
In shallow layers, attention is relatively uniformly distributed across all views, reflecting the reasoning process driven by semantic diversity. This observation motivates us to start from the information richness within each view, preserving more information for views containing more diverse content, supplying the deeper layers with a more comprehensive visual context. To this end, we propose the Diversity-aware Ratio Allocation (DRA) strategy that dynamically assigns the pruning budget based on the measured feature diversity of each view, maximizing the diversity and completeness of visual representations during compression.
Firstly, we compute the semantic distances between all token pairs \(\left(X_{p, i}^{v}, X_{p, j}^{v}\right)\) within each view \(p\), \(\left(i, j\right)\in M,i \neq j\) to estimate feature diversity. The distance \(d_{p}(\cdot, \cdot)\) is defined as: \[d_{p}\left(X_{p, i}^{v}, X_{p, j}^{v}\right) = 1 - \frac{X_{p, i}^{v} \cdot X_{p, j}^{v}}{\left\|X_{p, i}^{v}\right\| \left\|X_{p, j}^{v}\right\|}. \label{eq:distance}\tag{1}\]
For each token, the minimum pairwise distance is used as its uniqueness score \(S_{p,i}^u\), where a higher score indicates that the token is more unique relative to others in the same view: \[S_{p,i}^u = min\left(d_{p}(X_{p, i}^{v}, X_{p, 1:M}^{v})\right). \label{eq:min}\tag{2}\]
Subsequently, the diversity score of a view \(S_{p}^u= \frac{1}{M} \sum_{i=1}^{M} S_{p,i}^u\) is obtained by averaging the uniqueness scores of all its tokens, with a higher score indicating a greater feature density compared to other views. The retention ratio \(r_{1,p}\) and number \(k_{1,p}\) for each view is determined based on the normalized \(S_{p}^u\): \[r_{1,p} = r_{1} \times P \times S_{p}^u,\quad k_{1,p}=r_{1,p}\times M, \label{eq:allcate1}\tag{3}\] where \(r_{1}\) is the average retention ratio for the first-stage pruning.
3.5 Cross-stage Contribution-aware Token Selection↩︎
Tokens retained in shallow layers should support task-agnostic global semantic modeling while providing a foundation for task-relevant representation extraction in deeper layers. To this end, we propose the Cross-stage Contribution-aware Token Selection (CCTS) mechanism that jointly evaluates a token’s immediate contribution to the current stage and its potential contribution to subsequent stages, aligning selection with the model’s dynamic information requirements.
Specifically, we formulate the token selection in shallow layers as an optimization problem aimed at maximizing cross-stage contribution. In this formulation, immediate contribution is quantified by semantic uniqueness within visual space, while future contribution is estimated through task relevance which is computed as the cosine similarity between visual and instruction tokens. An iterative greedy selection strategy is then employed to efficiently search for the optimal token subset.
Firstly, the token with the highest task relevance is selected to initialize the set of selected tokens. Subsequently, the minimum semantic distance between a candidate token and the selected set is computed using 2 and used as the semantic uniqueness metric, denoted by \(U_{p,i}\). This metric is further weighted by the task relevance score \(R_{p,i}\), enabling a comprehensive assessment of the token’s cross-stage contribution \(Imp_{p,i}\): \[Imp_{p,i}=U_{p,i} \times R_{p,i}.\]
After that, the token with the highest cross-stage contribution is appended to the selected set. By iterating this process until the target token budget is reached, this mechanism maximizes cross-stage information utility. The algorithm flow is detailed in [alg].
3.6 Instruction-aware Ratio Allocation↩︎
In deeper layers, attention is focused on local task-related regions, and the model can distinguish the task relevance of different views, reflecting the reasoning process driven by task relevance. Motivated by this, we design the Instruction-aware Ratio Allocation (IRA) strategy for the second stage, which adaptively allocates pruning budget based on semantic relevance between visual and instruction tokens, retaining more critical information for task-related views.
Specifically, we extract the attention matrix between the visual tokens and textual instruction tokens, and compute the mean attention score of all tokens within each view as its semantic relevance score \(S_{p}^r\). A higher score indicates that the view exhibits greater task relevance than other views: \[S_{p,i}^r =\sum_{j=1}^{\tilde{N}}A_{i,j}^{p},\quad S_{p}^r = \sum_{i=1}^{k_{1,p}}S_{p,i}^r, \label{eq:sematic}\tag{4}\] where \(\tilde{N}\) is number of instruction tokens, \(S_{p,i}^r\) is attention score of \(i\)-th token in view \(p\), \(A^{p}\) is attention matrix between view \(p\) and instruction tokens.
After that, we reallocate the remaining token budget across views using a compensatory normalization scheme: \[r_{2,p} = r_{2} \times \left(1 + S_{p}^r - S_{p}^u \right),\quad k_{2,p}=r_{2,p}\times k_{1,p}, \label{eq:allcate2}\tag{5}\] where \(r_{2}\) denotes the average retention ratio for the second-stage pruning, and \(S_{p}^r - S_{p}^u\) represents the relative deviation from the diversity score. \(r_{2,p}\) and \(k_{2,p}\) are the retention ratio and number for each view.
Finally, the top \(k_{2,p}\) tokens with the highest attention score \(S_{p,i}^r\) to instruction tokens are selected to further enhance the task-alignment capability.
4 sec:Experiments↩︎
This section first outlines the experimental setup, followed by a comparative evaluation against state-of-the-art (SOTA) methods. We then conduct ablation studies to validate the effectiveness of the proposed method. Finally, visualization results are presented to demonstrate its adaptability.
l c c c c c >p2.1cm Methods & Risk Assessment Accuracy & Traffic Rule Adherence & Scene Awareness and Object Und. & Relevance & Missing
Details & Overall Reasoning
& 73.81 & 82.26 & 75.64 & 79.51 & 71.38 & *74.37
FastV(ECCV24) & 71.35 & 80.58 & 72.85 & 76.74 & 69.25 & 72.41(97.4%)
DART(EMNLP25) & 71.50& 78.21& 72.80& 77.33& 69.63 & 71.46(96.1%)
*SparseVLM(ICML25) & 70.08 & 79.59 & 71.87 & 75.66 & 67.43 & 71.23(95.8%)
PACT(CVPR25) & 71.20 & 78.11 & 71.44 & 77.21 & 68.95 & 70.95(95.4%)
DivPrune(CVPR25) & 71.66 & 78.39 & 73.04 & 77.45 & 69.92 & 71.67(96.4%)
Prune2Drive(CVPR26) & 71.31 & 80.74 & 73.07 & 77.48 & 69.30 & 72.69(97.7%)
Ours & 72.91 & 81.96 & 74.85 & 78.55 & 71.58 & 73.79(99.2%)
FastV(ECCV24) & 67.74 & 78.07 & 69.37 & 73.24 & 66.02 & 69.56(93.5%)
DART(EMNLP25) & 68.60& 75.81& 69.97& 74.94&67.60& 69.01(92.8%)
*SparseVLM(ICML25) & 68.18 & 77.81 &69.87& 73.98& 66.20& 69.40(93.3%)
PACT(CVPR25) & 68.20& 76.23& 69.11& 74.14&66.10& 68.38(91.9%)
DivPrune(CVPR25) & 69.32& 76.47& 70.78& 75.53&68.22& 69.65(93.7%)
Prune2Drive(CVPR26) & 70.76 & 77.66 & 72.29 & 77.01 & 68.94 & 70.31(94.5%)
Ours & 72.05 & 81.81 & 73.80 & 77.61 & 70.25 & 72.91(98.0%)
4.1 Experimental Setup↩︎
Models. We verify the effectiveness of the proposed method by experiment on two specialist autonomous driving visual-language models: DriveMM [29] and DriveLMM-o1 [14]. Specifically, DriveMM is built based on LLaVA-OneVision-7B [3] and fine-tuned across various autonomous driving datasets, can process diverse data inputs and driving tasks. DriveLMM-o1 is built based on InternVL2.5-8B [35] and fine-tuned in the DriveLMM-o1 dataset to enhance step-by-step reasoning ability in driving scenarios.
Benchmarks. We evaluate our method on three large-scale image-based autonomous driving VQA benchmarks: DriveLM [13], DriveLMM-o1 [14], and MAPLM [22], as well as the video-based benchmark STSnu [23]. DriveLM is annotated on the NuScenes [10] dataset, including QA pairs of perception, prediction, and planning tasks with six-view images. DriveLMM-o1 is also annotated on NuScenes to advance step-by-step visual reasoning for autonomous driving with six-view images as input. MAPLM is tailored for advanced map and traffic scene understanding, supporting tasks such as scene classification, lane counting, and point cloud quality analysis based on three-view images and synchronized LiDAR point cloud maps. STSnu is proposed to evaluate the spatio-temporal reasoning capabilities of VLMs based on six-view videos. We follow the common metrics in each benchmark for fair comparison. Details are in the supplementary materials.
Evaluation Metrics. We follow the common metrics in each benchmark for fair comparison: (1) DriveLM [13] implements four evaluation metrics: accuracy, GPT score, language-based evaluation (BLEU-4, Rouge, and CIDEr), and match score. (2) DriveLMM-o1 [14] adopts five driving-specific reasoning evaluation metrics: Risk Assessment Accuracy, Traffic Rule Adherence, Scene Awareness and Object Understanding, Relevance, and Missing Details, as well as Overall Reasoning. (3) MAPLM [22] uses FRM (Frame-overall-accuracy) and QNS (Question-overall-accuracy). (4) STSnu [23] categorizes questions into four interaction types and evaluates them separately: (i) Ego: Which of the following options best describes the ego vehicle’s driving maneuver? (ii) Agent: Which of the following options best describes the driving behavior of the <reference to agent>? (iii) Ego-to-agent: Which of the following options best describes the ego vehicle’s driving behavior with respect to the <reference to agent>? (iv) Agent-to-agent: Which of the following options best describes <reference to agent 1>’s maneuver with respect to <reference to agent 2>?
Comparison Methods. We compare our method with several recent representative token pruning methods, including FastV [15], DART [34], SparseVLM [16], PACT [36], and DivPrune [33]; as well as Prune2Drive [17] which is specifically designed for multi-view VLMs. To ensure a fair comparison, we optimized the pruning configuration for Prune2Drive on each benchmark using its official open-source implementation. To assess pruning robustness, we evaluate all methods under two widely adopted pruning settings, where on average 75% and 90% of visual tokens are pruned. The pruning ratio is computed as the average token reduction across all layers of the LLM, following prior works [15]–[17], [33], [34], [36]. Additional experiments under more pruning settings are provided in the supplementary material.
Implementation Details. Our two-stage pruning is applied before layers 0 and 16, with average pruning ratios of \(r\) and \(r^2\), causing an exponential reduction.
| Model | Accuracy | Chatgpt | BLEU-4 | Rouge | CIDEr | Match | Average |
|---|---|---|---|---|---|---|---|
| DriveMM | 0.81 | 65.44 | 0.61 | 0.74 | 0.19 | 33.9 | 59.1 |
| FastV | 0.77 | 63.95 | 0.54 | 0.72 | 0.15 | 32.8 | 56.6(95.8%) |
| DART | 0.78 | 64.02 | 0.56 | 0.73 | 0.18 | 33.6 | 57.2(96.8%) |
| SparseVLM | 0.79 | 63.74 | 0.57 | 0.73 | 0.19 | 33.4 | 57.4(97.1%) |
| PACT | 0.78 | 64.24 | 0.56 | 0.73 | 0.19 | 33.3 | 57.3(97.0%) |
| DivPrune | 0.79 | 64.67 | 0.59 | 0.74 | 0.21 | 33.7 | 57.9(98.0%) |
| Prune2Drive | 0.80 | 64.92 | 0.60 | 0.75 | 0.20 | 34.0 | 58.3(98.6%) |
| Ours | 0.79 | 65.52 | 0.61 | 0.75 | 0.22 | 34.3 | 58.7(99.3%) |
| FastV | 0.68 | 63.52 | 0.49 | 0.71 | 0.08 | 32.3 | 54.1(91.5%) |
| DART | 0.69 | 64.20 | 0.51 | 0.71 | 0.12 | 32.0 | 54.7(92.6%) |
| SparseVLM | 0.75 | 63.38 | 0.52 | 0.72 | 0.15 | 32.9 | 55.9(94.6%) |
| PACT | 0.76 | 64.76 | 0.54 | 0.73 | 0.15 | 32.9 | 56.8(96.1%) |
| DivPrune | 0.78 | 64.16 | 0.55 | 0.73 | 0.17 | 32.7 | 57.0(96.4%) |
| Prune2Drive | 0.78 | 64.52 | 0.56 | 0.74 | 0.16 | 33.4 | 57.4(97.1%) |
| Ours | 0.79 | 65.41 | 0.58 | 0.74 | 0.17 | 34.0 | 58.2(98.5%) |
4.2 Main Results↩︎
This section presents comparative results between our method and SOTA methods on DriveLMM-o1, DriveLM, MAPLM and STSnu benchmarks.
DriveLMM-o1. The results are presented in [tab:drivelmm]. Our method retains 98.0% and 99.2% of the original performance using only 10% and 25% of the visual tokens, respectively, outperforming SOTA methods. Notably, under the 90% pruning setting, our method outperforms the second-best method Prune2Drive by a substantial margin of 2.6 on the Overall Reasoning metric, without any offline hyperparameter search.
DriveLM. The results in 1 show that our method retains near-original performance with only 10% and 25% tokens, achieving 98.5% and 99.3% of the original model, respectively. Notably, it even surpasses the original model on Chatgpt, Rouge, CIDEr and Match under the 75% pruning setting. Compared to Prune2Drive, our method yields consistent improvements of 0.8 and 0.4.
MAPLM. Results in 2 show that under the more challenging multi-sensor input setting, our method exhibits a substantially smaller accuracy drop than SOTA methods. With 90% pruning, FastV, SparseVLM, and Prune2Drive degrade by 17.63, 15.84, and 7.43, respectively, whereas our method drops by only 3.94, remaining close to the original model. This indicates that our method can effectively identify critical information across diverse inputs and exhibits stability and strong generalization capability across tasks.
| Model | FRM | QNS | Average | FRM | QNS | Average |
|---|---|---|---|---|---|---|
| 58.60 | 85.87 | 72.24 | 58.60 | 85.87 | 72.24 | |
| FastV | 64.69(89.5%) | 54.61(75.6%) | ||||
| (93.1%) | (82.7%) | |||||
| (91.6%) | (78.1%) | |||||
| (92.3%) | (73.1%) | |||||
| (94.5%) | (89.7%) | |||||
| 53.47 | 83.10 | 68.29(94.5%) | 48.33 | 81.28 | 64.81(89.7%) | |
| Ours | 56.07 | 84.55 | 70.31(97.3%) | 53.27 | 83.33 | 68.30(94.5%) |
4.5pt
l c c c c c Methods & Ego & Ego-to-Agent & Agent & Agent-to-Agent & Average
& 43.16 & 59.33 & 43.70 & 30.67 & 44.21
FastV & 40.02 & 63.08 & 42.72 & 26.81 & 43.16(97.6%)
DART & 38.70& 58.32& 44.24& 28.79& 42.51(93.9%)
*SparseVLM & 31.04 & 59.62 & 43.29 & 32.89 & 41.71(94.3%)
PACT & 31.28 & 59.98 & 42.79 & 31.63 & 41.42(93.7%)
DivPrune& 36.65 & 58.16 & 42.60 & 29.82 & 41.81(94.6%)
Prune2Drive & 36.65 & 58.52 & 43.04 & 30.51 & 42.18(95.4%)
Ours & 44.59 & 58.34 & 43.08 & 30.99 & 44.25(100%)
FastV & 19.55 & 61.90 & 38.33 & 28.19 & 36.99(83.7%)
DART& 15.41& 37.76& 36.08& 26.35&28.90(65.4%)
*SparseVLM & 26.06 & 69.18 &40.30& 29.16& 41.18(93.1%)
PACT& 15.86& 65.96& 39.00& 27.79&37.15(84.0%)
DivPrune & 30.75& 59.50& 43.79& 30.95&41.25(93.3%)
Prune2Drive & 31.89 & 72.76 & 40.20 & 27.84 & 43.17(97.6%)
Ours & 39.33 & 57.75 & 43.49 & 31.46 & 43.00(97.3%)
STSnu. On the STSnu benchmark, we apply our MVPruner and other methods independently to each frame of the six-view video sequences. The results are presented in [tab:STSnu]. Under the 75% pruning setting, our method even surpasses the original model by 0.04, outperforming SOTA methods and demonstrating its effectiveness in removing spatiotemporal redundancy.
By preserving information aligned with the dynamic requirements of the model’s reasoning process, our method achieves consistent performance gains across diverse benchmarks and exhibits strong generalization.
Inference Efficiency. We analyze the efficiency of our method using speedup in the prefilling and decoding stages, FLOPs, GPU peak memory usage and throughput, under the 90% pruning setting. The results are shown on 3, our method achieves the fastest decoding time, the lowest FLOPs, and GPU peak memory usage. Despite achieving efficiency comparable to Prune2Drive, our method preserves performance more effectively and can be directly applied to diverse models and scenarios without additional adjustments or costly hyperparameter tuning, offering a more practical and scalable solution.
We further profile each component using DriveMM on DriveLM with six views and 729 tokens/view under the 90% pruning setting. As shown in 4, CCTS dominates the overhead due to pairwise similarity computation, taking 26.48 ms/view. The total pruning overhead is 163.9 ms, only 2.9% of the full model’s end-to-end latency (5580 ms). The detailed computing budget of our method is detailed in the supplementary materials.
| Methods | Speedup (ms) \(\downarrow\) | FLOPs \(\downarrow\) | GPU Peak \(\downarrow\) | Throughput \(\downarrow\) | Performance \(\uparrow\) | |||
| (Prefilling) | (Decoding) | Memory (GB) | (samples/s) | |||||
| 591 | 51.9 | 11.84 | ||||||
| + FastV | 128.5(\(\uparrow\)4.60\(\times\)) | 45.9(\(\uparrow\)1.13\(\times\)) | 20.3% | 16.61 | 10.57(\(\uparrow\)1.12\(\times\)) | 69.56 | ||
| + SparseVLM | 164.6(\(\uparrow\)3.59\(\times\)) | 47.6(\(\uparrow\)1.09\(\times\)) | 20.7% | 16.65 | 9.55(\(\uparrow\)1.24\(\times\)) | 69.40 | ||
| + Prune2Drive | 127.1(\(\uparrow\)4.65\(\times\)) | 47.2(\(\uparrow\)1.10\(\times\)) | 20.3% | 16.58 | 8.58(\(\uparrow\)1.38\(\times\)) | 70.31 | ||
| + Ours | 155.5(\(\uparrow\)3.80\(\times\)) | 42.2(\(\uparrow\)1.23\(\times\)) | 20.0% | 16.08 | 8.60(\(\uparrow\)1.38\(\times\)) | 72.91 | ||
| 65.3 | 8.12 | |||||||
| + FastV | 340.5(\(\uparrow\)5.78\(\times\)) | 63.4(\(\uparrow\)1.03\(\times\)) | 13.3% | 15.99 | 5.56(\(\uparrow\)1.46\(\times\)) | 55.4 | ||
| + SparseVLM | 484.7(\(\uparrow\)4.06\(\times\)) | 64.0(\(\uparrow\)1.02\(\times\)) | 14.4% | 16.00 | 5.88(\(\uparrow\)1.38\(\times\)) | 55.9 | ||
| + Prune2Drive | 307.5(\(\uparrow\)6.40\(\times\)) | 59.9(\(\uparrow\)1.09\(\times\)) | 13.4% | 16.00 | 5.49(\(\uparrow\)1.48\(\times\)) | 57.4 | ||
| + Ours | 395.6(\(\uparrow\)4.97\(\times\)) | 51.8(\(\uparrow\)1.26\(\times\)) | 12.7% | 15.93 | 58.2 | |||
| 1182 | 61.7 | 1.72 | ||||||
| + FastV | 286.9(\(\uparrow\)4.12\(\times\)) | 58.2(\(\uparrow\)1.06\(\times\)) | 15.6% | 15.91 | 0.82(\(\uparrow\)2.10\(\times\)) | 54.61 | ||
| + SparseVLM | 353.9(\(\uparrow\)3.34\(\times\)) | 59.9(\(\uparrow\)1.03\(\times\)) | 15.8% | 15.93 | 1.11(\(\uparrow\)1.55\(\times\)) | 56.40 | ||
| + Prune2Drive | 274.9(\(\uparrow\)4.30\(\times\)) | 55.6(\(\uparrow\)1.11\(\times\)) | 15.6% | 15.91 | 0.69(\(\uparrow\)2.49\(\times\)) | 64.81 | ||
| + Ours | 299.1(\(\uparrow\)3.95\(\times\)) | 51.6(\(\uparrow\)1.20\(\times\)) | 15.2% | 15.88 | 0.64(\(\uparrow\)2.69\(\times\)) | 68.30 | ||
4.5pt
| DRA | CCTS (/view) | IRA | ITS (/view) | Total |
|---|---|---|---|---|
| 3.55ms | 26.48ms | 1.11ms | 0.06ms | 163.9ms |
4.3 Ablation study↩︎
In this section, we conduct ablation studies on DriveLM and MAPLM under the 90% pruning setting to evaluate the key design choices of our method.
Two-stage Strategy. We validate the proposed two-stage pruning strategy. Specifically, each stage is independently applied to the shallow layers to meet the target pruning ratio. Additionally, a variant in which stage 2 adopts the same strategy as stage 1 is evaluated, denoted as Stage1*, the results are shown in Figs. 5 (a, d). Applying stage 1 alone effectively removes redundant information while maximizing the semantic coverage of the retained token subset, satisfying the diversity-driven information requirements of shallow layers and outperforming SOTA methods even when used independently. In contrast, applying stage 2 independently yields inferior results, as early layers attend broadly to visual content and have not yet focused on task-relevant regions. Stage1* exhibits accuracy decline because the textual and visual modalities have already been aligned in deeper layers, and the model needs task-relevant information. These results confirm the necessity of the two-stage design, which aligns with the dynamic information requirements of the model during the reasoning process.
Diversity Measure in DRA. We compare different diversity metrics for visual tokens in DRA, including distance to the global frame feature of the view, maximum pairwise distance, and minimum pairwise distance which we adopted. As shown in Figs. 5 (b, e), the choice of diversity metrics has only a minor impact on overall performance. However, the minimum pairwise distance consistently provides the most effective measure of token uniqueness.
Metrics in CCTS. We evaluate the impact of the future contribution metric on the CCTS. As shown in Figs. 5 (c, f), removing the future contribution metric causes the premature loss of potentially task-related information and degrades accuracy. By assessing each token’s contribution throughout the entire reasoning process, our method maximizes the representational capacity of the retained tokens in the shallow layer.
Pruned Layers Selection. We provide a detailed analysis of the pruning layer selection strategies, the results are shown in 6. The placement of stage 1 has an impact on the model, exhibiting a trend of performance degradation as the layer depth increases. One possible explanation is that, as the layer deepens, visual features progressively align with the text modality, resulting in the loss of fine-grained visual information and consequently reducing the reliability of the diversity metric. In contrast, the placement of stage 2 has only a minor effect on accuracy, indicating that the model in deeper layers can recognize task-related regions. Notably, the layer at which stage 2 achieves peak accuracy aligns with the layer exhibiting the highest task-related view recognition accuracy in Sec. 3.2, indicating that our prior analysis accurately captures the model’s ability to identify task-relevant information.
4.4 Qualitative Analysis↩︎
The qualitative comparison results between our method and Prune2Drive are shown in 7. Since most samples in the calibration set focus on front and back views, Prune2Drive allocates fixed, larger token budgets to them. Consequently, it fails to preserve relevant information when the model must attend to objects in the front-left view. In contrast, our method dynamically allocates the token budget across different reasoning stages according to the scene and task instructions. It maximizes the semantic representational capacity of retained tokens in shallow layers to provide comprehensive contextual information for subsequent layers, and precisely preserves task-relevant tokens in deeper layers.
8 shows samples with a fixed scene and varying instructions. Our method captures the resulting shifts in view importance and adaptively reallocates budgets, demonstrating strong generalization capability. More visualizations are in the supplementary materials.
5 Conclusion↩︎
In this paper, we analyze the ability of multi-view VLMs to identify important views and the underlying mechanisms. Results reveal that attention scores can effectively indicate task-related views in deeper layers and the reasoning process of VLMs exhibits dynamic information requirements. Guided by these findings, we propose MVPruner, which dynamically adjusts pruning ratios across views and retains the most critical tokens at each stage, aligning with the model’s dynamic information requirements. Extensive experiments demonstrate that our method achieves a superior efficiency–performance trade-off across diverse benchmarks and exhibits strong generalization.
Acknowledgments↩︎
This work is supported in part by the National Natural Science Foundation of China under Grants U24B20127 and U25B20238, in part by the National Key Research and Development Program of China under Grant 2024YFB4303102, in part by the Chang’an University Graduate Student Research Innovation and Practice Program under Grant 300103267062.
References↩︎
Corresponding author.↩︎