1 Introduction↩︎

Vision-language models (VLMs) achieve strong performance across image and video understanding tasks, but their long visual-token sequences incur substantial inference costs [1]–[4]. High-resolution, multi-crop, and multi-frame inputs can introduce thousands of visual tokens, many of which are redundant for a given query yet must still be processed by the language model. This overhead is especially pronounced during prefilling, where visual tokens dominate sequence length, memory consumption, and attention computation. Visual token pruning therefore provides a direct route to efficient multimodal inference by retaining only the evidence needed for accurate prediction [5]–[11].

The central challenge is not merely estimating token importance, but determining which evidence must be protected before the remaining budget is allocated. As illustrated in Fig. 1, saliency-based methods may retain redundant high-importance regions while overlooking query-specific cues [5]–[9]. Diversity-based methods distribute selections across the visual feature space, but can fragment locally coherent evidence or preserve visually distinct yet query-irrelevant regions [10]. Query-aware methods improve instruction alignment, but relevance-driven selection may still overconcentrate the budget on correlated local evidence. Recent approaches address this limitation by combining relevance and diversity within a unified objective [11]. Under aggressive compression, however, such coupling provides no explicit protection for indispensable evidence: diversity can displace query-critical tokens, while weak or diffuse relevance guidance cannot be compensated by a separate contextual expansion stage.

We argue that query-critical evidence and complementary context play asymmetric roles and should therefore be selected in a fixed order. Evidence directly required by the query, such as a small object, scene text, fine-grained attribute, or spatial relation, is largely non-substitutable: once removed, it cannot be recovered by subsequent selection. Contextual evidence, by contrast, is more substitutable, as multiple informative subsets may provide comparable support for grounding, disambiguation, or holistic reasoning. Effective pruning should therefore first establish a protected relevance anchor and only then expand it with complementary context. This expansion should prioritize tokens that are both informative and non-redundant with respect to the retained anchor, rather than rewarding visual distinctiveness alone.

Based on this principle, we introduce AnchorPrune, a training-free framework for relevance-anchored contextual expansion. AnchorPrune first ranks visual tokens using architecture-appropriate query-conditioned priority scores and constructs a protected relevance anchor. Rather than assigning a fixed fraction of the token budget to relevance, it determines the anchor size adaptively from the novelty profile of relevance-ranked tokens, accommodating both concentrated and distributed query evidence. The remaining budget is then allocated through importance-weighted novelty, expanding the retained set with tokens that are globally informative and complementary to the current selection. By separating anchor construction from contextual expansion, AnchorPrune protects indispensable query evidence while allowing broader context to compensate when relevance guidance is incomplete or spatially diffuse.

AnchorPrune requires neither retraining nor architectural modification and applies to both CLIP-aligned and non-CLIP VLMs using architecture-specific representation spaces. Across image and video benchmarks, it consistently improves the accuracy–efficiency trade-off over representative training-free pruning methods, with the largest gains under aggressive compression. On LLaVA-NeXT-7B, AnchorPrune preserves 97.6% of full-token performance while retaining only 160 of 2,880 visual tokens. These results establish relevance-anchored contextual expansion as an effective principle for efficient multimodal inference.

Our contributions are threefold:

• We identify an ordering limitation in existing visual-token pruning methods: jointly optimizing relevance and diversity can both displace non-substitutable query evidence and leave weak or diffuse relevance guidance without a separate mechanism for contextual compensation.

• We introduce AnchorPrune, a training-free framework that constructs an adaptive protected relevance anchor and expands it through importance-weighted novelty to retain informative, non-redundant context.

• We validate AnchorPrune across CLIP-aligned and non-CLIP image and video VLMs, demonstrating consistent performance preservation and strong accuracy–efficiency trade-offs under severe visual-token compression.

2 Preliminaries↩︎

2.1 Visual Token Pruning in VLMs↩︎

Given a visual input \(I\) and a text query \(q\), a vision encoder produces a sequence of visual tokens \[X=\{x_i\}_{i=1}^{N}, \qquad x_i\in\mathbb{R}^{d}.\] Visual token pruning selects an index set \(S \subseteq \{1,\ldots,N\}\) with \(|S|=K\ll N\) and passes the reduced sequence \(X_S = \{x_i : i \in S\}\) to the language model. The objective is to reduce inference cost while approximately preserving the conditional output distribution: \[p_{\theta}(y\mid X,q) \approx p_{\theta}(y\mid X_S,q)\] Because direct subset optimization is combinatorial, existing methods rely on surrogate criteria such as token importance, query relevance, and feature diversity.

2.2 Relevance, Diversity, and Contextual Expansion↩︎

Let \(\rho_i\) denote a query-conditioned priority score for token \(x_i\). Relevance-driven selection prioritizes instruction-aligned evidence, but may concentrate the retained budget on correlated local tokens. Diversity-driven selection instead favors candidates that contribute information not already represented by the retained set. Given visual features \(h_i\), the novelty of candidate token \(i\) relative to a selected set \(S\) can be measured as \[\Delta(i;S) = \min_{j\in S} \left( 1- \frac{h_i^{\top}h_j}{\lVert h_i\rVert_2\lVert h_j\rVert_2} \right).\]

Recent query-aware methods combine relevance and diversity within a unified subset-selection objective [11]. For example, let \(\widetilde{r}\in\mathbb{R}^{N}\) denote normalized token-relevance scores. A visual-similarity kernel \(L\) can be conditioned on relevance as \[\widetilde{L} = \operatorname{diag}(\widetilde{r}) L \operatorname{diag}(\widetilde{r}), \qquad S^{\star} = \arg\max_{|S|=K} \log\det(\widetilde{L}_S).\] This formulation jointly promotes instruction relevance and subset diversity, but it provides no explicit mechanism for protecting indispensable evidence before diversity influences selection. Moreover, when relevance guidance is weak or spatially diffuse, the unified objective offers no separate stage for compensating through broader contextual selection.

We use contextual expansion to denote the addition of visual tokens that are both informative and complementary to a protected set of query-critical evidence. This notion is stricter than diversity alone: a visually distinct but uninformative region may increase feature dispersion without providing useful contextual support. In AnchorPrune, contextual expansion is therefore performed only after a relevance anchor has been established.

3 Method↩︎

3.1 Overview↩︎

Figure 2 provides an overview of AnchorPrune. AnchorPrune is a training-free visual token pruning framework that establishes a protected relevance anchor before expanding it with complementary visual context. Given visual tokens \(X=\{x_i\}_{i=1}^{N}\), a text query \(q\), and a target token budget \(K\), AnchorPrune constructs the retained set as \[S = S_{\mathrm{rel}}\cup S_{\mathrm{ctx}}, \qquad |S_{\mathrm{rel}}|=K_{\mathrm{rel}}, \qquad |S_{\mathrm{ctx}}|=K_{\mathrm{ctx}}.\] where \(K_{\mathrm{rel}}+K_{\mathrm{ctx}}=K\).

The two subsets are selected sequentially. Stage 1 constructs a protected relevance anchor \(S_{\mathrm{rel}}\) from tokens prioritized by the input instruction. Stage 2 uses the remaining budget to expand this anchor with tokens that are globally informative and non-redundant with respect to the current retained set. Because \(S_{\mathrm{rel}}\) is fixed before contextual expansion, subsequent selections cannot displace query-critical evidence.

3.2 Stage 1: Protected Relevance Anchoring↩︎

Stage 1 assigns each visual token an anchoring-priority score \(s_i\), where a larger value indicates higher priority for inclusion in the protected relevance anchor. The score is computed in an architecture-appropriate representation space in which visual and textual features are comparable.

CLIP-aligned VLMs.↩︎

For models equipped with a paired CLIP vision–language encoder [12], we compute the priority score before the multimodal projector. Let \[v_i = \frac{\phi(x_i)}{\lVert\phi(x_i)\rVert_2}\] denote the normalized visual embedding of token \(i\), and let \(\{t_m\}_{m=1}^{M}\) denote normalized text embeddings produced by the corresponding text encoder. If the instruction exceeds the encoder context length, it is divided into \(M\) segments. We define \[s_i = -\frac{1}{M} \sum_{m=1}^{M} v_i^{\top}t_m . \label{eq:negated95clip95similarity}\tag{1}\]

Equation 1 defines the negated patch–text similarity. Prior analysis found that raw CLIP local similarities can respond more strongly to background than to discriminative foreground regions, and that reversing these maps can improve localization [13]. We therefore use negated similarity as a training-free anchoring-priority signal, without interpreting it as calibrated semantic relevance. The two similarity directions are compared in Supplementary Sec. 8.3.

Non-CLIP VLMs.↩︎

For architectures without a comparable pre-projector vision–language space, we compute the priority score after the multimodal projector. Let \[z_i=g_{\mathrm{mm}}(x_i)\] denote the projected visual token, and let \(\{e_m\}_{m=1}^{M}\) denote the language-model embeddings of the instruction tokens. We define \[s_i = \max_{1\leq m\leq M} \left\langle \frac{z_i}{\lVert z_i\rVert_2}, \frac{e_m}{\lVert e_m\rVert_2} \right\rangle. \label{eq:post95projector95relevance}\tag{2}\] Token-wise maximum matching preserves strong alignment with individual semantic components of the instruction, which can be obscured when all instruction embeddings are averaged into a single vector.

Relevance ranking.↩︎

Using the architecture-specific score \(s_i\), we sort the visual tokens in descending priority: \[\pi=(\pi_1,\ldots,\pi_N), \qquad s_{\pi_1}\geq s_{\pi_2}\geq\cdots\geq s_{\pi_N}.\] For an anchor budget \(K_{\mathrm{rel}}\), the protected relevance anchor is \[S_{\mathrm{rel}} = \{\pi_1,\ldots,\pi_{K_{\mathrm{rel}}}\}. \label{eq:relevance95anchor}\tag{3}\] No diversity penalty is applied in this stage. Consequently, correlated tokens may remain in the anchor when they jointly contain useful local evidence for answering the query.

3.3 Adaptive Anchor Budget↩︎

A fixed relevance ratio is unnecessarily rigid because the structure of query-critical evidence varies across image–query pairs. Some queries depend on a compact local region, while others require evidence distributed across multiple objects, attributes, or textual regions. AnchorPrune therefore determines \(K_{\mathrm{rel}}\) from the novelty profile of the relevance-ranked sequence.

We begin with a minimum anchor size \(K_{\min}\) and impose an upper bound \[K_{\max} = \left\lfloor \frac{K}{2}\right\rfloor ,\] ensuring that at least half of the total budget remains available for contextual expansion.

Let \[A_{\min} = \{\pi_1,\ldots,\pi_{K_{\min}}\}\] denote the initial relevance anchor. Using visual features \(h_i\), we measure the novelty of each subsequent relevance-ranked candidate \(\pi_t\) relative to \(A_{\min}\): \[\Delta(\pi_t;A_{\min}) = \min_{j\in A_{\min}} \left( 1- \frac{h_{\pi_t}^{\top}h_j}{\lVert h_{\pi_t}\rVert_2\lVert h_j\rVert_2} \right) \label{eq:anchor95novelty}\tag{4}\] For \(K_{\min}<t\leq K_{\max}\), we count the number of sufficiently novel candidates encountered after the initial anchor: \[c_t = \sum_{u=K_{\min}+1}^{t} \mathbf{1} \left[ \Delta(\pi_u;A_{\min})>\tau \right],\] where \(\tau\) is a novelty threshold. Given a patience parameter \(P\), the anchor budget is selected as \[K_{\mathrm{rel}} = \begin{cases} t, & \text{if t is the first index satisfying } c_t \geq P, \\[2pt] K_{\max}, & \text{otherwise}. \end{cases} \label{eq:adaptive95krel}\tag{5}\] The remaining context budget is \[K_{\mathrm{ctx}} = K-K_{\mathrm{rel}}.\]

When sufficiently novel candidates appear early in the relevance-ranked sequence, AnchorPrune expands the anchor through the \(P\)-th novelty event. If fewer than \(P\) such events are observed, it conservatively retains the maximum anchor budget \(K_{\max}\).

3.4 Stage 2: Contextual Expansion↩︎

After constructing the protected relevance anchor, AnchorPrune allocates the remaining budget to contextual expansion. Unlike diversity-only sampling, Stage 2 does not favor novelty in isolation. Each candidate must be both globally informative and complementary to the evidence already retained.

Global importance prior.↩︎

We assign each visual token a prior \(p_i\) estimating its contribution to the global visual representation. The prior is instantiated according to the vision-encoder architecture.

For encoders with a dedicated [CLS] token, let \(A_{h,\mathrm{cls},i}\) denote the attention from the [CLS] token to visual token \(i\) at attention head \(h\). We compute \[p_i = \frac{1}{H} \sum_{h=1}^{H} A_{h,\mathrm{cls},i}. \label{eq:cls95prior}\tag{6}\]

For encoders without a dedicated global token, we use the average attention mass received by token \(i\): \[\widetilde{p}_i = \frac{1}{HN} \sum_{h=1}^{H} \sum_{j=1}^{N} A_{h,j,i}.\] If the architecture merges multiple encoder tokens into a single candidate token, let \(G(i)\) denote the corresponding pre-merge token group. The aligned prior is \[p_i = \frac{1}{|G(i)|} \sum_{j\in G(i)} \widetilde{p}_j . \label{eq:merged95prior}\tag{7}\] When no merging is applied, \(G(i)=\{i\}\).

Anchor-conditioned novelty.↩︎

We initialize the retained set with the protected relevance anchor: \[S\leftarrow S_{\mathrm{rel}}.\] For candidate token \(i\notin S\), its novelty relative to the current retained set is \[\Delta(i;S) = \min_{j\in S} \left( 1- \frac{h_i^{\top}h_j}{\lVert h_i\rVert_2\lVert h_j\rVert_2} \right) . \label{eq:context95novelty}\tag{8}\] Because \(S\) is initialized with \(S_{\mathrm{rel}}\), novelty is measured relative to both the protected anchor and the contextual tokens selected in preceding iterations.

Importance-weighted expansion.↩︎

At each iteration, AnchorPrune selects \[i^{\star} = \arg\max_{i\notin S} p_i\,\Delta(i;S), \qquad S\leftarrow S\cup\{i^{\star}\}. \label{eq:importance95weighted95novelty}\tag{9}\]

until \(|S|=K\). The resulting context subset is \[S_{\mathrm{ctx}} = S\setminus S_{\mathrm{rel}}.\]

The multiplicative objective in Eq. 9 suppresses both visually distinct but uninformative tokens and globally important but redundant ones. Each selected token must therefore be simultaneously informative and complementary to the current retained set, producing contextual expansion rather than feature dispersion alone. The selected tokens are restored to their native visual order and passed to the unchanged language model without retraining or architectural modification.

4 Experiments↩︎

We evaluate AnchorPrune in terms of performance preservation under aggressive visual-token compression, generalization across heterogeneous image and video VLM architectures, and the effectiveness of relevance-anchored contextual expansion. Our evaluation spans two CLIP-aligned image VLMs with substantially different visual-sequence lengths, a non-CLIP image VLM, and a video VLM. Detailed implementation settings, video breakdowns, and additional ablations are provided in Supplementary Secs. 6, 7, and 8, respectively.

4.1 Experimental Setup↩︎

Models and evaluation.↩︎

We evaluate AnchorPrune on LLaVA-1.5-7B [1], LLaVA-NeXT-7B [2], Qwen2.5-VL-7B [3], and LLaVA-Video-7B [4].

For LLaVA-1.5-7B, we retain 128, 64, or 32 tokens from 576 visual tokens. For LLaVA-NeXT-7B, we retain 640, 320, or 160 tokens from 2,880 visual tokens. Both models are evaluated on VQAv2 [14], TextVQA [15], GQA [16], ScienceQA-IMG [17], MME [18], POPE [19], MMBench-EN/CN [20], and MM-Vet [21].

For Qwen2.5-VL-7B, we retain 256, 128, or 64 tokens from 1,296 visual tokens and evaluate on MME, TextVQA, DocVQA [22], AI2D [23], MMMU [24], and MMBench-EN/CN.

For LLaVA-Video-7B, we evaluate 16-frame inputs and retain 1,024 or 512 tokens from 2,704 visual tokens on Video-MME [25], EgoSchema [26], and TempCompass [27].

Results for all evaluated model–budget settings are reported in Tables 1, 2, 3, and 4. We additionally conduct a controlled Stage-2 ablation on LLaVA-1.5-7B.

Baselines and protocol.↩︎

We compare against representative training-free methods based on attention or saliency, progressive token reduction, token merging, diversity, and unified relevance–diversity selection [5]–[11]. All methods use matched model checkpoints, input processing, decoding settings, benchmark protocols, and retained-token budgets. We use lmms-eval [28] whenever supported. Architecture-specific implementation details, hyperparameters, and hardware configurations are provided in Supplementary Sec. 6.

Relative retained performance.↩︎

To aggregate benchmarks with different metric scales, we report the average performance retained relative to the unpruned backbone: \[\mathrm{Rel.}(m) = \frac{1}{|\mathcal{B}|} \sum_{b\in\mathcal{B}} \frac{\mathrm{Score}_{b}(m)}{\mathrm{Score}_{b}(\mathrm{Full})} \times 100, \label{eq:relative95performance}\tag{10}\] where \(\mathcal{B}\) denotes the set of metrics reported in the corresponding table.

3.6pt

4.2 Main Results↩︎

CLIP-aligned image VLM.↩︎

Table 1 reports results on LLaVA-1.5-7B, whose visual representation contains 576 tokens. This compact setting provides a demanding test because the original sequence contains substantially less redundancy than high-resolution multi-crop representations.

AnchorPrune achieves the highest aggregate retention at all three budgets. It retains 98.7% and 96.2% of the full-token baseline with 128 and 64 tokens, respectively. Under the most aggressive setting, it retains 93.9% using only 32 visual tokens, exceeding the second-best method by 1.3 percentage points.

The gains are distributed across benchmarks rather than being driven by a single metric. At 32 tokens, AnchorPrune is particularly strong on MME, TextVQA, MMBench-CN, and MM-Vet, showing that relevance-anchored contextual expansion remains effective even when the original visual sequence is comparatively compact.

3.0pt

Scaling to high-resolution visual sequences.↩︎

Table 2 evaluates AnchorPrune on LLaVA-NeXT-7B, whose high-resolution multi-crop representation contains 2,880 visual tokens. This setting tests whether the same selection principle scales to substantially longer and more redundant visual sequences.

AnchorPrune achieves the highest retained performance at every evaluated budget. With 640 tokens, it retains 99.4% of the full-token baseline, outperforming the second-best method by 2.7 percentage points in Rel. At 320 tokens, AnchorPrune achieves 98.3%, compared with 95.9% for the strongest competing method.

The advantage becomes larger under the most aggressive setting. With only 160 of 2,880 tokens retained, AnchorPrune preserves 97.6% of the full-token baseline, exceeding the second-best result by 4.7 percentage points. It also remains close to the unpruned model on TextVQA, MME, POPE, and both MMBench variants. Together with the LLaVA-1.5 results, this demonstrates that relevance-anchored contextual expansion is effective across both compact and high-resolution CLIP-aligned visual sequences.

3.0pt

Generalization to a non-CLIP architecture.↩︎

Table 3 evaluates AnchorPrune on Qwen2.5-VL-7B. Unlike the LLaVA models, Qwen2.5-VL does not expose the same paired pre-projector CLIP representation. AnchorPrune therefore computes its Stage-1 priority in the post-projector embedding space while preserving the same protected-anchor design.

AnchorPrune achieves the highest Rel. at all three budgets. At 256 tokens, it retains 93.5% of the full-token baseline, improving over the second-best result by 1.6 percentage points. At 128 tokens, AnchorPrune obtains 88.1%, compared with 86.6% for the strongest baseline. Even when only 64 of 1,296 tokens remain, AnchorPrune achieves the best overall result at 80.8%.

The improvements are consistent across general, text-oriented, and document-oriented benchmarks, indicating that the anchoring principle transfers beyond CLIP-aligned architectures. These results demonstrate that AnchorPrune is not tied to a specific visual encoder or relevance representation.

2.4pt

Video Generalization.↩︎

Table 4 evaluates AnchorPrune on LLaVA-Video-7B under 16-frame inputs. AnchorPrune achieves the highest aggregate retained performance at both budgets, preserving 98.3% and 94.1% of the full-token baseline with 1,024 and 512 tokens, respectively. This result indicates that relevance-anchored contextual expansion extends to temporally structured visual inputs.

Overall trend.↩︎

Across compact and high-resolution CLIP-aligned image VLMs, a non-CLIP image VLM, and a video VLM, AnchorPrune consistently achieves the strongest retained performance. Its advantage generally becomes clearer as the token budget decreases, supporting the principle that query-critical evidence should be protected before complementary context is expanded.

4.3 Efficiency Analysis↩︎

Finally, we evaluate the practical efficiency of AnchorPrune, including token-selection overhead. To summarize the joint trade-off between prefill latency and resident memory in a single quantity, we define an Efficiency Score as \[\mathrm{Efficiency\;Score} = \mathrm{Prefill\;Speedup} \times \frac{\mathrm{Memory}_{\mathrm{Full}}}{\mathrm{Memory}_{\mathrm{Method}}}, \label{eq:efficiency95score}\tag{11}\] where \(\mathrm{Prefill\;Speedup}\) denotes the speedup relative to the unpruned model, \(\mathrm{Memory}_{\mathrm{Full}}\) is the resident GPU memory usage of the full-token baseline, and \(\mathrm{Memory}_{\mathrm{Method}}\) is the resident GPU memory usage of the pruned method.

Figure 3 plots this score against MME performance under a matched budget of 32 retained visual tokens. AnchorPrune achieves the highest MME score among the evaluated pruning methods while maintaining a competitive efficiency score of \(5.24\). VisionZip and DivPrune obtain slightly higher efficiency scores of \(5.28\) and \(5.41\), respectively, due to marginally lower resident memory, but their MME scores are substantially lower. CDPruner reaches a similar prefill speedup but has a lower efficiency score of \(4.68\) because of its larger resident memory footprint. These results show that AnchorPrune provides the strongest accuracy–efficiency operating point rather than improving accuracy through a costly selection mechanism.

This result is consistent with its lightweight design: Stage 1 uses simple anchoring-priority scoring, while Stage 2 performs greedy importance-weighted expansion without requiring kernel-based subset optimization. Detailed FLOPs, latency, memory, and performance measurements are provided in Supplementary Sec. 9.

4.4 Ablation Study↩︎

3.0pt

Stage-2 expansion strategy.↩︎

We isolate the contribution of the Stage-2 selection criterion in Table 5. All variants share the same protected Stage-1 relevance anchor and differ only in the criterion used to allocate the remaining token budget. We compare diversity-only selection, additive relevance–diversity coupling, multiplicative relevance–diversity coupling, and the proposed importance-weighted contextual expansion rule. For this ablation, Rel. is computed over MME [18], ChartQA [29], DocVQA [22], TextVQA [15], and MMBench-CN [20], as reported in the table.

At 64 retained tokens, diversity-only selection obtains 85.4% Rel. The additive and multiplicative relevance–diversity formulations improve Rel. to 87.3% and 87.2%, respectively. In contrast, importance-weighted contextual expansion achieves 91.0%, outperforming the strongest alternative by 3.7 percentage points.

The same conclusion holds under stronger compression. At 32 retained tokens, the proposed rule achieves 85.1% Rel., compared with 81.1% for the strongest alternative and 79.4% for diversity-only selection. The gains are especially pronounced on MME, TextVQA, and MMBench-CN.

These results show that contextual expansion is not equivalent to diversity maximization. Diversity-only selection may allocate capacity to visually distinct but weakly informative regions, while direct relevance–diversity coupling permits strength in one criterion to compensate for weakness in the other. Starting from the same protected relevance anchor, importance-weighted contextual expansion selects tokens that are simultaneously informative and complementary to the evidence already retained.

5 Conclusion↩︎

We presented AnchorPrune, a training-free visual token pruning framework that constructs an adaptive protected relevance anchor before expanding it through importance-weighted novelty. This sequential design preserves query-critical evidence while adding informative, non-redundant context. Across CLIP-aligned and non-CLIP image and video VLMs, AnchorPrune consistently improves performance preservation under matched token budgets, with the largest gains under severe compression. On LLaVA-NeXT-7B, it preserves 97.6% of full-token performance using only 160 of 2,880 visual tokens. These results establish relevance-anchored contextual expansion as an effective principle for efficient multimodal inference.

Acknowledgements.↩︎

This research was supported by the Electronics and Telecommunications Research Institute (ETRI) Grant funded by the Korean Government (Fundamental Technology Research for Human-Centric Autonomous Intelligent Systems) under Grant 25ZB1200.

6 Implementation Details and Experimental Protocol↩︎

This section specifies the implementation choices and experimental protocol used throughout our evaluation. We describe the model-specific candidate-token sequences, pruning locations, architecture-dependent signals, adaptive anchor-budget configurations, controlled comparison protocol, and hardware and efficiency-measurement settings.

6.1 Model-Specific Visual Tokenization↩︎

Table 6 summarizes the candidate visual-token sequence processed by AnchorPrune for each backbone. To ensure a controlled comparison, all pruning methods evaluated on the same backbone use an identical fixed input resolution, visual preprocessing pipeline, tokenization procedure, and candidate sequence. This prevents changes in resolution or token granularity from confounding the effect of the pruning method itself.

4.0pt

For LLaVA-NeXT-7B [2], the five visual units form the model’s high-resolution multi-crop representation, with each unit contributing a \(24\times24\) spatial grid. For LLaVA-Video-7B [4], we uniformly sample 16 frames, process each frame at \(384\times384\), and represent it using a \(13\times13\) post-pooling grid before concatenating the frame-level sequences in temporal order.

6.2 Pruning Location and Native Token Ordering↩︎

AnchorPrune is applied after visual feature extraction and before the retained visual sequence is consumed by the language model. It leaves the parameters and layer structure of the vision encoder, multimodal projector, and language model unchanged.

For LLaVA-1.5-7B and LLaVA-NeXT-7B, pruning operates on the visual tokens associated with the input image or each high-resolution visual unit. For Qwen2.5-VL-7B, selection is applied after the model-specific spatial processing and token merging that produce the visual sequence supplied to the language model. For LLaVA-Video-7B, post-pooling tokens from the 16 sampled frames are concatenated into a frame-major sequence before selection.

After pruning, retained tokens are restored to their native order before language-model inference. Image models preserve raster-scan order within each visual unit and the original ordering across multiple units, whereas LLaVA-Video-7B preserves temporal frame order first and raster-scan order within each frame. This operation changes only the sequence order of the selected tokens and does not alter subset membership. Section 8.1 isolates its effect on spatially sensitive benchmarks.

6.3 Architecture-Specific Signal Extraction↩︎

The main paper defines the anchoring-priority score, novelty measure, and global-importance prior. This section details the architecture-specific representation spaces and attention signals used to instantiate these quantities.

LLaVA-1.5-7B and LLaVA-NeXT-7B.↩︎

For LLaVA-1.5-7B [1] and LLaVA-NeXT-7B [2], Stage 1 matching and novelty use normalized pre-projector visual features in the paired CLIP representation space [12], with instruction embeddings produced by the corresponding CLIP text encoder. Instructions that exceed the supported context length are partitioned into valid segments, and the resulting patch–text scores are averaged across segments. The Stage 2 global-importance prior is obtained from a late vision-transformer layer by averaging [CLS]-to-patch attention across heads. Section 8.3 compares raw and negated CLIP patch–text similarity.

Qwen2.5-VL-7B.↩︎

Because Qwen2.5-VL-7B [3] does not expose a paired pre-projector vision–language space, Stage 1 matching is performed after the multimodal projector. Projected visual tokens and instruction-token embeddings are normalized, and each visual token is assigned its maximum cosine similarity over instruction tokens. This preserves alignment with individual semantic components that may be diluted by averaging the instruction into a single representation. Section 8.4 compares maximum matching with mean aggregation.

Qwen2.5-VL-7B spatially merges vision-encoder tokens before constructing the final candidate sequence, so the attention-derived importance prior is aligned with the post-merge candidates. We first average the attention mass received by each pre-merge token across source positions and heads, and then average these values within the pre-merge group associated with each final candidate. Projected candidate features are used for both adaptive-anchor novelty and Stage 2 contextual novelty.

LLaVA-Video-7B.↩︎

For LLaVA-Video-7B [4], the CLIP-aligned implementation is applied independently to each sampled frame, after which frame-level candidates are concatenated into a temporally ordered sequence. Selection can therefore remove both spatial redundancy within individual frames and repeated evidence across adjacent frames. The retained sequence is restored to frame-major temporal order before language-model inference.

6.4 Adaptive Anchor-Budget Configuration↩︎

We use a shared novelty threshold \(\tau=0.2\) and patience parameter \(P=3\) across all models and benchmarks. For a visual unit with assigned retained-token budget \(K_u\), the maximum anchor size is \(K_{\max,u}=\lfloor K_u/2\rfloor\). For single-unit image models, \(K_u=K\); for LLaVA-NeXT-7B and LLaVA-Video-7B, \(K_u\) denotes the budget assigned to each image unit or frame. Neither \(\tau\), \(P\), nor \(K_{\min}\) is tuned per benchmark. Table 7 reports the complete \(K_{\min}\) schedule. A single configuration is used for every benchmark evaluated at a given backbone and retained-token budget.

5pt

For LLaVA-NeXT-7B and LLaVA-Video-7B, the protected relevance anchor is constructed independently within each visual unit or frame using the corresponding per-unit \(K_{\min}\) schedule and \(K_{\max,u}\). The resulting unit-level anchors are concatenated to form the global protected anchor, after which the remaining total budget is allocated globally by Stage 2 over the full candidate sequence. Retained tokens are finally restored to their native unit/frame-major order.

Fallback behavior.↩︎

Within each visual unit, the adaptive procedure terminates before \(K_{\max,u}\) only when the relevance-ranked sequence contains the required \(P\) novelty events. Otherwise, it conservatively uses \(K_{\max,u}\). Because each unit-level anchor is bounded by half of its assigned budget, at least half of the total retained-token budget remains available for global contextual expansion.

6.5 Evaluation Protocol↩︎

For each backbone and retained-token budget, all methods use the same checkpoint, input resolution, visual preprocessing, tokenization procedure, prompt template, decoding configuration, benchmark split, and evaluation pipeline. Only the visual-token selection rule is changed. Fixing the complete preprocessing and tokenization pipeline is essential because changes in resolution or spatial merging alter both the number and granularity of candidate tokens, thereby confounding the effect of the pruning method itself.

We use lmms-eval [28] whenever the corresponding model–benchmark pair is supported and otherwise follow the official benchmark evaluator. Baseline implementations follow their published selection rules and recommended configurations. A method is evaluated on a backbone only when its required architectural signals and candidate-token structure are available without modifying its underlying selection objective. This accounts for the smaller comparison sets reported for Qwen2.5-VL-7B and LLaVA-Video-7B. All comparisons use exactly matched final retained-token budgets.

Benchmark definitions and the computation of Rel. are provided in the main paper. In the detailed video evaluation, Video-MME [25] with and without subtitles are treated as separate aggregate metrics, EgoSchema [26] uses the official 500-example subset, and TempCompass [27] contributes its average score.

6.6 Hardware and Efficiency Measurement↩︎

Image experiments on LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B use four NVIDIA RTX A5000 GPUs, whereas video experiments on LLaVA-Video-7B use two NVIDIA RTX 4090 GPUs. Within each reported comparison, all methods are evaluated in the same hardware and software environment.

The detailed efficiency experiment in Sec. 9 uses LLaVA-1.5-7B with 32 retained visual tokens. Prefill latency includes visual-token scoring and selection overhead in addition to the reduced model forward pass, while decode latency is reported per generated token. Estimated FLOPs, resident GPU memory, and MME performance are measured under the same inference setting.

7 Detailed Video Results↩︎

The main paper reports aggregate video results. Table 8 provides the complete Video-MME [25] breakdown by duration and subtitle condition, together with EgoSchema [26] and TempCompass [27]. We compare AnchorPrune against FastV [5], DivPrune [10], and CDPruner [11].

3.3pt

At 1,024 retained tokens, AnchorPrune achieves the highest aggregate retention, preserving 98.3% of full-token performance. Its advantage is especially pronounced on EgoSchema, where it obtains 55.0 compared with 52.6–53.2 for the competing methods. At 512 tokens, AnchorPrune again yields the highest Rel. at 94.1%, together with the strongest EgoSchema and TempCompass scores among the pruning methods.

No method dominates every Video-MME duration and subtitle condition. AnchorPrune nevertheless provides the strongest overall balance across short-, medium-, and long-duration videos and across the complementary video benchmarks. This pattern supports relevance-anchored contextual expansion for temporally structured inputs, where repeated evidence across adjacent frames creates substantial redundancy.

8 Additional Ablation Studies↩︎

8.1 Effect of Retained-Token Ordering↩︎

AnchorPrune restores retained tokens to their native spatial or temporal order before language-model inference. Table 9 isolates this operation by evaluating an identical selected subset under raw selection order and native raster-scan order on MME-OCR [18], OCRBench [30], and TextVQA [15].

8pt

Restoring raster-scan order improves all three benchmarks, with the largest gains on MME-OCR and OCRBench, which are especially sensitive to spatial organization and local reading order. Because subset membership is held fixed, the improvement is attributable solely to presenting the retained evidence in the positional progression expected by the pretrained model.

8.2 Sensitivity to Adaptive Anchor-Budget Hyperparameters↩︎

Table ¿tbl:tab:adaptive95sensitivity? evaluates sensitivity to the novelty threshold \(\tau\) and patience parameter \(P\) on MME [18]. Each experiment varies only the parameter under study while holding all remaining settings fixed.

4.2pt

Performance is stable across the tested threshold range. The shared setting \(\tau=0.2\) is best on LLaVA-1.5-7B and Qwen2.5-VL-7B, while \(\tau=0.3\) is moderately better on LLaVA-NeXT-7B. We retain \(\tau=0.2\) as a single architecture-independent setting rather than selecting a backbone-specific optimum.

The patience study shows a clearer pattern: \(P=3\) achieves the highest MME score on all three architectures. A smaller value can terminate anchor construction before sufficiently distributed evidence is retained, whereas a larger value can delay the transition to contextual expansion. These results support \(P=3\) as a stable shared configuration.

8.3 Stage-1 CLIP Similarity Direction↩︎

For CLIP-aligned VLMs [12], AnchorPrune uses negated patch–text similarity as the Stage 1 anchoring-priority signal. This choice is motivated by prior analysis of CLIP’s dense image–text similarity maps, which found that raw local similarities can respond more strongly to background regions than to discriminative foreground content and that reversing the similarity map can improve localization [13].

Figure 4 visualizes the spatial priority induced by raw and negated similarity. Raw similarity can emphasize broad background or non-discriminative regions, whereas negated similarity assigns greater priority to localized regions associated with the queried evidence in these examples.

Table 10 evaluates the downstream effect of the similarity direction under identical adaptive anchor-budget and Stage 2 contextual-expansion settings. Negated similarity consistently improves MME and POPE at both retained-token budgets, while TextVQA remains comparatively stable. At 32 retained tokens, it improves MME from 1245.4 to 1394.7 and POPE from 74.2 to 85.3. These results indicate that the gain is not attributable to a uniform rescaling of token scores; rather, the negated direction induces a Stage 1 ranking that better preserves evidence required for broad visual perception and hallucination-sensitive evaluation.

Consistent with ECLIP [13], we use negated similarity strictly as an empirical anchoring-priority signal for the evaluated CLIP-aligned models. We do not interpret it as calibrated semantic relevance or claim that raw CLIP similarity is universally inverted. The qualitative and quantitative results instead indicate that the negated direction is better suited to constructing the protected Stage 1 relevance anchor.

7pt

8.4 Post-Projector Relevance Matching↩︎

For Qwen2.5-VL-7B [3], Stage 1 matching is computed in the post-projector language-model input space. Table 11 compares token-wise maximum matching with mean aggregation of the instruction-token embeddings.

4.2pt

Token-wise maximum matching improves all seven benchmarks. Although the individual gains are moderate, their consistency indicates that averaging instruction-token embeddings can obscure alignment with specific semantic components. Maximum matching preserves each visual token’s strongest correspondence to any instruction token and is therefore better suited to Stage 1 ranking in the post-projector representation space.

9 Detailed Efficiency Analysis↩︎

Table 12 reports the raw measurements underlying the efficiency–performance comparison in the main paper. We compare against FastV [5], PyramidDrop [6], SparseVLM [7], VisionZip [9], DivPrune [10], and CDPruner [11]. All pruning methods retain 32 visual tokens, and the reported prefill latency includes the complete token-scoring and subset-selection procedure.

3.1pt

AnchorPrune reduces estimated computation from 8.94 to 1.67 TFLOPs and prefill latency from 144.6 to 28.2 ms, corresponding to an 81.3% reduction in FLOPs and a \(5.13\times\) prefill speedup relative to the full-token baseline. It also matches the lowest measured decode latency and uses 14.2 GB of resident GPU memory while including the full selection procedure.

CDPruner attains the same measured prefill latency but uses 15.9 GB of resident GPU memory, 1.7 GB more than AnchorPrune and 1.4 GB more than the full-token baseline. Because these values reflect end-to-end resident memory rather than operator-level peak allocation, we do not attribute the difference to a specific algorithmic component. The measurement nevertheless shows that comparable latency and FLOPs do not necessarily imply comparable end-to-end memory behavior.

VisionZip and DivPrune use slightly less resident memory than AnchorPrune, but obtain substantially lower MME scores. PyramidDrop and SparseVLM also remain below AnchorPrune in task performance under the same evaluation setting. CDPruner improves performance over these alternatives but incurs the largest memory footprint among the evaluated pruning methods. AnchorPrune therefore provides the strongest overall operating point in this setting: it matches the lowest measured FLOPs, prefill latency, and decode latency; remains close to the baseline memory footprint; and achieves the highest MME score among all pruning methods.

These results show that AnchorPrune’s performance gains do not depend on a computationally or memory-intensive selection mechanism. Its relevance-anchored contextual expansion preserves task performance while maintaining practical end-to-end inference efficiency.

References↩︎

---

1. : corresponding author↩︎

2. : corresponding author↩︎