1 Introduction↩︎

Modern vision-language models (VLMs) suffer substantial inference costs due to the large number of visual tokens introduced by high-resolution images and videos [1]. To improve efficiency, recent studies have explored various approaches, including token compression, adaptive computation, and visual-token pruning [2]–[4]. Among them, visual-token pruning offers a structurally direct form of efficiency improvement, as it reduces sequence length without introducing additional latent representations or auxiliary decoding stages[5]. However, effective pruning requires a reliable token importance scorer, which is non-trivial in practice.

Existing visual-token pruning methods can be broadly divided into training-free and training-based approaches depending on whether the token importance scorer is learned. Training-free methods use heuristic signals from pretrained backbones (e.g., attention or feature redundancy) to estimate token importance [7]–[10], requiring no additional training and enabling direct inference-time deployment, but they are often misaligned with downstream tasks and lack adaptability across models [11], [12]. In contrast, training-based methods [13]–[15] learn a scorer to estimate token importance. To enable optimization, they rely on relaxations such as Gumbel-Softmax (GS) or straight-through estimators (STE), which replace the discrete selection process with a differentiable surrogate. However, this surrogate does not match the true discrete pruning operation, so the training signal is based on surrogate gradients rather than the real pruning objective.

As shown in 1 (a), existing pipelines convert continuous importance scores into binary keep-or-drop decisions under a fixed budget, making the pruning operation inherently non-differentiable. GS and STE are used to approximate gradients through this discrete process, but they optimize a relaxed continuous surrogate rather than the original discrete selection problem [16], [17]. This mismatch between the surrogate objective and the true pruning objective leads to unstable optimization dynamics [18]. As illustrated in 1 (c), the resulting gradients exhibit high variance across training iterations, since small changes in importance scores can induce abrupt changes in the selected token subset. This effect becomes more pronounced as model size increases, where more tokens lie near the selection boundary.

In contrast, our method, as illustrated in 1 (b), avoids optimizing through discrete relaxation. Instead, it maintains a continuous importance formulation throughout training and directly modulates token information flow under a fixed budget constraint. This yields a smoother optimization landscape and significantly improves gradient consistency. Importantly, our formulation remains fully differentiable throughout training, leading to substantially higher gradient alignment (28.4× on DeiT-Base).

In this paper, we propose DiffPrune, a fully differentiable visual-token pruning framework for VLMs. During training, every operator on the path from scorer logits to language-model loss has a backward gradient given by the analytic derivative of the same continuous function used in the forward pass. The gradient path uses no surrogate gradients, argmax, hard selection, or straight-through estimators. During training, DiffPrune does not make binary keep-or-drop decisions, but instead changes the amount of information carried by each visual token in a continuous way. A Soft Top-\(K\) head polarises the scorer logits into continuous weights and constrains their total budget to match hard top-\(K\). The Information Throttler has two components. First, a variance-preserving (VP) noise injector uses these weights to modulate each token. Second, a trainable denoiser maps the noised tokens back to the representation space expected by the downstream LLM. At inference, Soft Top-\(K\) is replaced by hard top-\(K\), the denoiser is removed, and only the scorer is added to the inference path. The base VLM remains frozen throughout.

Contributions can be concluded as follows.

• We show that the limitation of surrogate-gradient pruning lies at the operator level rather than the scorer level, and propose DiffPrune as a fully differentiable framework for visual-token pruning.

• We introduce VP-Noise, a diffusion-inspired Throttler that interpolates each token with Gaussian noise according to its retention weight, suppressing token content without relying on a discrete keep-or-drop operation.

• In controlled DeiT probes, DiffPrune reaches up to \(28.4\times\) higher cross-batch gradient-direction coherence than Gumbel-Softmax; across LLaVA-1.5-7B, LLaVA-NEXT-7B, and Qwen2.5-VL-7B evaluations, it preserves high task performance under aggressive pruning while adding only \(0.69\) ms of inference overhead.

2 Related Work↩︎

2.1 Training-Free Visual Token Compression↩︎

Training-free compression keeps the VLM backbone fixed and score tokens using signals already present in the model. We organize this line by the source of the token score [19], [20]. Attention-based methods use saliency from the vision tower or the language decoder [7], [8], [21]–[25]. Redundancy-based methods remove or merge visually similar features [9], [10], [26]–[28]. Coverage-based methods keep tokens that are diverse or spatially well spread [11], [29]–[32]. These methods are strong plug-in baselines because they do not train a new selector. While strong as plug-in baselines, these methods do not adapt their compression rules to downstream language-model loss. Learned pruning instread must convert continuous scores into discrete keep-or-drop decisions while enabling gradient flow.

2.2 Learning-based- Token Selection↩︎

Learned pruning must train the steps. A common solution is surrogate-gradient pruning. The forward pass uses a discrete or nearly discrete mask, while the backward pass differentiates a smooth relaxation. This pattern appears in semi-structured weight sparsity, KV-cache eviction, and text-side token pruning [33]–[36]. These methods use different relaxations, including Gumbel-Softmax with STE, differentiable sorting, and hard-concrete gates, and compress different objects. The core issue remains: the score-to-mask function in the forward pass is not the one used for backpropagation.

Learned visual-token pruning follows the same pattern. DynamicViT [37] samples token survival with Gumbel-Softmax and passes gradients through the soft relaxation. ATP-LLaVA [38] uses a temperature-sharpened sigmoid-threshold mask. Dynamic-LLaVA [39] and LightVLA [40] use binary or argmax-like selections with Gumbel-Softmax-style backward paths. These methods differ in scorer architecture, pruning location, and budget control, but are similar in how continuous scores become selected tokens.

Recent variants point to the same bottleneck. Shiva-DiT [41] replaces the softmax surrogate with a residual-aware STE in diffusion transformers. TwigVLM++ [42] trains a pruning head through distillation and policy-gradient reinforcement learning. These routes either keep a discrete forward selection with a surrogate backward path, or avoid analytic differentiation through selection. DiffPrune instead uses a continuous forward operator, whose backward gradients are the analytic derivatives of the same function. 3 formalises this contrast at the operator and loss-landscape levels.

3 Pilot Studies↩︎

3.1 Limitations of Gumbel-Softmax Pruning↩︎

The main problem of surrogate-gradient pruning lies in the discrete selection operator itself, rather than only in the scorer design. Let the logits output by the scorer be \(s \in \mathbb{R}^{N}\) and Gumbel perturbation be \(g\). In the forward pass, the model generates a hard mask via top-\(K\); in the backward pass, the gradient is usually passed directly from the mask \(m\) back to the logits \(s\): \[\begin{align} \mathrm{forward:}\quad & m = \mathrm{topK}(s + g) \in \{0,1\}^{N}, \nonumber\\ \mathrm{backward:}\quad & \frac{\partial L}{\partial s} \leftarrow \frac{\partial L}{\partial m}. \label{eq:gs-ste} \end{align}\tag{1}\] The forward computation and the backward gradient here do not correspond to the same differentiable function. The top-\(K\) selection actually executed in the forward pass is piecewise constant with respect to \(s\): as long as the ordering of the elements does not change, the retained token set remains unchanged, and therefore its derivative is almost everywhere zero [43], [44]. Only when a score crosses the \(K\)-th order statistic and triggers an index swap does the output mask change discontinuously. In contrast, the backward pass does not use the true Jacobian of this hard selection operator. Instead, it approximates that Jacobian with the Jacobian of the identity map, directly passing \(\partial L/\partial m\) to \(s\). Therefore, the gradient is not from the actual forward computation but a surrogate estimate, introducing a systematic gradient bias [45] rather than unbiased random noise.

3.2 A Gradient-Continuous Alternative↩︎

Based on the above analysis, we propose an improved scheme: route every token through one analytic function whose value is the forward and whose Jacobian is the backward. Rather than ask whether token \(i\) is kept, we ask how much of its information is admitted, encode the answer as \(\alpha_i \in [0,1]\), and apply a gradient-continuous Throttler \(T(x_i;\alpha_i)\) to every token, with \(\alpha\) supplied by a Soft Top-\(K\) head under \(\sum_i \alpha_i \approx K\); unlike the soft companion in 1 , Soft Top-\(K\) is the operator the forward executes. The probe below instantiates \(T\) as a per-token scale composed with variance-preserving Gaussian mixing, with the design unpacked in 4.

We visualize the scorer loss landscape with the filter-normalized two-dimensional slice of Li et al. [46]. The probe inserts a two-layer MLP scorer with roughly \(14\)M parameters between the patch embedding layer and the frozen DeiT-base encoder. The patch embedding layer outputs \(N=196\) patch tokens, and the pruning ratio removes \(70\%\) of them, i.e., \(K=58\). We compare Gumbel-softmax, which sends the hard top-\(K\) tokens to the encoder with an STE backward pass, against DiffPrune, which applies Soft Top-\(K\) scores through a continuous token throttler.

2 (a) shows that DiffPrune yields a smooth loss surface with gradual contour changes. In contrast, 2 (b) exhibits more irregular loss contours with sharper local variations. 2 (c) makes this difference more explicit by plotting the stepwise loss variation along the \(d_2\) axis, measured as \(\lvert L(\beta_{i+1}) - L(\beta_i)\rvert\). The mean stepwise loss variation of Gumbel-softmax is roughly \(24\times\) that of DiffPrune. These results suggest that the discontinuity introduced by hard token selection and STE is not only a backward-pass mismatch, but also appears directly in the scorer optimization landscape. By replacing discrete selection with continuous throttling during training, DiffPrune provides a smoother objective for optimizing the pruning scorer.

4 Methodology↩︎

4.1 Overview↩︎

DiffPrune builds a fully differentiable framework for visual-token pruning in VLMs. 3 gives the overview: DiffPrune trains a token scorer through a continuous score-to-loss path and deploys it as a hard top-\(K\) pruner. Given an image \(\mathbf{I}\), the frozen vision encoder produces visual tokens \(\mathbf{X}^v=\mathcal{E}_v(\mathbf{I})\in\mathbb{R}^{N\times d_v}\). The framework consists of a Scorer \(\mathcal{S}_\theta\), a budgeted Soft Top-\(K\) map \(\Phi_K\), a token-wise Throttler \(T\), and a train-only Denoiser \(\mathcal{D}_\phi\). The base VLM is frozen; only \(\theta\) and \(\phi\) are optimized.

Let \(\mathbf{W}\) denote the text tokens, \(p_\Theta\) the frozen language model, and \(\mathbf{Z}\) the multimodal sequence consumed by it. Training is written as \[\begin{align} \min_{\theta,\phi}\quad & -\sum_t \log p_\Theta\!\left(y_t^* \mid y_{<t}^*, \mathbf{Z}\right),\\ \mathbf{Z} &= [\,\mathbf{P}_{v\to\ell}(\bar{\mathbf{X}}^v);\,\mathcal{E}_t(\mathbf{W})\,], \end{align} \label{eq:train-fwd}\tag{2}\] where \(\bar{\mathbf{X}}^v=\mathcal{D}_\phi(T(\mathbf{X}^v;\boldsymbol{\alpha}))\) and \(\boldsymbol{\alpha}=\Phi_K(\mathcal{S}_\theta(\mathbf{X}^v)/\tau)\) with \(\sum_i\alpha_i=K\). Here \(K\) is the token budget, i.e., the target number of visual tokens retained by the deployed pruner; during training, it is represented as the total retention mass \(\sum_i\alpha_i\). Hard token selection is absent from 2 ; the scorer is trained through the continuous operators that are actually executed in the forward pass.

At inference, DiffPrune uses the trained scorer as a ranking function and gathers the original top-\(K\) visual tokens: \[\hat{\mathbf{X}}^v = \operatorname{TopK}\!\left(\mathbf{X}^v;\mathcal{S}_\theta(\mathbf{X}^v),K\right) \in \mathbb{R}^{K\times d_v}. \label{eq:infer-fwd}\tag{3}\] Here \(\operatorname{TopK}(\mathbf{X};\mathbf{s},K)\) denotes score-indexed token gathering. The retained tokens keep their original order and position indices. The Throttler and Denoiser are removed from the deployed graph, leaving one scorer pass plus this gather operation.

4.2 Scorer and Budgeted Soft Top-\(K\)↩︎

The Scorer \(\mathcal{S}_\theta\) is a two-layer Transformer encoder followed by a linear head that assigns one scalar logit \(s_i\) to each visual token \(\mathbf{x}_i^v\). It predicts only an ordering signal; budget control is delegated to \(\Phi_K\). We then apply a budgeted Soft Top-\(K\) map, a differentiable top-\(K\) operator that returns continuous retention weights under a fixed cardinality budget [47]: \[\boldsymbol{\alpha} = \Phi_K\!\left(\mathbf{s}/\tau\right)\in[0,1]^N, \qquad \sum_{i=1}^{N}\alpha_i = K. \label{eq:soft-topk}\tag{4}\] The budget is enforced by \(\Phi_K\), so no auxiliary budget loss is used. The temperature \(\tau\) controls sharpness: cosine annealing lowers \(\tau\) during training, making \(\boldsymbol{\alpha}\) increasingly close to a binary top-\(K\) mask. With no score tie at the \(K\)-th boundary, the low-temperature limit matches the hard top-\(K\) ranking used at inference in 3 . During training, the downstream VLM consumes \(\boldsymbol{\alpha}\) itself, and the backward pass differentiates the same continuous map used in the forward pass. This differs from the surrogate-gradient operator in 1 , where the forward executes a hard mask but the backward follows a different smooth path.

4.3 Information Throttling with VP-Noise↩︎

Soft Top-\(K\) produces continuous retention weights, but the downstream VLM must experience them as token-level information restriction. DiffPrune therefore uses a token-wise Throttler \(T(\mathbf{x}_i;\alpha_i)\mapsto\tilde{\mathbf{x}}_i\) for each visual token \(\mathbf{x}_i\in\mathbb{R}^{d_v}\) and weight \(\alpha_i\in[0,1]\). A simple scale gate, \(\tilde{\mathbf{x}}_i=\alpha_i\mathbf{x}_i\), is continuous but weak: low-scoring tokens keep their feature direction, and normalization layers can reduce the effect of magnitude shrinkage.

We therefore instantiate \(T\) with a variance-preserving noising form inspired by the forward process of diffusion models [48], \[\tilde{\mathbf{x}}_i = \sqrt{\alpha_i}\,\mathbf{x}_i + \sqrt{1-\alpha_i}\,\boldsymbol{\epsilon}_i, \qquad \boldsymbol{\epsilon}_i \sim \mathcal{N}(\mathbf{0},\mathbf{I}), \label{eq:vp-noise}\tag{5}\] which we refer to as VP-Noise. For high-scoring tokens, \(\alpha_i\approx1\) preserves the original representation; for low-scoring tokens, \(\alpha_i\approx0\) replaces most visual content with isotropic noise. For normalized visual tokens, the square-root coefficients keep feature scale approximately stable while changing how much token content is available. For fixed noise, this operation is differentiable in \(\alpha_i\), so the Scorer is trained through the same noised representation consumed in the forward pass. 4 validates VP-Noise from complementary visual and quantitative views. Panel (a) compares hard top-\(K\) pruning and VP-Noise gating at \(25\%\), \(50\%\), and \(75\%\) keep ratios, showing that VP-Noise keeps the retained spatial layout while corrupting low-retention patches instead of masking them out. Panel (b) reports ImageNet-1K Top-1/Top-5 accuracy under random scores; the small gap to hard top-\(K\) indicates that the continuous throttling path imposes comparable restriction strength.

4.4 Diagonal-Attention Denoiser↩︎

VP-Noise makes token removal differentiable, but low-retention tokens may deviate from the feature distribution expected by the frozen downstream stack. We therefore insert a train-only Denoiser \(\mathcal{D}_\phi\) after the Throttler to realign the throttled sequence while the scorer is being learned.

The Denoiser should not undo the information restriction imposed by VP-Noise. DiffPrune implements it as a single Transformer block with an identity attention mask, \[\mathcal{D}_\phi(\tilde{\mathbf{X}}^v) = \operatorname{Block}\!\left(\tilde{\mathbf{X}}^v;\operatorname{mask}=\mathbf{I}_N\right),\] so each token is transformed independently through projection, normalization, and feed-forward layers, without copying content from other positions. The Denoiser is removed at inference.

3pt

5 Experiments↩︎

5.1 Experimental Setup↩︎

DiffPrune is trained on \(10\%\) of the ImageNet-1K training split [49] with captions from ImageNet-1K-VL-Enriched [50]. Only the Scorer and Denoiser are optimized, adding approximately \(84\)M trainable parameters, while the base VLM remains frozen. Complete implementation, training, baseline, and profiling details are provided in [sec:appendix-impl]; benchmark definitions and reporting conventions are provided in 8.

5.2 Comparative Results↩︎

5.2.0.1 Results on LLaVA-1.5-7B.

LLaVA-1.5-7B processes \(336{\times}336\) inputs that produce \(N{=}576\) visual tokens, and 1 reports three compression regimes that retain \(192/128/64\) tokens (66.7%/77.8%/88.9% pruning). DiffPrune attains the strongest average retention at \(K{=}128\) and \(K{=}64\); against the closest prior method, PRUNESID, the gap moves from a \(0.3\%\) deficit at \(K{=}192\) to a \(1.4\%\) lead at \(K{=}64\), a direction consistent with the prediction of 3.1 that tighter budgets amplify the gap between fully differentiable and surrogate-gradient routes. The \(K{=}192\) gap to PRUNESID should be read together with the efficiency analysis of 5.3, where the PRUNESID selection module is shown to cost roughly \(60\times\) as much as ours.

4pt

5.2.0.2 Results on LLaVA-NEXT-7B.

LLaVA-NEXT-7B uses \(672{\times}672\) inputs and produces \(N{=}2880\) visual tokens, five times the token count of LLaVA-1.5-7B. This setting gives a stronger test of visual-token pruning because the language-model prefill cost grows with the visual prefix length. 2 reports results at \(K{=}320\), where each method keeps only \(11.1\%\) of the visual tokens. DiffPrune obtains \(96.1\%\) average retention, the best result among the reported methods and \(0.4\%\) higher than HoloV. We use the same scorer and pruning recipe as in LLaVA-1.5-7B, with no resolution-specific scheduling or architectural change.

5.2.0.3 Results on Qwen2.5-VL-7B.

Qwen2.5-VL-7B differs from the LLaVA family in vision encoder, projector and language backbone, and processes images at native resolution, so the visual-token count varies per image and results are reported by pruning rate rather than a fixed budget \(K\). Under the three pruning rates listed in 3, DiffPrune leads every evaluated baseline across the five benchmarks, with the margin over HoloV widening from \(4.0\%\) at \(66.7\%\) to \(6.1\%\) at \(88.9\%\). No architectural or hyperparameter adaptation is applied across backbones; the same Scorer, Soft Top-\(K\), Throttler, Denoiser and schedule used on LLaVA-1.5-7B are reused without modification, supporting the claim that the framework is backbone-agnostic by construction. We further examine this behavior across Qwen2.5-VL model scales in 9.

3pt

5.3 Efficiency Analysis↩︎

We profile all methods on LLaVA-1.5-7B with \(336{\times}336\) inputs and \(N{=}576\) visual tokens, using a single NVIDIA A6000 GPU, batch size one, and FP16 precision. Latencies are averaged over \(30\) forward passes after warm-up. 4 reports TTFT at \(K{=}64\), decomposed into vision encoding, pruning-module decision, and LLM prefill.

3pt

Vision encoding takes about \(30\) ms for all methods, since pruning is applied after the encoder. The main differences come from the pruner and the LLM prefill. PyramidDrop prunes inside the language model, so early layers still process the full visual prefix. Its LLM prefill remains \(70.08\) ms with \(4.89\) T FLOPs. In contrast, Pre-LLM methods reduce the visual prefix before it enters the language model, bringing LLM prefill to about \(40\) ms and FLOPs to \(2.13\)–\(2.15\) T. Among these methods, the pruner itself becomes the main source of latency. PRUNESID spends \(43.39\) ms on token selection, over \(60\times\) the cost of DiffPrune, while HoloV spends \(2.77\) ms. DiffPrune needs only \(0.69\) ms for selection and reaches the lowest TTFT in the comparison (\(73.73\) ms). This follows directly from its inference path in 3 : the Throttler and Denoiser are removed, leaving only one Scorer pass and index gathering.

5.4 Ablation Studies↩︎

We isolate three training-time design choices in DiffPrune: the gradient-continuous Throttler, the VP-Noise form, and the diagonal Denoiser. All variants keep the base VLM, Scorer architecture, training data, and schedule fixed. 5 reports average retention over GQA, MMBench, POPE, and MME on LLaVA-1.5-7B.

3pt

Replacing the Throttler with a Gumbel-Softmax + STE variant gives the largest drop. Retention decreases by \(8.1\), \(6.0\), and \(4.3\) points at \(K{=}64/128/192\), respectively. Since the rest of the pipeline is unchanged, this result supports the claim from 3 that the gradient-continuous training path is important in the full VLM setting, not only in the DeiT probe.

The other two rows separate DiffPrune’s internal design choices. Replacing VP-Noise with the fully differentiable scale gate \(\tilde{\mathbf{x}}_i=\alpha_i\mathbf{x}_i\) reduces retention by \(2.3\), \(2.5\), and \(1.8\) points, showing that noise injection gives a stronger information restriction than simple rescaling. Replacing diagonal attention in the Denoiser with global self-attention reduces retention by \(2.8\), \(2.3\), and \(2.4\) points. This supports per-token denoising, since global attention can let corrupted low-score tokens recover information from high-score tokens.

5.5 Qualitative Analysis↩︎

5 visualizes the token scores learned by DiffPrune on an example from the VQAv2 benchmark used in our evaluation. For readability, we show the two extremes of the score ranking: the top-\(64\) tokens cover visually salient regions such as the face, hands, and clothing, while the bottom-\(64\) tokens mainly fall on visually uniform background areas. The accompanying \(128{\times}128\) cosine-similarity matrix \([\text{top-}64;|;\text{bottom-}64]\) provides a feature-space view of the same ranking: in this example, the retained-token block has lower internal similarity, whereas the lowest-scored block is more redundant. This qualitative evidence is consistent with the quantitative results, while remaining illustrative rather than conclusive: the learned Scorer appears to allocate its limited budget toward visually informative and less redundant tokens.

6 Conclusion↩︎

We identify the discrete selection operator as a central obstacle in learned visual-token pruning: continuous importance scores must eventually become hard keep-or-drop decisions, and common surrogate-gradient solutions train through a backward path that is not the function executed in the forward pass. DiffPrune addresses this mismatch by replacing training-time selection with continuous token-information control. Its Soft Top-\(K\) head and Information Throttler keep the scorer-to-loss path analytically differentiable during training, while inference collapses to a hard top-\(K\) scorer-only path. The resulting framework improves gradient-direction coherence by up to \(28.4\times\), reduces local loss variation by roughly \(24\times\) in our diagnostic study, and adds only \(0.69\) ms of selection overhead on LLaVA-1.5-7B. These findings suggest that full differentiability is a useful design rule for learned pruning, not merely an implementation detail. Extending the same principle to other sparsity operators remains future work.

7 Implementation Details and Training Hyperparameters↩︎

7.0.0.1 Implementation.

All experiments are implemented in PyTorch and run on a single NVIDIA RTX A6000 GPU. The implementation is built on the LLaVA-v1.5 codebase [51]. For every evaluated backbone, the vision encoder, multimodal projector, and language model remain frozen; only the DiffPrune Scorer and Denoiser are optimized.

7.0.0.2 Architecture.

The Scorer contains two Transformer encoder blocks with hidden dimension \(1024\) and \(16\) attention heads, followed by a linear projection that maps each visual token to a scalar importance logit. The Denoiser is a single Transformer block with the diagonal attention mask described in 4.4. Together, these two trainable modules add approximately \(84\)M parameters, about \(1.2\%\) of LLaVA-1.5-7B. At inference time, both the Throttler and Denoiser are removed; the deployed graph contains only the Scorer and hard top-\(K\) gathering in 3 .

7.0.0.3 Training.

The Scorer and Denoiser are trained jointly in a single stage on a \(10\%\) subset of the ImageNet-1K training split [49], paired with captions from ImageNet-1K-VL-Enriched [50]. We use AdamW with learning rate \(2{\times}10^{-4}\), cosine decay, and batch size \(32\). The objective is caption generation under the next-token negative log-likelihood in 2 ; no labels from downstream evaluation benchmarks are used for training. The Soft Top-\(K\) temperature is annealed from \(\tau_{\mathrm{start}}{=}2.0\) to \(\tau_{\mathrm{end}}{=}0.1\) over \(2{,}000\) steps, gradually sharpening the continuous retention weights toward the hard top-\(K\) ranking used at inference. Training runs for one epoch over the subset with early stopping based on validation loss and takes approximately ten GPU-hours.

7.0.0.4 Backbones and token budgets.

LLaVA-1.5-7B uses \(336{\times}336\) images and produces \(N{=}576\) visual tokens. We evaluate \(K{\in}\{192,128,64\}\), corresponding to \(66.7\%\), \(77.8\%\), and \(88.9\%\) pruning. LLaVA-NEXT-7B uses \(672{\times}672\) inputs and produces \(N{=}2880\) visual tokens; we report \(K{=}320\). Qwen2.5-VL-7B processes images at native resolution, so the number of visual tokens varies by example; for this backbone, results are reported by pruning rate rather than by a fixed \(K\). Unless otherwise stated, the same Scorer, Soft Top-\(K\) operator, Throttler, Denoiser, and training schedule are reused across backbones.

7.0.0.5 Baselines and reporting convention.

Baselines include FastV [7], SparseVLM [8], DART [29], PyramidDrop [24], DivPrune [11], VisionZip [10], HoloV [30], PRUNESID [32], ATP-LLaVA [38], Dynamic-LLaVA [39], p-MoD [52], GlimpsePrune [53], and GumbelDiffPrune. GumbelDiffPrune is an in-house controlled variant that replaces the DiffPrune Throttler with Gumbel-Softmax and a Straight-Through Estimator while keeping the rest of the pipeline fixed. When applicable, Loc.records whether pruning happens before the language model (Pre) or inside it (L\(a\)–L\(b\)). Entries are marked “–” when a compatible result is unavailable.

7.0.0.6 Efficiency profiling.

Latency is profiled on LLaVA-1.5-7B with \(336{\times}336\) inputs, \(N{=}576\) visual tokens, batch size one, and FP16 precision on the same NVIDIA A6000 GPU. We average \(30\) forward passes after warm-up. Time-to-first-token is decomposed into vision encoding, pruning-module decision time, and language-model prefill, so the reported pruning overhead isolates the cost added by each selector.

8 Benchmark Descriptions↩︎

We evaluate DiffPrune on ten standard VLM benchmarks spanning compositional reasoning, perception, hallucination, text-centric understanding, and real-world VQA.

8.0.0.1 GQA [54].

GQA is a compositional visual question answering benchmark built from Visual Genome scene graphs. It emphasizes multi-step reasoning over objects, attributes, spatial relations, and logical operations, making it useful for testing whether token pruning preserves structured visual evidence.

8.0.0.2 MMBench and MMBench-CN [55].

MMBench evaluates multimodal perception and reasoning through multiple-choice questions organized into fine-grained ability dimensions. We report both the English benchmark and its Chinese counterpart, MMBench-CN, which follows the same evaluation taxonomy and enables testing under a language distribution shift.

8.0.0.3 MME [56].

MME evaluates perception and cognition through manually designed yes/no questions. Its perception subset covers skills such as existence, count, position, and color recognition, while its cognition subset covers broader reasoning categories. Because each image is paired with controlled questions, MME is sensitive to both visual information loss and hallucinated content.

8.0.0.4 POPE [57].

POPE measures object hallucination by querying whether specific objects are present in an image. Questions are sampled under random, popular, and adversarial strategies, probing whether a model relies on visual evidence rather than dataset-level object co-occurrence priors.

8.0.0.5 ScienceQA-IMG [58].

ScienceQA-IMG contains image-grounded science questions that require both visual understanding and domain knowledge. We use the image-containing subset, following standard VLM evaluation practice.

8.0.0.6 VQAv2 [59].

VQAv2 is an open-ended visual question answering benchmark designed to reduce language priors through complementary image pairs. It tests whether the model’s answer changes with the visual evidence rather than being driven only by question statistics.

8.0.0.7 TextVQA [60].

TextVQA focuses on reading and reasoning over text appearing inside images, such as signs, labels, documents, and scene text. This benchmark stresses OCR-sensitive visual tokens and is therefore a stringent test for pruning methods that remove large fractions of the visual prefix.

8.0.0.8 SEED-Bench [61].

SEED-Bench evaluates image-centric multimodal comprehension across dimensions such as scene understanding, spatial relation, instance identity, and instance attribute recognition. We use it to assess whether pruning preserves broad semantic coverage beyond conventional VQA accuracy.

8.0.0.9 VizWiz [62].

VizWiz contains questions about images taken by blind users in everyday settings. The images often contain unusual framing, low quality, or unanswerable queries, making the benchmark a realistic stress test for robustness under noisy visual inputs.

For each benchmark, we report its standard score. To summarize performance across heterogeneous metrics, we additionally report average retention: each pruned score is normalized by the corresponding unpruned upper bound, and the normalized values are averaged across benchmarks.

9 Scalability Across Model Scales↩︎

To further test whether DiffPrune depends on a particular model capacity, we evaluate the 3B, 7B, and 32B variants of Qwen2.5-VL [63] on MME, ScienceQA, and MMBench. We use five visual-token pruning rates: \(25\%\), \(50\%\), \(66.7\%\), \(77.8\%\), and \(88.9\%\). The same pruning formulation and training recipe are used across model sizes.

6 shows that, across all three scales, performance decreases smoothly as the pruning rate increases, with no abrupt accuracy collapse even at \(88.9\%\) pruning. On MME, the 32B model remains above \(2{,}400\) points near the \(77.8\%\) pruning setting, while the 7B and 3B variants show steady rather than discontinuous degradation from their respective unpruned baselines. MMBench and ScienceQA show the same pattern: the 32B and 7B results remain close through high pruning rates, and all models preserve usable performance under aggressive compression. These results suggest that DiffPrune’s budgeted, score-driven formulation is not tied to a single Qwen2.5-VL model size.

10 Limitation and Future Work↩︎

DiffPrune relies on the continuity of visual-token embeddings. VP-Noise throttles a token by interpolating its feature vector with isotropic Gaussian noise under a variance-preserving schedule before the frozen language model consumes it. This assumption is natural for VLM visual tokens, but it does not transfer directly to pure LLM settings, where the objects being pruned may be discrete text tokens, token-indexed KV entries, or weight structures whose semantics are not preserved under the same feature-space perturbation. DiffPrune further introduces training-time cost through the Scorer and Denoiser, even though the Throttler and Denoiser are removed at inference. Finally, while our pilot studies show substantially higher gradient-direction coherence and smoother scorer loss landscapes than Gumbel-Softmax pruning, a complete theoretical account of how feature dimensionality, the geometry of the \(K\)-th order statistic, and surrogate-gradient bias interact remains open.

Despite these limitations, these properties also suggest promising directions for future work in high-resolution visual domains. In particular, medical imaging and related settings are characterized by substantial redundancy in visual tokens [64]–[66]. This phenomenon has been widely observed in medical image analysis [67]–[70], multimodal medical foundation models [71]–[74], and long-context visual reasoning tasks [75], [76], suggesting that DiffPrune may be especially effective in these scenarios.

References↩︎