1 Introduction↩︎

Multimodal large language models (MLLMs) [1]–[5] have become a central paradigm for visual understanding, visual reasoning, and visual question answering. A typical MLLM encodes an image into visual tokens, projects them into the embedding space of a large language model (LLM), and concatenates them with text tokens for autoregressive generation [1], [4], [5]. To preserve fine-grained visual evidence, recent MLLMs increasingly adopt high-resolution inputs, image tiling, or dense visual encoders, expanding the visual sequence from hundreds to thousands of tokens [3]–[6]. While this improves perceptual coverage, it also makes the prefill stage expensive, as the LLM backbone must process all visual and textual tokens simultaneously through self-attention and feed-forward transformations [7]. A growing body of work therefore accelerates MLLMs by reducing visual computation, either by pruning, selecting, or merging visual tokens [8]–[11], or by skipping visual-token updates in selected Transformer layers [12]–[14]. These methods show that not all visual computation is equally necessary, but they rely on coarse notions of redundancy. Token-level methods may permanently discard fine-grained evidence, layer-level skipping removes attention and FFN computation together, and V-Skip [14] assumes a fixed preference for preserving FFN while skipping visual attention. Such fixed granularities cannot decide whether attention, FFN, or both are redundant at each layer, and may suppress useful computation when only part of a layer is redundant.

In this paper, we revisit visual redundancy in MLLMs by examining how visual tokens evolve inside the LLM backbone. We begin with a counterintuitive observation, in late layers, visual-token hidden states can still change substantially, even when these changes have little influence on the final answer. This indicates that update magnitude alone is not a reliable measure of useful visual computation. To distinguish representational movement from computation that actually affects answer generation, we introduce a set of answer-observable diagnostics, which measure how visual-token updates propagate to answer-token hidden states. Across multiple MLLM architectures and VQA benchmarks, we consistently find that late visual tokens often undergo large hidden-state updates while exhibiting small answer-observable effects. These updates are therefore not inactive, but answer-silent, they consume computation while contributing little to the representations that determine the generated response.

This observation suggests that visual redundancy is not merely a property of individual tokens, nor is it always aligned with entire Transformer layers. Instead, redundancy may emerge at a finer operator level within each layer. This view is supported by recent analyses of LVLM backbones [15], which reveal a functional decoupling between attention and FFN operators, attention primarily reconfigures information within a relatively preserved representation subspace, whereas FFNs expand the representation subspace and drive semantic transformation. If these two operators serve different representational roles, their importance for visual tokens should not be assumed to be coupled in every MLLM layer. Some layers may require attention to route visual evidence toward the question, some may rely more on FFN transformations to refine visual semantics, and others may contribute little through either operator.

Motivated by this observation, we decompose each Transformer layer into its attention and FFN operators and ask a finer question: which operator is actually necessary for visual-token computation in each layer? By separately measuring the answer-observable contribution of attention and FFN, we find strong layer-wise heterogeneity, some layers are attention-dominant, some are FFN-dominant, and others show little answer-observable contribution. Based on this observation, we propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while bypassing redundant visual computation. This design avoids the irreversible evidence loss of token pruning and releases visual skipping from the static assumption that either attention or FFN is always the important operator. Experiments across multiple MLLM architectures and 10 visual question answering benchmarks show that our framework substantially reduces visual computation during inference while maintaining answer quality, achieving up to 33.7% visual-computation reduction with only 0.5% average performance change. Beyond efficiency, our analysis suggests that useful visual computation in MLLMs is operator-dependent rather than block-aligned, and that efficient inference should be designed around answer-observable computation rather than raw update magnitude or fixed architectural boundaries.

Our contributions are summarized as follows:

• We identify answer-silent visual-token updates in late MLLM layers and introduce answer-observable diagnostics to measure whether visual computation actually propagates to answer-token representations.

• We reveal that useful visual computation is operator-dominant and layer-dependent by decomposing each Transformer layer into attention and FFN operators, exposing visual redundancy beyond token- and layer-level views.

• We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while bypassing redundant visual computation, saving up to 33.7% visual computation while retaining 99.5% model performance across Qwen3VL and 10 VQA benchmarks.

2 Answer-Silent Visual Updates in MLLM↩︎

We first examine when visual-token information is effectively used by the textual stream during the prefill stage. In mainstream decoder-only MLLMs [1], [4], [5], visual tokens are placed before the textual prompt and processed under causal masking. As a result, the last prompt token is the first position that can integrate both the visual prefix and the textual question. Its hidden state, which we refer to as the final prompt-token hidden state, is directly projected by the language-modeling head to produce the next-token logits. This motivates the following diagnostic question:

Let \(H_v^l \in \mathbf{R}^{N_v \times d}\) denote the visual-token hidden states at layer \(l\), where \(N_v\) is the number of visual tokens and \(d\) is the hidden dimension. We first quantify the magnitude of visual-token state changes between consecutive layers: \[{!}{\displaystyle \Delta V_l = H_v^{l+1} - H_v^l,\quad \mathrm{VUM}_l = \frac{|\Delta V_l|_F}{\max_j |\Delta V_j|_F + \epsilon}. } \label{eq:vum}\tag{1}\] We refer to this normalized quantity as visual update magnitude (VUM), where larger values indicate stronger visual-state changing.

To measure whether such changing is observable from the answer side, we estimate its first-order influence on the final prompt-token hidden state. Let \(h_{\mathrm{last}}\) denote the final prompt-token hidden state during prefill, and let \[J_l = \frac{\partial h_{\mathrm{last}}}{\partial H_v^{l+1}}\] be the Jacobian that maps perturbations of visual-token states at layer \(l+1\) to changes in \(h_{\mathrm{last}}\). We define \[\delta h_l = J_l[\Delta V_l], \qquad \mathrm{AOI}_l = |\delta h_l|_2 . \label{eq:aoi}\tag{2}\] and we call \(\mathrm{AOI}_l\) answer-observable influence (AOI), where \(J_l[\Delta V_l]\) is the Jacobian-vector product estimating how the visual update at layer \(l\) changes the final prompt-token hidden state.

Finally, we normalize AOI by the size of the visual update: \[{!}{\displaystyle \mathrm{AOE}_l = \frac{\mathrm{AOI}_l}{\|\Delta V_l\|_F + \epsilon}\; \widetilde{\mathrm{AOE}}_l = \frac{\mathrm{AOE}_l}{\max_j \mathrm{AOE}_j + \epsilon}. } \label{eq:aoe}\tag{3}\] We refer to this quantity as answer-observable efficiency (AOE). In summary, VUM measures the magnitude of visual-token changes, AOI measures how much these changes reach the final prompt-token hidden state, and AOE measures the answer-observable influence per unit of visual-token update.

Figure 2 shows the layer-wise behavior of VUM, AOI, and AOE on GQA and TextVQA for LLaVA-1.5 and Qwen3-VL. The curves reveal a consistent late-layer decoupling: visual-token states continue to change substantially, while their answer-observable influence decreases. In LLaVA-1.5, VUM rises sharply after layer 24, whereas both AOI and AOE continue to decline. Qwen3-VL exhibits the same pattern around its decoupling onset layer. Thus, raw update magnitude systematically overestimates the usefulness of late visual computation.

This observation suggests a direct intervention. If late visual-token updates are weakly coupled to answer generation, freezing visual-token states after the decoupling onset layer should have limited impact on task performance. We therefore freeze visual-token hidden states after layer 24 while keeping all text-token computation unchanged. Concretely, for layers \(l > 24\), we reuse the visual states from layer 24 and allow textual tokens to continue attending to these fixed visual representations. This removes late visual-state changes while preserving the visual evidence accessible to textual tokens.

As shown in Table 1, freezing late visual-token states causes only marginal degradation on 4 benchmarks across the two MLLM architectures. This result indicates that repeatedly updating visual-token states is often not the computation that determines next-token prediction. The model can still access visual evidence through the textual stream, while the continued changes of late visual-token states become increasingly answer-silent.

This finding motivates a finer-grained analysis of where useful visual computation resides within each Transformer block. We therefore move beyond the block-level view and decompose each layer into its attention and FFN operators, analyzing which operator dominates answer-observable visual computation at different depths.

3 Methodology↩︎

3.1 Operator Risk Analysis↩︎

We first describe how to identify the layer-wise operator structure of the visual branch without retraining the model or using human-annotated answers. Given a decoder-only MLLM, let the hidden states at layer \(l\) be \[H^l = [H_{\mathcal{V}}^l; H_{\mathcal{T}}^l].\] where \(H_{\mathcal{V}}^l \in \mathrm{R}^{N_v \times d}\) denotes the visual-token hidden states and \(H_{\mathcal{T}}^l\) denotes the textual hidden states, including the system prompt, question tokens, and possible answer-prefix tokens. We decompose the visual branch of a Transformer block into two residual updates: \[\begin{align} \bar{H}_{\mathcal{V}}^{l} &= H_{\mathcal{V}}^{l} + \big[A^{l}(H^{l})\big]_{\mathcal{V}}, \\ H_{\mathcal{V}}^{l+1} &= \bar{H}_{\mathcal{V}}^{l} + \big[F^{l}(\bar{H}^{l})\big]_{\mathcal{V}} . \end{align} \label{eq:operator95decomposition}\tag{4}\] where \(A^{l}\) is the self-attention update and \(F^{l}\) is the FFN update at layer \(l\). The subscript \(\mathcal{V}\) denotes the visual-token rows. This decomposition allows us to ask a more fine-grained question than whether an entire layer can be skipped:

For an input sample \(x\), the full model produces a reference next-token distribution from the final prompt-token hidden state after prefill: \[p_0(\cdot|x)=\mathrm{softmax}(z_0(x)).\] This reference distribution is obtained from the model itself and does not require ground-truth labels. We estimate layer-wise operator risks on a small calibration set \(\mathcal{D}\) sampled from multiple benchmarks, using these samples only to probe how the output distribution changes under counterfactual visual-operator interventions. \[\begin{align} &\frac{1}{|\mathcal{D}|} \sum_{x\in\mathcal{D}} D_{\mathrm{KL}}\!\left( p_0(\cdot|x)\Vert p_{s}^{(l)}(\cdot|x) \right), \\ &\quad s\in\{\mathcal{A},\mathcal{F},\mathcal{AF}\}. \end{align}\]

Here, \(\mathcal{R}{\mathcal{A}}(l)\), \(\mathcal{R}{\mathcal{F}}(l)\), and \(\mathcal{R}_{\mathcal{AF}}(l)\) measure the output risk of removing visual attention, removing visual FFN, and freezing the visual branch at layer \(l\), respectively.

\[\log \frac{\mathcal{R}{\mathcal{A}}(l)+\epsilon}{\mathcal{R}{\mathcal{F}}(l)+\epsilon}.\] A positive \(\Gamma_l\) indicates an attention-dominant layer, while a negative \(\Gamma_l\) indicates an FFN-dominant layer. Layers with uniformly small risks, especially small \(\mathcal{R}_{\mathcal{AF}}(l)\), are treated as frozen layers.

3.2 Operator-Aware Skipping↩︎

We convert the layer-wise risk profile into an inference-time skipping policy through two steps, budgeted layer selection and policy assignment. Given a compute budget \(B\), we rank layers by \(\mathcal{R}{\mathcal{A}}(l)+\mathcal{R}{\mathcal{F}}(l)\) and select the \(B\) lowest-risk layers for simplification. These are layers where removing either visual attention or visual FFN individually induces only a small output shift.

For each selected layer, we assign the least risky visual-branch policy. If \(\mathcal{R}{\mathcal{A}}(l) < \mathcal{R}{\mathcal{F}}(l)\), we apply FFN-only, skipping visual attention while preserving visual FFN. If \(\mathcal{R}{\mathcal{F}}(l) < \mathcal{R}{\mathcal{A}}(l)\), we apply Attention-only, preserving visual attention while skipping visual FFN. If \(\mathcal{R}{\mathcal{AF}}(l)\) is sufficiently small, we apply the more aggressive Freeze-update policy, skipping both visual operators and directly propagating \(H{\mathcal{V}}^l\) to the next layer.

Unlike token pruning, our method preserves the full visual-token sequence and avoids irreversible evidence removal. Unlike layer-level skipping, it does not treat attention and FFN as an inseparable block, but preserves the dominant visual operator while bypassing redundant visual-branch computation.

3.3 Complexity Analysis↩︎

Let \(N_v\) be the number of visual tokens, \(d\) the hidden dimension, and \(d_{\mathrm{ff}}\) the FFN intermediate dimension. Since our method modifies only visual rows and keeps text-token computation unchanged, the computational reduction comes from skipping visual-row attention, visual-row FFN, or both.

\[\begin{align} \Delta \mathcal{C}_{\mathcal{A}} &= \Theta\left(N_v d^2 + M_{\mathcal{V}} d\right), \\ \Delta \mathcal{C}_{\mathcal{F}} &= \Theta\left(\kappa N_v d d_{\mathrm{ff}}\right). \end{align}\] where \(\kappa=3\) for gated FFNs used in LLaMA- and Qwen-style backbones, and \(\kappa=2\) for standard two-layer FFNs. \[\begin{align} \Delta \mathcal{C}_{\mathrm{skip}} &= \sum_{l\in \mathcal{L}_{\mathrm{FFN}}} \Delta \mathcal{C}_{\mathcal{A}}^{(l)} \\ &\quad + \sum_{l\in \mathcal{L}_{\mathrm{Attn}}} \Delta \mathcal{C}_{\mathcal{F}}^{(l)} \\ &\quad + \sum_{l\in \mathcal{L}_{\mathrm{Freeze}}} \left( \Delta \mathcal{C}_{\mathcal{A}}^{(l)} + \Delta \mathcal{C}_{\mathcal{F}}^{(l)} \right). \end{align}\] Here, FFN-only skips visual attention, Attention-only skips visual FFN, and Freeze-update skips both operators. In practice, wall-clock speedup also depends on kernel implementation, memory movement, and hardware utilization; therefore, we report both theoretical FLOPs and measured first-token latency.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Models.

We evaluate our method on three representative open-source MLLMs, LLaVA-1.5-7B [1], Qwen2.5-VL-7B [4], and Qwen3-VL-8B [5]. LLaVA-1.5-7B uses a fixed-resolution visual encoder and represents each image with 576 visual tokens, while Qwen2.5-VL-7B and Qwen3-VL-8B adopt more recent Qwen-VL architectures with stronger visual encoders and dynamic-resolution image processing. This evaluation tests whether operator-level visual redundancy generalizes across different MLLM designs. All evaluations are conducted with lmms-eval under the official benchmark protocols.

4.1.0.2 Benchmarks.

We evaluate on 10 multimodal benchmarks covering general VQA, text-rich understanding, hallucination sensitivity, multimodal reasoning, and scientific QA, GQA [16], TextVQA [17], MME [18], MMBench [19], MMMU [20], POPE [21], ScienceQA [22], AI2D [23], OCRBench [24], and VizWiz [25].

4.1.0.3 Baselines.

We compare with training-free acceleration baselines including layer-wise visual skipping methods VTW [13], ShortV [12], and V-Skip [14], as well as token-reduction methods V2Drop [26] and APET [27]. Unlike these coarse-grained baselines, our method preserves all visual tokens and selectively skips attention, FFN, or both according to layer-wise operator dominance.

4.1.0.4 Implementation details.

For each model, we estimate layer-wise visual operator risks using a small calibration set, sampling \(7\) examples from each of GQA, POPE, TextVQA, MME, MMMU, and ScienceQA. These examples are used only for estimating \(\mathcal{R}{\mathcal{A}}(l)\), \(\mathcal{R}{\mathcal{F}}(l)\), and \(\mathcal{R}_{\mathcal{AF}}(l)\). Based on the risks, each selected layer is assigned one of three policies, Attention-only, FFN-only, or Freeze-update. The resulting layer-wise policy is fixed for all samples of the same model. We report relative FLOPs ratio, and first-token latency measured on NVIDIA RTX A800 GPUs.

4.2 Main Results↩︎

Table 2 compares our method with representative layer-wise skipping and token-pruning baselines across three MLLM architectures and ten benchmarks. Our method achieves a strong accuracy–efficiency trade-off by preserving the full visual-token sequence while selectively skipping redundant operators, leading to the highest average performance retention across the evaluated backbones.

This robustness comes from keeping the dominant visual operator in each selected layer, which helps preserve fine-grained and text-rich evidence while removing answer-silent computation. Following the V-Skip paper, VSkip⁺ uses dataset-specific optimal settings, whereas the other methods use one configuration for all benchmarks on each model. Our single per-model policy remains competitive or superior under this comparison setting.

4.3 Efficiency Comparison on TextVQA↩︎

We further compare efficiency on TextVQA against ShortV, a competitive layer-wise visual-computation baseline. As shown in Table 4, ShortV reduces more FLOPs but causes larger accuracy degradation, whereas our method achieves comparable prefill efficiency and first-token latency while retaining stronger answer quality. This shows that operator-level skipping provides a better accuracy–efficiency balance than freezing full visual-token updates.

Figure 5 visualizes model responses under operator-aware visual-token skipping. Our method still produces accurate answers on both reasoning and recognition examples, suggesting that it preserves the operators responsible for key layer-wise visual reasoning functions and largely maintains the model’s answer quality.

4.4 Effect of Operator-Skipping Budget↩︎

We study how the operator-skipping budget affects performance across the three evaluated MLLMs. As shown in Table 3, moderate budgets simplify redundant visual computation through Attention-only, FFN-only, or Freeze-update while preserving most benchmark scores, whereas overly aggressive budgets eventually degrade performance on fine-grained or text-rich tasks. This consistent trend across LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B suggests that operator-level redundancy is broadly present, but the best budget remains model-dependent and should be treated as an efficiency–performance knob.

4.5 Ablation Study↩︎

To isolate the contribution of operator-aware policy assignment, we conduct a fixed-budget ablation on Qwen3-VL-8B with budget \(B=28\). All variants simplify the same number of layers, but use different operator actions, Attn-Skip always skips visual attention, FFN-Skip always skips visual FFN, and Frozen skips both operators. As shown in Table 5, fixed rules are not uniformly reliable, while our layer-wise risk-based assignment better balances efficiency and accuracy by matching the skipped operator to each layer’s dominant visual operator.

4.6 Combination with Token Pruning↩︎

Finally, we study whether operator-aware skipping is complementary to token pruning. Token pruning reduces the visual sequence length, while our method removes redundant operator computation on the remaining tokens, so the two strategies target different sources of redundancy. As shown in Table 6, applying our method on top of VisionZip [11] or V2Drop [26] further reduces computation while preserving answer quality better than more aggressive pruning. This suggests that, especially in the low-token regime, retaining more visual evidence and skipping redundant operators is preferable to further shortening the visual sequence.

4.7 Qualitative Comparison↩︎

Figure 5 shows that operator-aware visual-token skipping preserves accurate responses on examples requiring visual reasoning and object recognition. These qualitative results suggest that our policy maintains answer capability while removing redundant visual computation.

5 Conclusion↩︎

We demonstrated that visual redundancy in MLLMs is answer-observable and operator-dependent, with late visual-token updates often contributing little to answers. Our operator-aware skipping preserves all visual tokens while bypassing redundant attention, FFN, or both, reducing visual computation across multiple backbones with little performance loss, without irreversible evidence loss from pruning.

6 Related Work↩︎

6.0.0.1 Layer-wise visual-computation skipping.

As high-resolution inputs, image tiling, and dense visual encoders increase the length of visual prefixes, the LLM prefill stage has become a major bottleneck for MLLM inference [7], [12]–[14]. One line of work reduces this cost by modifying how visual tokens are processed across Transformer layers. VTW withdraws visual tokens from later layers to avoid repeatedly propagating visual states through the full LLM backbone [13], while ShortV freezes visual-token states in layers identified as visually ineffective [12]. V-Skip studies block-wise attention skipping and shows that visual attention computation can be redundant in selected layers [14]. Related structural acceleration methods also exploit layer-wise or block-wise redundancy to reduce visual computation without directly deleting all visual inputs [7], [14]. Together, these studies show that visual computation can be redundant in a depth-dependent manner, but they differ in the granularity at which redundancy is modeled. VTW and ShortV primarily act on full visual-token states, whereas V-Skip focuses on attention computation. Our work is complementary, instead of treating a layer or a single operator family as the default unit of analysis, we compare the answer-observable risks of visual-row attention and visual-row FFN within each layer. This operator-level view allows the policy to preserve the operator that remains useful at a given depth while skipping the redundant one, or to freeze the visual update when both operators are answer-silent.

6.0.0.2 Visual-token pruning and compression.

Another major direction accelerates MLLM inference by reducing the number of visual tokens. Early and recent methods prune, merge, or select visual tokens according to attention scores, token similarity, spatial redundancy, diversity, or output sensitivity [7], [9]–[11], [26]–[29]. These approaches directly shorten the visual sequence and can reduce both attention and FFN costs in subsequent layers. However, token reduction is inherently destructive, once visual tokens are removed, merged, or prematurely withdrawn, the corresponding spatial evidence cannot be recovered by later textual reasoning. This limitation is especially pronounced in fine-grained recognition tasks, including OCR-centric VQA, chart or diagram understanding, small-object reasoning, and visually grounded question answering, where the decisive evidence may occupy only a small image region or become relevant only after interacting with the question. Recent studies further show that aggressive token pruning can damage spatial integrity, shift relevant visual information during decoding, and even underperform random selection in certain settings [8], [30]–[32]. In contrast, our method preserves the full visual-token sequence throughout inference and reduces computation at the operator level. This avoids irreversible loss of fine-grained visual evidence while still removing answer-silent visual updates, matching the motivation in the introduction: efficient MLLM inference should preserve visual information and reduce only computation that is not observable from the answer side.

7 Implementation Environment↩︎

All experiments were conducted on an internal GPU cluster equipped with NVIDIA A800-SXM4 GPUs. Each compute node contains two Intel(R) Xeon(R) Platinum 8358 CPUs at 2.60GHz, providing 128 logical CPU cores in total, approximately 1 TiB of system memory, and 8 NVIDIA A800-SXM4 GPUs with 80GB VRAM each. The NVIDIA driver version is 535.161.08. The system CUDA toolkit is CUDA 12.2, while the PyTorch binaries used in our conda environments are built with CUDA 12.1.

We use the official lmms-eval framework for all reported multimodal benchmark results and follow its default evaluation protocols. Since different model families rely on different dependency constraints, we maintain separate conda environments for Qwen2-VL, Qwen3-VL, LLaVA-1.5, and LLaVA-NeXT. Model checkpoints are loaded from local copies of the corresponding public Hugging Face repositories. The evaluated benchmark suite includes MME, MMBench, OCRBench, POPE, GQA, TextVQA, ScienceQA, MMMU, AI2D, and VizWiz when applicable.

8 Computation of Answer-Observable Metrics↩︎

In Section 2, we define answer-observable influence (AOI) using the exact Jacobian from visual-token states to the final prompt-token hidden state. This definition is mathematically direct, but computing the full Jacobian is prohibitively expensive for modern MLLMs. We therefore estimate AOI with randomized probes. This section derives the estimator used in our implementation.

Let \(H_v^{l+1} \in \mathbf{R}^{N_v \times d}\) denote the visual-token hidden states after layer \(l\), and let \[\Delta V_l = H_v^{l+1} - H_v^l\] be the visual-token update. We write \(v_l=\mathrm{vec}(\Delta V_l)\) for its flattened representation. Let \(h_{\mathrm{last}}\in\mathbf{R}^{d}\) be the final prompt-token hidden state after prefill, and let \[J_l = \frac{\partial h_{\mathrm{last}}}{\partial \mathrm{vec}(H_v^{l+1})} \in \mathbf{R}^{d \times N_vd}\] be the Jacobian mapping perturbations of visual-token states at layer \(l+1\) to perturbations of the final prompt-token hidden state. The exact first-order change induced by the visual update is \[\delta h_l = J_l v_l,\] and the exact answer-observable influence is \[\mathrm{AOI}_l = \|\delta h_l\|_2 = \|J_l v_l\|_2 .\] Directly forming \(J_l\) is infeasible because it has size \(d \times N_vd\). Instead, we estimate the norm of \(J_l v_l\) through random projections.

Let \(r_k \in \mathbf{R}^{d}\) be an independent random probe satisfying \[\mathbf{E}[r_k r_k^\top] = I_d .\] In practice, \(r_k\) can be sampled from a standard Gaussian distribution or a Rademacher distribution. For each probe, we define \[g_{l,k} = \left\langle \frac{\partial (r_k^\top h_{\mathrm{last}})}{\partial H_v^{l+1}}, \Delta V_l \right\rangle_F .\] This quantity can be computed with a single reverse-mode gradient call, without explicitly constructing the full Jacobian. By the chain rule, \[\frac{\partial (r_k^\top h_{\mathrm{last}})}{\partial \mathrm{vec}(H_v^{l+1})} = J_l^\top r_k .\] Therefore, \[g_{l,k} = \langle J_l^\top r_k, v_l\rangle = r_k^\top J_l v_l = r_k^\top \delta h_l .\] Thus, \(g_{l,k}\) is a random projection of the exact answer-side perturbation \(\delta h_l\).

The squared projection gives an unbiased estimator of the squared AOI: \[\begin{align} \mathbf{E}_{r_k}[g_{l,k}^2] &= \mathbf{E}_{r_k} \left[(r_k^\top \delta h_l)^2\right] \\ &= \delta h_l^\top \mathbf{E}[r_k r_k^\top] \delta h_l \\ &= \|\delta h_l\|_2^2 \\ &= \mathrm{AOI}_l^2 . \end{align}\] With \(K\) independent probes, we estimate AOI as \[\widehat{\mathrm{AOI}}_l = \left( \frac{1}{K} \sum_{k=1}^{K} g_{l,k}^2 \right)^{1/2}.\] This estimator is consistent for \(\mathrm{AOI}_l\) as \(K\) increases, while requiring only vector-Jacobian products rather than the full Jacobian. In our diagnostic figures, the plotted AOI is computed using this randomized estimator.

After estimating AOI, we compute answer-observable efficiency (AOE) by normalizing the estimated answer-observable influence by the visual update magnitude: \[\widehat{\mathrm{AOE}}_l = \frac{ \widehat{\mathrm{AOI}}_l }{ \|\Delta V_l\|_F + \epsilon } .\] For visualization across layers, we further normalize AOE within each model and dataset: \[\widetilde{\mathrm{AOE}}_l = \frac{ \widehat{\mathrm{AOE}}_l }{ \max_j \widehat{\mathrm{AOE}}_j + \epsilon } .\]

Therefore, VUM measures how much visual-token states move, AOI estimates how much this movement reaches the final prompt-token hidden state, and AOE measures the answer-observable influence induced per unit visual update.

In implementation, we first run the original model forward pass and cache \(H_v^l\), \(H_v^{l+1}\), and \(h_{\mathrm{last}}\) for each layer. For each layer \(l\), we sample \(K\) random probes \(\{r_k\}_{k=1}^K\), compute the scalar \(r_k^\top h_{\mathrm{last}}\), backpropagate it to obtain the gradient with respect to \(H_v^{l+1}\), and take its Frobenius inner product with \(\Delta V_l\). Model parameters are kept frozen, and no ground-truth answer labels are used. This makes the diagnostic label-free and applicable to arbitrary multimodal prompts.

9 Experiments on LLaVA-NeXT↩︎

To further test architectural generality, we additionally evaluate on LLaVA-NeXT. Compared with LLaVA-1.5, LLaVA-NeXT uses stronger visual processing and is commonly evaluated under higher-resolution or multi-image settings, making visual-token computation more expensive. This provides a useful stress test for operator-aware skipping because high-resolution visual inputs amplify the cost of repeatedly updating visual-token representations inside the LLM backbone.

Table 9 shows that our method remains effective in this higher-resolution setting. With \(67\%\) TFLOPs, our method preserves \(98.8\%\) average performance and achieves the best results among non-vanilla methods on TextVQA and POPE, while also ranking second on GQA, MMMU, and SQA. It also matches the best OCRBench score. These results indicate that the operator-level redundancy identified in the main experiments is not limited to the three primary backbones, but also appears in LLaVA-NeXT. Even when the visual input is more expensive to process, selectively skipping redundant visual operators can reduce computation while maintaining strong benchmark performance.

10 Calibration Sample Size Study↩︎

We study how the number of calibration samples from each benchmark affects the analysis decisions made by our method. For each of the seven calibration benchmarks used in the main experiments, we randomly sample \(8\), \(16\), or \(20\) examples and estimate the layer-wise operator risks for LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B. The resulting risk profile determines the selected layers and their operator-skipping policies, and the same policy is then evaluated on the full benchmark suite.

Table 6 reports the average retention under different layer budgets. The goal is to test whether using more calibration samples leads to a substantially more reliable policy. Since calibration is used only for risk estimation and does not require labels or model training, a small sample size is desirable if it already supports stable policy selection.

The results show that increasing the number of calibration samples can bring improvements at some layer budgets, but the gains are not consistently large or monotonic. For example, more samples slightly improve several LLaVA-1.5-7B settings, while the trends for Qwen2.5-VL-7B and Qwen3-VL-8B vary across budgets. This suggests that \(8\) samples per benchmark are already sufficient to support the risk analysis used by our method. Adding more samples can provide stronger evidence in some cases, but it is not always necessary for obtaining stable operator-skipping decisions.

11 Ablation on Risk-Based Layer Allocation↩︎

We compare our budgeted layer allocation with a simple last-layer heuristic. As described in Section 3.2, our policy first ranks layers by the summed single-operator risk \(\mathcal{R}_{\mathcal{A}}(l)+\mathcal{R}_{\mathcal{F}}(l)\) and allocates the skipping budget to the lowest-risk layers. This design assumes that layers with smaller summed risk can safely discard their non-dominant visual operator earlier, whereas layers with larger summed risk should be postponed because even their non-dominant operator may still affect the output.

This ablation isolates the effect of this layer-allocation rule. For both methods, we keep the per-layer operation type fixed; each selected layer uses the same skip-operator decision or the same frozen update determined by the operator-risk profile. The only difference is which layers are selected. Our method selects the 20 layers with the smallest \(\mathcal{R}_{\mathcal{A}}(l)+\mathcal{R}_{\mathcal{F}}(l)\), while the Last-20 heuristic directly selects the final 20 Transformer layers regardless of their risk scores. Table 8 reports the Qwen3-VL-8B comparison, testing whether risk-based allocation is preferable to choosing late layers purely by position.

The results support our hypothesis. With the same number of selected layers and the same per-layer operator actions, risk-based allocation consistently outperforms the Last-20 heuristic across all reported benchmarks. This indicates that small-\(\mathcal{R}_{\mathcal{A}}+\mathcal{R}_{\mathcal{F}}\) layers are indeed safer places to discard non-dominant visual operators, while simply choosing the final layers may remove useful visual computation whose operator risk remains high.

12 Different Budget of Our Method↩︎

Table [tbl:tab:app95budget] studies how the operator-skipping budget affects performance across four MLLM backbones. Overall, the results show that our operator-aware policy is not tied to a particular architecture, LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and Qwen3-VL all maintain high average retention under moderate budgets, despite their different visual encoders, language backbones, and dependency stacks. In particular, small and medium budgets usually preserve nearly all vanilla performance, and several individual benchmarks even improve over the budget-0 setting. This suggests that the skipped operators are often answer-silent rather than uniformly useful, supporting the reliability of the proposed answer-observable selection criterion.

The budget trend also reveals a consistent efficiency–accuracy pattern. As the budget increases, average retention decreases smoothly rather than collapsing abruptly, indicating that the policy removes redundant visual computation in a controlled manner. This behavior is especially important for practical deployment, users can choose conservative budgets when accuracy is the priority, or larger budgets when higher efficiency is needed. Across all models, moderate budgets retain strong performance, while aggressive settings expose the trade-off when too much visual computation is skipped. These results show that operator-level visual skipping is broadly applicable and robust under different layer-level budgets.

References↩︎