Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts
April 09, 2026
Abstract
Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.
1 Introduction↩︎
Mixture-of-Experts (MoE) architectures [1] have become the dominant paradigm for scaling large vision-language models [2]–[6], powering a wide range of downstream multimodal applications [7], [8]. By activating only a sparse subset of experts for each input, MoE models efficiently handle the intricate interactions between visual and textual information while maintaining computational tractability. However, beneath this success lies a puzzling phenomenon that challenges our fundamental understanding of how these models integrate perception and reasoning.
Consider a simple scenario illustrated in Figure 1 and Appendix 8: when presented with a grade-school mathematics problem [9] as an image [10], Qwen3-VL-30B-A3B [11] accurately extracts all numerical values and textual content, yet produces an incorrect answer due to reasoning errors. When the identical problem is presented as pure text, the same model solves it correctly with ease. We term this phenomenon Seeing but Not Thinking: the model perceives visual content accurately but fails to reason correctly, despite possessing the requisite capability on semantically equivalent text inputs. This raises a fundamental question: what factor causes multimodal MoE models to fail at reasoning when visual inputs are correctly perceived?
To systematically quantify this phenomenon while minimizing interference from perceptual errors, we construct rigorously controlled experiments based on the MATH500 [12] dataset. We render all pure-text problems as high-resolution images (detailed in Appendix 9) to ensure visual inputs are clear and legible. To pinpoint the source of these failures, we conduct error analysis on samples that succeed on text but fail on images. The results are striking: 68.2% to 73.1% of failures stem from reasoning errors, while only 26.9% to 31.8% are attributable to perception errors (Table 6). This confirms that visual inputs degrade reasoning performance even when the content is correctly perceived. This finding aligns with conclusions from recent benchmarks [13], [14].
A natural hypothesis is cross-modal semantic alignment failure: visual information, though correctly perceived, may fail to align with the textual semantic space at the representation level. Prior work has demonstrated that dense-architecture VLMs achieve cross-modal semantic sharing [15], [16], but whether MoE-based VLMs possess the same property remains unexplored. We design cross-modal concept intervention experiments that manipulate hidden state representations across modalities. Our results reveal a clear inverted U-shaped pattern across layers: intervention success rates are low in early layers where visual features have not yet aligned, peak in middle layers where semantic sharing occurs, and decline in terminal layers where output distributions are already determined. This finding confirms that MoE architectures also exhibit cross-modal semantic sharing, indicating that semantic alignment failure alone cannot account for the observed reasoning degradation.
If semantic alignment is not the primary bottleneck, what other factors might contribute? We examine the routing mechanism, the core component distinguishing MoE from dense models. Through systematic analysis of expert activation patterns, we uncover two critical findings. First, experts exhibiting high activation on visual inputs concentrate in early and terminal layers, while domain-specific reasoning experts cluster in middle layers. Second, image inputs induce significant routing divergence from text inputs precisely in these middle layers. Crucially, greater routing divergence correlates with lower reasoning accuracy across our controlled conditions. These observations lead us to propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts, instead directing computation toward less suitable experts. The visual modality does not impair intrinsic reasoning capabilities; rather, it causes suboptimal expert selection, preventing full utilization of domain-specific reasoning capacity.
To validate this hypothesis, we design a routing-guided intervention that enhances domain expert activation during inference. The core idea is straightforward: if routing distraction significantly contributes to reasoning failures, then explicitly increasing the activation weights of domain-relevant experts should recover reasoning performance. We evaluate on three MoE models across six benchmarks spanning semantically equivalent and natural visual scenarios. The soft routing guidance strategy yields consistent improvements. Even on tasks involving complex geometric figures and function graphs where visual information cannot be replaced by text, enhancing domain expert activation helps models better integrate perception with reasoning, with gains of up to 3.17%. Further analysis reveals that domain expert identification is robust to information completeness of text references: as long as the reference elicits target domain reasoning patterns, identified experts transfer effectively to visual tasks with different information structures. This suggests that expert identification locates computational units responsible for cognitive functions, rather than memorizing sample-specific solutions.
Our contributions are threefold. First, we systematically characterize the Seeing but Not Thinking phenomenon and demonstrate its prevalence across multiple state-of-the-art multimodal MoE models. Second, we provide mechanistic insights into this phenomenon through the Routing Distraction hypothesis, revealing the layer-wise separation between visual and domain experts and the routing divergence induced by visual inputs. Third, we propose and validate a routing-guided intervention method that effectively mitigates routing distraction, achieving consistent improvements across diverse benchmarks and model scales.
2 Related Work↩︎
2.0.0.1 Multimodal Semantic Sharing
Vision-language models achieve cross-modal understanding by connecting visual encoders with large language models. Although image and text embeddings exhibit separated distributions in the shared space [17], [18], semantic sharing arise at deeper representation levels. Image representations can transfer to frozen language models through a single linear projection [19], and representations from different modalities may converge toward a shared statistical model [20]. In VLMs specifically, visual and text tokens activate similar LLM weights despite being distinct in representation space [16], and semantically equivalent text-image inputs can be aligned into modality-invariant task vectors [21]. More broadly, shared semantic spaces have been observed across diverse modalities and languages [15], [22], [23].
2.0.0.2 MoE Expert Specialization
Mixture-of-Experts models scale effectively through sparse activation. Research has revealed functional-level expert roles: cognitive experts control meta-level reasoning operations [24], and safety-related refusal behavior concentrates in a small number of experts [25]. Different domains activate different expert subsets, with such associations emerging early in pretraining [26], [27]. Expert differentiation also increases with layer depth [28].
2.0.0.3 Routing Intervention
Inference-time routing intervention [29], [30] has emerged as a promising direction. R2-T2 refines expert selection by shifting routing weights toward correctly predicted samples [31]. SCMoE enhances reasoning by contrasting output distributions between selected and unselected experts [32]. SteerMoE identifies key experts through routing differences and adjusts routing logits for lightweight behavior control [33]. Dynamic routing mechanisms can also enable complex tasks to activate more experts [34].
3 Analyzing Routing Distraction↩︎
We systematically probe the mechanisms underlying the Seeing but Not Thinking phenomenon. Our analysis proceeds in three stages: we first verify that cross-modal semantic sharing exists in MoE architectures (§3.1), then examine the spatial distribution of specialized experts across layers (§3.2), and finally characterize routing behavior differences across modalities (§3.3). All experiments are conducted on Qwen3-VL-30B-A3B using controlled datasets that minimize perceptual confounds.
3.1 Cross-Modal Semantic Sharing in MoE↩︎
3.1.0.1 Experimental Design.
To examine whether cross-modal semantic sharing exists in MoE architectures, we design a concept intervention experiment that directly alters hidden states across modalities. We construct an arithmetic completion task where the input consists of a digit image followed by a textual arithmetic expression (e.g., an image of "3" followed by "+ 2 ="). We extract hidden state vectors for the source digit \(S\) and target digit \(T\) from pure text inputs, denoted as \(\mathbf{h}_{src}^{(l)}\) and \(\mathbf{h}_{tgt}^{(l)}\) at layer \(l\). We then perform the following intervention on the hidden states of image tokens: \[\hat{\mathbf{h}}_{img}^{(l)} \leftarrow \mathbf{h}_{img}^{(l)} - \alpha \cdot \mathbf{h}_{src}^{(l)} + \alpha \cdot \mathbf{h}_{tgt}^{(l)}\] where \(\alpha = 1\) controls intervention strength. This operation removes the source concept’s semantic vector from the image representation while adding the target concept’s vector. If the model’s output changes to match the answer for the target digit, the intervention is deemed successful. We randomly generate 100 test instances with simple digit images to ensure perception is not a confounding factor.
3.1.0.2 Results.
Figure 3 (a) shows intervention success rates across layers. The results exhibit a clear inverted U-shaped pattern: early layers show low success rates, indicating that visual features have not yet aligned with the textual semantic space; middle layers (8–42) show significantly elevated success rates exceeding 90%, suggesting that the two modalities achieve substantial semantic sharing in this region; terminal layers show a sharp decline in intervention effectiveness, likely because the model has already committed to its output distribution. This pattern confirms that MoE-based vision-language models exhibit cross-modal semantic sharing in middle layers, consistent with findings in dense architectures [15], [16]. This result rules out semantic alignment failure as the sole explanation for Seeing but Not Thinking, motivating us to explore other factors specific to MoE architectures.
Figure 3: Analysis of routing mechanisms in multimodal MoE models. (a) Cross-modal semantic sharing verification showing inverted U-shaped intervention success rates. (b) Expert specialization quantification using Gini coefficients. (c) Routing divergence across modalities for three image versions..
3.2 Layer-wise Expert Specialization↩︎
Having established that semantic alignment is preserved across modalities, we now investigate another distinctive aspect of MoE architectures: expert specialization. We examine how different types of experts are distributed across layers and whether this distribution reveals structural patterns relevant to the observed phenomenon.
3.2.0.1 Quantifying Expert Specialization.
We first measure the degree of expert specialization at each layer using the Gini coefficient. For a sequence of length \(L\), let \(\mathbf{p}_{l,t}\) denote the expert probability vector for the \(t\)-th token at layer \(l\). The average expert importance at layer \(l\) is \(\mathbf{q}_l = \frac{1}{L}\sum_{t=1}^{L}\mathbf{p}_{l,t}\), with \(q_{l,i}\) representing the importance of expert \(i\). The Gini coefficient is computed as: \[\small G_l = \frac{\sum_{i=1}^{E}\sum_{j=1}^{E}|q_{l,i} - q_{l,j}|}{2E\sum_{k=1}^{E}q_{l,k}}\] where \(E\) is the total number of experts. Higher values indicate greater concentration of computation among fewer experts. As shown in Figure 3 (b), early layers exhibit lower Gini coefficients while middle and terminal layers show elevated values, indicating that expert functional specialization intensifies in deeper layers.
3.2.0.2 Identifying Domain Experts.
To locate experts responsible for domain-specific reasoning, we compare activation frequencies between domain data and general data. Define the Top-K activation frequency of expert \(E_{l,i}\) on dataset \(D\) as: \[\small \Phi(E_{l,i}, D) = \frac{1}{N_D}\sum_{t \in D}\mathbf{1}[E_{l,i} \in \text{TopK}(x_t)]\] where \(N_D\) is the total token count. The frequency difference \(\Delta\Phi_{l,i} = \Phi(E_{l,i}, D_{\text{dom}}) - \Phi(E_{l,i}, D_{\text{gen}})\) captures domain-specific activation patterns. Experts with \(\Delta\Phi_{l,i} > \tau\) are set as domain experts. Using GSM8K as domain data and Alpaca as general data with \(\tau = 0.3\), we find that math experts cluster predominantly in middle layers (Figure 4).
3.2.0.3 Identifying Visual Experts.
To locate experts associated with visual processing, we render Alpaca text as images and compute frequency differences between the image and text versions. With \(\tau = 0.2\) (we also tried \(\tau = 0.3\) but found too few experts), we identify visual experts that are preferentially activated for image inputs. These experts concentrate in early and terminal layers, exhibiting minimal overlap with math experts in middle layers.
Figure 4: Layer-wise distribution of domain experts and visual experts. Left: Heatmap showing activation frequency differences (red: higher on math data; blue: higher on general data), with deep red concentrated in layers 6–42. Right: Expert counts per layer, where Overlap indicates experts identified as both math and visual experts..
3.2.0.4 Key Finding.
This analysis reveals a critical structural trait: visual experts and domain experts exhibit layer-wise separation. Visual experts cluster in early layers (for initial visual encoding) and terminal layers (preparing modality-specific outputs), while domain experts concentrate in middle layers where cross-modal semantic sharing occurs (§3.1). This spatial segregation raises a natural question: when processing visual inputs, does the routing mechanism in middle layers adequately activate the domain experts necessary for reasoning? We investigate this question in the following section.
3.3 Routing Divergence Across Modalities↩︎
The layer separation between visual and domain experts implies routing behavior in middle layers may be critical for reasoning performance. We now directly examine how expert activation patterns differ between image and text inputs, and whether such differences correlate with reasoning degradation.
3.3.0.1 Experimental Setup.
We conduct analysis on the MATH500 dataset with semantically equivalent text and image versions. Three image versions (v1/v2/v3) are constructed with increasing visual complexity. Error analysis confirms that reasoning errors rather than perception errors dominate these failures (Table 6), ensuring strict control over perceptual factors. To explicitly measure how visual inputs alter expert selection compared to text inputs for the same problem, we calculate the divergence at the sample level. Let \(\Phi_l(x)\) denote the expert activation frequency distribution at layer \(l\) for input \(x\). We quantify the average routing divergence as: \[\small Div_l = \frac{1}{N} \sum_{i=1}^{N} JSD(\Phi_l(x_i^{txt}), \Phi_l(x_i^{img}))\] where \(N\) is the total sample count, and \(JSD(\cdot)\) represents the Jensen-Shannon Divergence. JSD is computed over prompt-phase tokens only, as generation-phase tokens are exclusively textual.
3.3.0.2 Results.
Figure 3 (c) shows JSD across layers for the three image versions. Two patterns emerge. First, JSD exhibits a U-shaped distribution: early and terminal layers show larger divergence (expected due to visual encoding and output preparation), while middle layers show smaller divergence. Second, and more critically, the three curves diverge primarily in the middle layer (6–42) region while remaining nearly identical in early and terminal layers. This indicates that visual complexity predominantly affects routing behavior in middle layers, precisely where domain experts concentrate.
3.3.0.3 Correlation with Reasoning Accuracy.
Despite similar perception error rates, the three image versions exhibit different reasoning performance: v1/v2/v3 achieve 89.0%/88.2%/87.4% respectively, versus 92.8% for pure text. Notably, versions with lower reasoning accuracy exhibit greater JSD in middle layers. This correlation suggests that routing divergence in middle layers, rather than perceptual quality, contributes to reasoning degradation.
3.3.0.4 The Routing Distraction Hypothesis.
Synthesizing the findings from §3.1–§3.3, we propose the Routing Distraction hypothesis: when processing visual inputs, the MoE routing mechanism fails to adequately activate task-relevant domain experts in middle layers, instead directing computation toward other experts less suited for the reasoning task. This hypothesis explains why models can perceive correctly yet reason incorrectly: semantic alignment is preserved, but the computational resources required for reasoning are not fully mobilized. While our controlled analysis focuses on semantically equivalent scenarios where perception is not a confounding factor, we expect this hypothesis to generalize to natural visual scenarios involving complex figures and diagrams. In the following sections, we validate this hypothesis across both scenario types.
4 Routing-Guided Intervention↩︎
Based on the Routing Distraction hypothesis, we propose a simple intervention strategy: if insufficient activation of domain experts contributes to reasoning failures, then explicitly enhancing their routing weights should recover performance.
4.1 Domain Expert Identification↩︎
Following the method in §3.2, we find domain experts by comparing activation rates between domain-specific data and general-purpose data (Alpaca). This requires constructing text references that elicit the target domain’s reasoning patterns.
Text reference construction depends on the scenario type. For semantically equivalent scenarios, we directly use the original text problems before rendering. For natural visual scenarios, we adopt task-appropriate proxies that elicit similar domain reasoning patterns, such as text-only problem versions or model-generated descriptions (§5.1).
Given text references, we compute \(\Delta\Phi_{l,i}\) for each expert and apply threshold \(\tau\) to obtain the domain expert set \(\mathcal{E}_{\text{domain}}\). This procedure requires only 20 randomly sampled examples. Notably, the model need not solve these samples, nor must the text reference be strictly equivalent to the visual task. We analyze this robustness property in §5.3.0.1.
4.2 Routing Weight Adjustment↩︎
During inference, we enhance routing weights of identified domain experts. We investigate two strategies with a random baseline as control.
4.2.0.1 Soft Intervention.
We apply moderate additive enhancement to domain expert logits: \[r'_{l,k} \leftarrow r_{l,k} + \lambda \cdot s(r_l), \quad \forall E_{l,k} \in \mathcal{E}_{\text{domain}}\] where \(r_{l,k}\) is the original routing logit, \(s(r_l)\) is the standard deviation of all expert logits at layer \(l\), and \(\lambda\) is the enhancement coefficient. This formulation preserves the router’s flexibility to adjust based on specific inputs while systematically increasing domain expert activation probability.
4.2.0.2 Hard Intervention.
We force domain expert logits to the layer maximum: \[r'_{l,k} \leftarrow \max_j r_{l,j} + \delta, \quad \forall E_{l,k} \in \mathcal{E}_{\text{domain}}\] where \(\delta \sim \mathcal{N}(0, 10^{-4})\) introduces small perturbations to prevent identical logits, particularly relevant for architectures with Top-1 routing.
4.2.0.3 Random Baseline.
To verify that improvements stem from activating specific domain experts rather than routing perturbation itself, we randomly select the same number of experts at each layer and apply identical enhancement as Soft intervention.
5 Experiments↩︎
5.1 Experimental Setup↩︎
5.1.0.1 Models.
We evaluate three multimodal MoE models spanning different scales: Qwen3-VL-30B-A3B [11], Kimi-VL-16B-A3B [35], and Llama4-Scout-109B-A17B [36]. Detailed model architectures are provided in Appendix 10.
5.1.0.2 Benchmarks.
We construct two complementary evaluation scenarios. Semantically Equivalent Scenarios include MATH500 and GPQA-Diamond (chemistry, physics subsets) [37] rendered as high-resolution images, providing controlled environments where perception is not a confounding factor. Natural Visual Scenarios include MathVerse (vision-only version) [38], MATH-Vision [39], and GSM8K-V [10], with text references constructed from the official text-only version for MathVerse, model-generated problem descriptions for MATH-Vision (Appendix 11.2), and corresponding GSM8K text problems for GSM8K-V.
5.1.0.3 Implementation.
We use vLLM [40] with EasySteer [41] for inference-time intervention. Intervention configurations are summarized in Table 1. All models use \(\tau = 0.3\) for domain expert identification. Intervention layers are selected based on findings from §3.2: middle layers for Qwen3-VL and Llama4-Scout, early and middle layers for Kimi-VL where domain experts emerge earlier. Due to slight non-determinism in vLLM, we run 16 trials with greedy decoding and report average accuracy verified by xVerify [42]. Additional details are provided in Appendix 12.
| Model | |||
| Layers | \(\bm{\tau}\) | \(\bm{\lambda}\) | |
| Kimi-VL-16B-A3B | [0, 20] | 0.3 | 0.5 |
| Qwen3-VL-30B-A3B | [6, 42] | 0.3 | 0.5 |
| Llama4-Scout-109B-A17B | [8, 40] | 0.3 | 0.2 |
5.2 Main Results↩︎
| Semantically Equivalent Scenarios | Natural Visual Scenarios | ||||||
|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 | Math | Chemistry | Physics | MathVerse | MATH-V | GSM8K-V | Average |
| Baseline | 52.30 | 25.54 | 29.51 | 35.41 | 21.05 | 8.11 | 28.65 |
| Random | 51.64-0.66 | 25.67+0.13 | 28.78-0.73 | 34.39-1.02 | 19.74-1.31 | 6.82-1.29 | 27.84-0.81 |
| Hard | 53.06+0.76 | 25.81+0.27 | 32.78+3.27 | 35.79+0.38 | 21.71+0.66 | 7.13-0.98 | 29.38+0.73 |
| Soft | 54.54+2.24 | 27.89+2.35 | 32.49+2.98 | 38.58+3.17 | 23.36+2.31 | 9.17+1.06 | 31.01+2.36 |
| Baseline | 88.20 | 41.94 | 75.58 | 69.29 | 55.92 | 24.49 | 59.24 |
| Random | 85.18-3.02 | 36.56-5.38 | 70.93-4.65 | 67.78-1.51 | 53.62-2.30 | 23.42-1.07 | 56.25-2.99 |
| Hard | 87.92-0.28 | 33.33-8.61 | 70.93-4.65 | 68.40-0.89 | 54.28-1.64 | 23.65-0.84 | 56.42-2.82 |
| Soft | 89.42+1.22 | 44.09+2.15 | 76.74+1.16 | 71.20+1.91 | 57.57+1.65 | 25.32+0.83 | 60.72+1.48 |
| Baseline | 77.95 | 42.20 | 55.60 | 56.09 | 32.24 | 23.50 | 47.93 |
| Random | 77.80-0.15 | 41.40-0.80 | 54.65-0.95 | 54.57-1.52 | 31.58-0.66 | 22.82-0.68 | 47.14-0.79 |
| Hard | 77.62-0.33 | 37.16-5.04 | 46.73-8.87 | 53.55-2.54 | 30.59-1.65 | 22.44-1.06 | 44.68-3.25 |
| Soft | 79.20+1.25 | 43.01+0.81 | 56.98+1.38 | 57.11+1.02 | 33.88+1.64 | 24.41+0.91 | 49.10+1.17 |
Table 2 presents reasoning accuracy across all models and benchmarks. We analyze the results from three perspectives.
5.2.0.1 Effectiveness in Semantically Equivalent Scenarios.
These setup directly test the Routing Distraction hypothesis by isolating reasoning from perception. The Soft intervention achieves consistent improvements across all models. Kimi-VL improves from 52.30% to 54.54% (+2.24%) on math and from 29.51% to 32.49% (+2.98%) on physics. Qwen3-VL shows gains of 1.22% on math and 2.15% on chemistry. These results confirm that enhancing domain expert activation effectively mitigates routing distraction when perception is fixed.
5.2.0.2 Generalization to Natural Visual Scenarios.
A key question is whether routing distraction extends to scenarios with complex visual elements. Kimi-VL rises from 35.41% to 38.58% (+3.17%) on MathVerse; Qwen3-VL improves from 55.92% to 57.57% (+1.65%) on MATH-Vision. These gains indicate enhancing domain expert activation helps models better integrate visual information. Improvements on GSM8K-V are modest, suggesting that when task difficulty concentrates in the perception phase (extracting information from multiple scene images), routing guidance offers less benefit.
5.2.0.3 Comparison of Intervention Strategies.
Soft intervention performs most consistently, balancing enhanced domain expert activation with preserved routing flexibility. Hard intervention shows mixed results: while it sometimes yields improvements (e.g., Kimi-VL on physics), it causes severe degradation on Llama4-Scout, even underperforming the Random baseline. We attribute this to Llama4’s Top-1 routing mechanism, where forcing expert logits throughout generation frequently disrupts the original routing decisions. The Random baseline generally produces no improvement, confirming that gains stem from activating correct domain experts rather than routing perturbation.
5.3 Analysis↩︎
| Model | Text | Vision | w/ Guidance | |
|---|---|---|---|---|
| 4-5 | \(\text{Ref}_{\text{MathVerse}}\) | \(\text{Ref}_{\text{GSM8K}}\) | ||
| Kimi-VL | 45.94 | 35.41 | 38.58+3.17 | 37.31+1.90 |
| Qwen3-VL | 67.26 | 69.29 | 71.20+1.91 | 69.54+0.25 |
| Llama4 | 63.32 | 56.09 | 57.11+1.02 | 55.08-1.01 |
5.3.0.1 Robustness of Expert Identification.
A critical question is whether domain expert identification demands semantically equivalent text references. This is particularly relevant for natural visual benchmarks where perfect text equivalents do not exist. Taking MathVerse as an example, its official text-only version cannot fully convey spatial relationships and geometric configurations, rendering many problems unsolvable from text alone. Indeed, Qwen3-VL achieves 67.26% accuracy on text-only, while achieving 69.29% on vision-only, indicating that visual information provides independent value.
However, experts identified using this partial text reference still yield substantial improvements: vision-only accuracy rises to 71.20% (+1.91%) with Soft intervention. This result carries two implications. First, visual information provides unique value that text misses; routing guidance helps models leverage this information rather than regressing to text-based reasoning paths. Second, expert identification is robust to information completeness. The procedure locates computational units responsible for domain-specific cognitive functions, not sample-specific solution paths. As long as the text reference elicits target domain reasoning patterns, identified experts transfer effectively to visual tasks with different information structures.
5.3.0.2 Domain Specificity.
While expert identification is robust to information completeness, it remains sensitive to domain mismatch. When using GSM8K (elementary arithmetic) instead of MathVerse text-only as the reference for MathVerse evaluation, improvements diminish substantially or even reverse. As shown in Table 3, Qwen3-VL’s vision-only accuracy rises from 69.29% to 71.20% (+1.91%) with MathVerse-matched guidance, but only reaches 69.54% (+0.25%) with GSM8K-mismatched guidance. Notably, Llama4 shows degradation: from 56.09% baseline to 57.11% (+1.02%) with matched guidance, but drops to 55.08% (-1.01%) when mismatched. Although both datasets involve mathematics, GSM8K emphasizes arithmetic operations while MathVerse requires geometric reasoning and function analysis. The experts activated by these distinct patterns do not fully overlap, confirming that effective intervention requires domain-matched references.
| Model | Baseline | w/ Early | Middle | w/ Late |
|---|---|---|---|---|
| Kimi-VL | 28.65 | 31.01 | 29.82 | 29.35 |
| Qwen3-VL | 59.24 | 59.76 | 60.72 | 58.97 |
| Llama4 | 47.93 | 46.85 | 49.10 | 47.82 |
5.3.0.3 Layer Selection.
We examine the impact of intervention layer range. As shown in Table 4, Qwen3-VL and Llama4-Scout achieve optimal performance when intervening only on middle layers. Adding early layers degrades performance, likely because visual experts in these layers handle necessary visual feature extraction; premature intervention disrupts this processing. Kimi-VL exhibits a different pattern, benefiting from early and middle layer intervention. Analysis reveals that Kimi-VL’s domain experts and cross-modal semantic sharing both emerge at earlier layers, making early intervention beneficial for this architecture.
5.3.0.4 Intervention Strength.
Figure 5 shows the effect of enhancement coefficient \(\lambda\) in Soft intervention. Kimi-VL and Qwen3-VL achieve optimal performance with \(\lambda \in [0.4, 0.6]\); excessive values degrade accuracy by overriding input-specific routing decisions. Llama4 requires weaker intervention (\(\lambda=0.2\)), due to its Top-1 routing mechanism where activating only one expert per layer makes routing decisions more sensitive to logit changes.
6 Conclusion↩︎
This paper investigates the Seeing but Not Thinking phenomenon in multimodal MoE models. We establish that cross-modal semantic sharing exists in MoE architectures, ruling out alignment failure as the sole explanation. Our analysis reveals layer-wise separation between visual and domain experts, with image inputs inducing routing divergence in middle layers that correlates with reasoning degradation. Based on these findings, we propose the Routing Distraction hypothesis and validate it through routing-guided intervention, achieving consistent improvements across three models and six benchmarks, with gains of up to 3.17% on complex visual reasoning tasks. Our work provides both mechanistic insights into multimodal reasoning failures and a practical method for mitigation.
Limitations↩︎
This work has several limitations. First, routing guidance cannot address perceptual errors; when visual information is incorrectly extracted, enhancing domain expert activation provides no benefit. The method targets the specific failure mode of correct perception coupled with reasoning failure, not the broader challenge of visual understanding.
Second, the current approach requires task-specific configuration. Domain expert identification relies on constructing appropriate text references, and optimal intervention layers and strengths must be determined empirically for each model-task combination. Developing adaptive methods that automatically identify relevant experts and calibrate intervention parameters would substantially improve practical applicability.
Third, our cross-modal semantic sharing experiments verify alignment for simple numerical concepts. Whether complex visual concepts (spatial relationships, geometric configurations, abstract diagrams) achieve equal alignment quality remains unclear. Insufficient alignment for complex concepts could constitute an additional factor contributing to reasoning failures beyond routing distraction.
Finally, while we demonstrate correlation between routing divergence and reasoning degradation, establishing strict causality would require more controlled interventions. The consistent improvements from routing guidance provide supporting evidence, but the precise causal mechanisms underlying expert selection in multimodal contexts merit further investigation.
7 Discussion↩︎
7.0.0.1 Scope of Routing Distraction.
The routing distraction hypothesis explains cases where perception succeeds but reasoning fails. However, it does not address perceptual errors, nor does it claim to be the sole factor underlying multimodal reasoning failures. For tasks where difficulty concentrates in perception (e.g., GSM8K-V requiring information extraction from multiple scene images), routing guidance provides limited benefit. The phenomenon likely arises from multiple interacting factors, with routing distraction being one identifiable and addressable component.
7.0.0.2 Generality Across Architectures.
Our experiments span three models with different scales and routing mechanisms (Top-K vs. Top-1), demonstrating consistent improvements. However, the optimal intervention parameters (layers, strength) vary across architectures, reflecting differences in where domain experts emerge and how routing decisions propagate. Future work could investigate whether architectural properties predict optimal intervention configurations.
7.0.0.3 From Inference-Time Remedy to Training-Time Solution.
While routing-guided intervention effectively mitigates routing distraction, it remains a post-hoc remedy requiring domain expert identification and hyperparameter tuning for each task. A more fundamental solution lies in the training phase: designing architectures or training objectives that encourage routers to allocate experts based on task requirements rather than input modality. Potential directions include routing regularization that penalizes modality-dependent expert selection, auxiliary losses that align routing distributions across semantically equivalent multimodal inputs or better behavior internalization [43], [44].
8 Case Study↩︎
Figure 1 presents a sample from the GSM8K dataset [9], with the corresponding image from GSM8K-V [10]. The model successfully extracts all necessary information and the question to be answered, but makes a reasoning error during inference by incorrectly counting the initial value as a separate day. Table 5 shows the complete model responses before and after routing-guided intervention.
| Baseline Response (Incorrect: Answer is 4) |
|---|
Of course, let’s solve this problem step-by-step. The scenario involves a beanstalk that doubles in height every day and starts at 4 feet tall. We need to find out how many days it will take for the beanstalk to grow taller than a window located 20 feet off the ground. We can calculate the height of the beanstalk each day until its height exceeds 20 feet. - Day 1: The beanstalk is \(4\) feet tall. - Day 2: It doubles from Day 1: \(4 \times 2 = 8\) feet tall. - Day 3: It doubles from Day 2: \(8 \times 2 = 16\) feet tall. - Day 4: It doubles from Day 3: \(16 \times 2 = 32\) feet tall. Now we compare these heights to the window: - On Day 3, the beanstalk is 16 feet tall, which is less than 20 feet (the window). - On Day 4, the beanstalk is 32 feet tall, which is greater than 20 feet. Therefore, on the fourth day, the beanstalk becomes taller than the window. It takes \(\boxed{\textcolor{red}{4}}\) days for the beanstalk to be taller than the window. |
| With Routing Intervention (Correct: Answer is 3) |
Of course, let’s solve this problem step-by-step. The scenario involves a beanstalk that doubles in height every day and starts at 4 feet tall. We need to find out how many days it will take for the beanstalk to grow taller than a window located 20 feet off the ground. We can model the growth of the beanstalk using an exponential function: - Initial Height (Day 0): \(h_0 = 4\) feet. - Growth Rate: The plant doubles its size each day, so we multiply by 2 daily. - Height on Day n: \(h_n = h_0 \times 2^n\) Let’s calculate the height of the beanstalk over several consecutive days until it surpasses the 20-foot mark: — Step 1: Calculate the height after each day - After Day 1: \(h_1 = 4 \text{ ft} \times 2^1 = 8 \text{ ft}\) - After Day 2: \(h_2 = 4 \text{ ft} \times 2^2 = 4 \text{ ft} \times 4 = 16 \text{ ft}\) - After Day 3: \(h_3 = 4 \text{ ft} \times 2^3 = 4 \text{ ft} \times 8 = 32 \text{ ft}\) — Step 2: Compare with the window height Now, compare these heights to the window’s height of 20 feet: - After Day 1 (\(8\) ft), the beanstalk is shorter than the window (\(8 < 20\)). - After Day 2 (\(16\) ft), the beanstalk is still shorter than the window (\(16 < 20\)). - After Day 3 (\(32\) ft), the beanstalk is now taller than the window (\(32 > 20\)). Therefore, the first time the beanstalk exceeds the height of the window is on the third day. It takes \(\boxed{3}\) days for the beanstalk to be taller than the window. |
9 Construction and Validation of Semantically Equivalent Dataset↩︎
We constructed semantically equivalent images based on the MATH500 dataset. To analyze the relationship between routing divergence and reasoning accuracy in §3.3, we created three versions with different visual styles.
9.0.0.1 Rendering Method.
We used LaTeX to render and crop text problems into high-resolution images (dpi=200). For mathematical formulas, we used the default LaTeX font; for code segments, we used JetBrainsMono-Regular. We constructed three different versions:
v1: Plain text uses the SimHei font (printed style)
v2: Plain text uses the Caveat font (handwritten style)
v3: Adds a light gray grid background to v2
Images of the three versions corresponding to the same text are padded to maintain identical dimensions. Examples are shown in Figure 6. The Alpaca dataset in §3.2 was rendered following the v1 approach, and GPQA-Diamond in §5 was rendered following the v2 approach.
Figure 6: Examples of the three semantically equivalent image versions (v1, v2, v3 from top to bottom) for the same text problem..
9.0.0.2 Verification.
We first tested a strategy of adding a “perform OCR recognition first” instruction in the prompt:
Prompt for OCR First Strategy
OCR first, then solve the problem.However, this strategy did not improve the results (Table 6), possibly because the model focused its attention on OCR rather than reasoning. Therefore, we adopted the approach of directly inputting images. We use
gpt-5.2-1211-global to perform error analysis on all samples where the model answered the text version correctly but the image version incorrectly. Errors are classified into information reading errors and reasoning errors using the following
prompt:
Prompt for Error Classification
You are given:
1) **Text Problem**:
`{text_problem}`
2) **Correct Text Solution** (guaranteed correct):
`{text_solution}`
Now the same problem has been converted into an **equivalent image** (same content, just visual). A model answered based on the image and produced:
3) **Model Answer (from the image)**:
`{model_answer}`
This model answer is **wrong**.
**Task:** Determine the most likely error type:
- **Information Reading Error**: The model misread, overlooked, or failed to extract key information from the image (e.g., missed a condition, misread a number/symbol, ignored part of the diagram/text).
- **Reasoning Error**: The model correctly extracted the relevant information, but made a logical/mathematical mistake in reasoning or computation.
**Instructions:**
1. Briefly list the key information that must be extracted from the problem to solve it.
2. Compare the **Model Answer** with the **Correct Text Solution** and infer whether the model likely failed at reading/extraction or at reasoning.
3. Output **only one label**: `Information Reading Error` or `Reasoning Error`.
4. Then give a **one-sentence justification**.| Dataset | Acc | |||
| (OCR first) | ||||
| Error | ||||
| Error | ||||
| MATH500 | 92.8 | - | - | - |
| MATH500-v1 | 89.0 | 87.4 | 31.8% (7/22) | 68.2% (15/22) |
| MATH500-v2 | 88.2 | 86.8 | 26.9% (7/26) | 73.1% (19/26) |
| MATH500-v3 | 87.4 | 86.8 | 31.0% (9/29) | 69.0% (20/29) |
10 Model Information↩︎
Table 7 provides detailed architectural specifications for the three multimodal MoE models evaluated in this work.
| Model | Layers | ||||
| Experts | |||||
| Experts | |||||
| Experts | |||||
| Experts | |||||
| Kimi-VL | 27 | 64 | 6 | 2 | 8 |
| Qwen3-VL | 48 | 128 | 8 | – | 8 |
| Llama4 | 48 | 16 | 1 | 1 | 2 |
11 Evaluation Benchmarks↩︎
11.1 Benchmark Details↩︎
For MATH500 and GPQA-Diamond, we render all problems as high-resolution images following the v2 approach described in §3.3 to construct semantically equivalent visual inputs.
MATH500 is a curated subset of 500 problems from the MATH dataset.
GPQA-Diamond consists of 198 graduate-level multiple-choice questions in biology, physics, and chemistry. Given the limited number of biology questions (19), we select only the physics (86) and chemistry (93) subsets for evaluation.
MathVerse is a visual math benchmark spanning plane geometry, solid geometry, and function graphs, with six versions offering varying degrees of multimodal information. We adopt its vision-only version (788 test samples) to maximize visual dependency.
MATH-Vision is a collection of 3,040 mathematical problems with visual contexts sourced from real math competitions. We use the mini test set (304 samples) for evaluation.
GSM8K-V systematically transforms GSM8K text problems into purely visual multi-image format, comprising 1,319 samples (5,343 images) that require models to extract information from scene images. We use the implicit version to maximize visual dependency.
11.2 MathVision Description Generation↩︎
For MATH-Vision, we generate textual descriptions of diagrams to construct text references for domain expert identification. We use gpt-5.2-1211-global to generate image descriptions, excluding problem types where answers (such as counting)
are directly evident in the description. The generation prompt follows the template shown below:
Prompt for MathVision Description Generation
Here is a diagram of a math problem, please describe the diagram in complete detail so that your description can be used to replace the diagram for math problem-solving.11.2.0.1 Example.
For the sample image shown in Figure 7, the generated description is:
A horizontal sequence of five identical flower shapes is shown, connected left-to-right by arrows.
- **Flower 1 (far left):** In the center circle is the number **48**.
- An arrow points from Flower 1 to **Flower 2** with the label **-20** above the arrow.
- **Flower 2:** Center circle is blank (no number shown).
- An arrow points from Flower 2 to **Flower 3** with the label **+9** above the arrow.
- **Flower 3:** Center circle is blank.
- An arrow points from Flower 3 to **Flower 4** with the label **-6** above the arrow.
- **Flower 4:** Center circle is blank.
- An arrow points from Flower 4 to **Flower 5 (far right)** with the label **+30** above the arrow.
- **Flower 5:** Center circle contains a **question mark (?)**.
Each flower has a round center and multiple petals (a simple daisy-like icon); only the center contents (48, blanks, and ?) matter for the math.These descriptions are concatenated with the original problem text to extract activation frequencies for domain expert identification.
12 Implementation Details↩︎
Generation parameters are set to: temperature=0 (greedy decoding), zero-shot prompting, and maximum generation length of 8192 tokens, with no external tools enabled. All experiments are conducted on 16 A100 GPUs using vLLM v0.11.0. We wrap the MoE router in vLLM via EasySteer to intercept and modify routing logits during inference.
We observe that Qwen3-VL-30B-A3B-Instruct tends to generate excessively long chains of thought, likely due to long-CoT cold-start during its training phase. To mitigate this, we set repetition_penalty=1.2 for all benchmarks except MATH500.