1 Introduction↩︎

Evaluation of large vision-language models (LVLMs) has shifted from benchmark scarcity to a harder question: how to obtain trustworthy results. Recent benchmarks span mathematical reasoning [1]–[3], document comprehension [4], [5], and other capabilities [6]–[12], offering an important basis for model comparison [13], [14]. However, as benchmark suites grow in number and scale, their usefulness increasingly hinges on quality: whether an item truly requires the image [15], still distinguishes model performances [16], and has reliable ground truth and scoring protocols [17], [18]. When these conditions fail, adding more items increases evaluation cost and can mix non-visual shortcuts, saturated samples, or annotation noise into the final score.

We first conduct an item-level audit of 18 widely used multimodal benchmarks and identify three systematic distortions in existing evaluation signals. The first is weak visual dependence, where models can answer questions from textual cues alone without genuinely understanding image content [15]. The second is item saturation, where many items are solved consistently by models with different capability levels and thus contribute little to model discrimination [16], [19]. The third is pseudo-hardness, where perceived item difficulty arises from incorrect gold answers, ambiguity, or fragile grading mechanisms [18]. These issues affect more than evaluation reliability. Running a single evaluation of Qwen3.5-27B on these 18 benchmarks requires 775 GPU hours, while our audit shows that at least 68.8% of the items are affected by at least one distortion type. In other words, the current evaluation pipeline consumes substantial computational resources while a considerable fraction of the resulting evaluation signals remains distorted.

These issues suggest that reliable multimodal evaluation requires item-level quality control, not merely larger question pools or higher overall difficulty. Following this idea, we propose MMGist, a curated multimodal benchmark built from 18 source benchmarks, containing 7,262 items and covering seven capability dimensions. MMGist maps each type of evaluation distortion to a specific filtering step: text ablation removes items that do not require visual information, cross-model saturation filtering targets items that poorly distinguish current models, and anomaly detection with human review filters out pseudo-difficult items. In other words, MMGist does not simply evaluate fewer items, but concentrates the evaluation budget on items that require image understanding, better distinguish models, and are more reliable.

We comprehensively evaluate 27 leading LVLMs on MMGist. Using only 31.2% of the raw question pool (totaling 23,250 questions), MMGist preserves highly consistent model-performance rankings, with Spearman \(\rho = 0.98\). After filtering, the models’ average score drops by 20.3%, indicating that weakly vision-dependent, saturated, and pseudo-hard questions in the raw pool indeed obscure true model performance. As illustrated in Figure 1, MMGist reduces evaluation cost by approximately 69% while improving cross-model discriminability by 78%, showing that the retained questions more directly characterize model differences. Finally, even the best-performing Gemini-3.1-Pro reaches only 66.8% Macro Avg, demonstrating that MMGist remains sufficiently challenging while being more efficient.

Our main contributions are threefold: 1) we conduct a systematic audit of 23,250 items from 18 widely used vision-language benchmarks and identify three pervasive issues: weak visual dependency, item saturation, and pseudo-hard questions; 2) we propose a reusable item-level quality-control pipeline combining text-only ablation filtering, saturation filtering, rule-based anomaly recall, multi-model adjudication, and human expert review; 3) we construct MMGist, a curated benchmark with 7,262 items spanning seven capability dimensions. Experiments on 27 LVLMs show that MMGist enables cost-efficient evaluation, ranking consistency with the raw pool, and more discriminative performance comparison.

2 Related Work↩︎

2.1 Evaluation Benchmarks for LVLMs↩︎

The rapid advancement of large vision-language models (LVLMs) has produced a broad ecosystem of multimodal evaluation benchmarks [13], [14]. Comprehensive benchmarks such as MMBench, MMMU, and MMMU-Pro offer unified testbeds for general perception, reasoning, and multidisciplinary knowledge [6], [7], [20], while newer benchmarks target specialized skills such as mathematical and scientific reasoning [1]–[4], [8], spatial understanding [21], [22], visual logic [9], [23], visual perception and hallucination detection [10], [24], [25], document and OCR understanding [5], and medical diagnosis [11], [12]. These benchmarks underpin LVLM evaluation, yet they often assume that each item is visually necessary, discriminative, and reliably annotated. Our work starts from this assumption and asks whether item-level quality in existing benchmarks is sufficient to support reliable multimodal evaluation.

2.2 Benchmark Quality and Reliability↩︎

Recent work has identified several benchmark-quality risks, but usually studies them in isolation. MMStar and other studies [15], [26]–[28] show that many multimodal items can be answered without images, and related shortcut and contamination studies show that question cues, world knowledge, or training-data overlap may inflate scores. Other work highlights saturation, where already-solved items add cost but little discrimination [8], [16], and annotation or scoring errors, where even small label-noise rates can change model rankings [17], [18]. Harder or dynamic benchmarks, meta-evaluation frameworks, and IRT-based methods improve evaluation, but mainly by adding new items, analyzing individual benchmarks, or targeting one quality dimension [19], [29]–[33]. In contrast, MMGist targets existing multimodal benchmarks with a unified item-level quality-control pipeline that combines text-ablation filtering, cross-model saturation filtering, multi-model adjudication, and expert review to address visual dependency, saturation, and anomalous items jointly.

3 Problems in Existing Vision-Language Benchmarks↩︎

3.1 Weak Visual Dependency↩︎

Vision-language benchmarks evaluate the multimodal capabilities of LVLMs, assuming that correctly answering each item requires understanding the visual content. However, we find that many items can be answered without accessing the image. We trace this weak visual dependency to two primary sources: (1) answer leakage from the question text, where phrasing or options make only one answer semantically plausible regardless of the image; and (2) world knowledge embedded in LLMs, where the question targets factual or commonsense knowledge requiring no image information [15]. Both pathways enable correct answers through language-only reasoning, undermining the validity of multimodal evaluation.

To quantify this issue, we use five models from different providers (Qwen3.6-35B-A3B, GPT-5-Mini, Gemini-3.1-Flash-Lite, Doubao-Seed-2.0-Mini, and Claude-Haiku-4.5) [34]–[38] as text-only inspectors. Each inspector receives only the image-free question text, and we sample eight responses per item to compute the per-item text-only accuracy. The problem is pervasive: AI2D [4] exhibits the highest text-only accuracy at 66.7%, indicating that its items are largely solvable without the diagram. Even expert-knowledge benchmarks, including MMMU [6] (51.2%), show substantial text-only solvability. This indicates that widely used benchmarks often do not require truly visual understanding.

3.2 Item Saturation↩︎

Beyond visual dependency, many benchmark items are now trivially solvable, with negligible discriminative value among current models. We define an item as saturated when models spanning a wide capability range consistently solve it. To assess saturation, we evaluate all 23,250 items with 12 LVLMs from five providers (Anthropic, OpenAI, Google, Alibaba, ByteDance), covering different level models (Appendix Table [tab:saturation-panel]).

The results reveal that nearly half the items have lost discriminative power. Across all 18 benchmarks, 49.7% of items exceed 90% average accuracy. These items compress the score range, obscuring capability differences and inflating cost.

3.3 Pseudo-hard Questions↩︎

Not all items that models answer incorrectly are truly challenging. Some failures arise from item-level defects rather than limited model capability. Label errors are well documented in NLP benchmarks: Northcutt et al. [18] reported an average label-error rate of 3.3% across ten major benchmarks, enough to alter model rankings. We observed similar signals in vision-language benchmarks: among 23,250 items, 695 received no correct answer from all 12 models, and 27.8% were manually verified as item-level defects.

We refer to such items as pseudo-hard and group them into three types. Ground-truth errors occur when the annotated answer is incorrect, penalizing models that reason correctly. Ambiguous items involve underspecified questions, non-unique answers, or insufficient evidence, making any answer debatable. Scorer risks arise when the scoring protocol cannot reliably match model outputs to the ground truth because of synonymous expressions, mathematical equivalence, rounding discrepancies, or response format mismatches. These pseudo-hard questions distort evaluation by penalizing models for correct reasoning and inflating the apparent difficulty of benchmarks. Together with weak visual dependency and item saturation, these three issues motivate a systematic quality control pipeline.

4 Constructing MMGist↩︎

Using the three quality issues identified in §3, we construct MMGist from 18 source benchmarks via a multi-stage pipeline (Figure 2). The pipeline applies three successive filters, each targeting one issue: text-only ablation filtering for visual necessity (§4.1), saturation filtering to preserve discriminative value (§4.2), and anomaly detection with review to enforce item integrity (§4.3)².

4.1 Text-only Ablation Filtering↩︎

Using the text-only inspection setup from §3.1, we compute average accuracy over the five models per item and remove items above type-dependent thresholds: 80% for binary (yes/no) items and 50% for others. This stage removes 9,465 items, 40.7% of the original pool.

4.2 Saturation Filtering↩︎

To remove saturated items, we apply the 12 models in §3.2 to the 13,785 samples remaining from the previous step. Each model samples each item eight times under the image-conditioned setting, and we compute the average accuracy across models. Items with an average accuracy above 90% are classified as saturated and removed ³. This stage leaves 8,831 items; CountBench loses 74.1% of its remaining items, while benchmarks targeting harder reasoning tasks, such as MME-SCI (1.0%) and ZeroBench (0.0%), are minimally affected.

4.3 Anomaly Detection and Review↩︎

The final stage targets pseudo-hard questions, whose difficulty stems from ground-truth errors, ambiguity, or scorer risks rather than from genuine visual-language challenges. We adopt a three-step approach: rule-based recall, multi-model adjudication, and human review.

4.3.0.1 Rule-based recall.

We designed five heuristic rules to flag suspicious items (Table 1). Each targets a specific failure mode: items scored zero by all 12 models may have ground-truth errors; items whose majority vote consistently disagrees with the labeled answer may contain annotation errors; high inter-model disagreement or near-tied top-vote answers suggest ambiguity; and abnormally low answer-extraction rates indicate scoring-compatibility issues. Applying these rules to 8,831 saturation-filtered samples yielded 5,477 candidates for further review.

4.3.0.2 Multi-model adjudication.

We employ five LVLMs from different providers (Claude-Sonnet-4.6, Gemini-3.1-Pro, GPT-5, Qwen3.6-Plus, and Doubao-Seed-2.0-Pro) [34]–[38] as independent judges in a two-round adjudication process [39]. In the first round, each judge independently reviews every candidate item, receiving the image, question text, annotated ground truth, and model responses from the sampling stage, and assigns one of four labels: GT error, ambiguous question, scorer risk, or no issue. In the second round, each judge receives the aggregated first-round judgments from all five models and re-evaluates the item using the collective evidence. The final decision is reached by majority voting over the second-round labels: an item is confirmed as problematic when at least three judges agree on the non-trivial label.

4.3.0.3 Human expert review.

All items deemed problematic by multi-model adjudication are further reviewed by human experts, who verify whether each flagged item is flawed or represents a genuinely difficult challenge. After human review, 1,569 items are confirmed and removed from the benchmark.

4.4 Final Benchmark↩︎

After all three filtering stages, MMGist retains 7,262 items from 18 source benchmarks, representing 31.2% of the original 23,250 items; per-benchmark breakdowns appear in Appendix Table [tab:composition]. The retained items span seven broad capability categories (Figure 3): STEM Reasoning (MathVista, MathVision, DynaMath, MME-SCI) [1]–[3], [8], Expert Knowledge (MMMU, MMMU-Pro) [6], [7], Visual Logic (LogicVista, zerobench, ZeroBench-Sub) [9], [23], Visual Perception (BLINK, HallusionBench, RealWorldQA) [10], [24], [25], Diagram & OCR (AI2D, OCRBench) [4], [5], Spatial Understanding (EmbSpatialBench, CountBench) [21], [22], and Medical (SLAKE, MedXpertQA) [11], [12]. Retention rates vary substantially across benchmarks, from 96.0% for ZeroBench, whose items are inherently visually grounded and challenging, to 15.7% for MMMU, reflecting genuine differences in item quality across existing evaluation suites.

5 Experiments↩︎

4.5pt

5.1 Experimental Setup↩︎

5.1.0.1 Models.

We evaluate 27 LVLMs across model families, scales, and deployment regimes. The closed-source models come from five commercial API providers, including Gemini-3.1-Pro and Flash-Lite [36], GPT-5 and GPT-5-Mini [35], Claude-Sonnet-4.6 and Claude-Haiku-4.5 [38], Qwen3.6-Plus [34], and Doubao-Seed-2.0-Pro/Mini/Lite [37]. The open-source models span seven families and range from 2B to 38B parameters: Qwen3.6-35B-A3B, Qwen3.5 (4B, 9B, 27B) [34], Gemma-4 (E2B, E4B, 26B-A4B, 31B) [40], InternVL3.5 (4B, 8B, 30B-A3B, 38B) [41], Ovis2 (8B, 16B) [42], Kimi-VL-A3B (Instruct, Thinking) [43], and Step3-VL-10B [44].

5.1.0.2 Implementation details.

To keep scores comparable, we use the same evaluation protocol for all models. Each model is queried \(R=8\) times per item with temperature \(T=1.0\) and a maximum output length of 16,384 tokens. A unified prompt schema asks models to reason step by step and then place the final answer in a \boxed{} block, which is parsed by the same rule-based extractor across benchmarks. For closed-source models that support configurable inference effort, we set it to medium. Open-source models are served via vLLM [45] on 8\(\times\)H200 GPUs. All scores are reported on a 100-point scale. Further details on prompt templates, scoring, and per-benchmark configurations are provided in Appendix [app:protocol].

5.2 Main Results↩︎

Table 2 shows whether MMGist remains challenging and discriminative after item-level filtering. We focus on three questions: whether current models are close to saturation, which capabilities drive the closed/open gap, and whether aggregate scores hide uneven skill profiles.

5.2.0.1 1) MMGist remains unsaturated for frontier models.

MMGist remains far from saturated: the strongest model, Gemini-3.1-Pro, reaches 66.8% Macro Avg, and only two models exceed 60%. This difficulty spans all model tiers: closed-source systems average 52.1%, large open-source models average 43.6%, and small open-source models average 32.4%. Because the pipeline has already removed weakly visual, saturated, and pseudo-hard items, remaining errors are more likely to reflect unresolved multimodal skills than label noise or adversarial artifacts.

5.2.0.2 2) The closed/open gap has shifted from perception to grounded expertise.

The closed/open gap is largest on tasks that pair visual evidence with domain knowledge, not on basic visual extraction. Large open-source models nearly match closed-source systems on Diagram & OCR (58.8% vs.%) and Spatial Understanding (64.8% vs.%), but trail by 16.5 points on Expert Knowledge and 10.0 points on Medical. This suggests that current open-source LVLMs are competitive at extracting visible structure but lag when visual evidence must be integrated with domain knowledge and multi-step reasoning.

5.2.0.3 3) Visual Logic exposes a system-level reasoning bottleneck.

Visual Logic is the clearest bottleneck: its cross-model average is only 22.5%, and even the best model reaches only 40.4%. The weakness is not explained by model scale or access pattern: the average rises from 14.8% for small open-source models to 22.4% for large open-source models and 29.5% for closed-source models, but all three groups remain far below their own Macro Avg. The result is a useful warning: better recognition, OCR, and spatial localization do not automatically lead to better reasoning over visual states, relation, and implicit rules.

5.2.0.4 4) Strong models often have jagged capability profiles.

Aggregate scores hide large within-model asymmetries, so capability-level reporting is necessary. Qwen3.6-35B-A3B achieves 77.2% on Spatial Understanding but only 31.5% on Visual Logic, a 45.7-point gap within the same model. Ovis2-8B reaches 54.5% on Diagram & OCR, exceeding several larger models on that dimension, yet obtains only 12.3% on Visual Logic. These non-monotonic profiles show that parameter count and overall average score do not guarantee balanced multimodal ability. The seven-dimensional structure of MMGist separates skills that benefit from transferable visual representations from skills that likely require targeted reasoning data, training objectives, or inference mechanisms.

5.3 Effect of Filtering on Scores and Rankings↩︎

We next test whether curation improves the evaluation signal, not just whether it makes the test set smaller and harder. Table 3 compares the same 27 models on the raw pool of 23,250 items and on MMGist with 7,262 retained items. The curated set keeps the main ordering from the raw pool: Spearman rank correlation is \(\rho = 0.98\) under the benchmark-level average; 21 models change rank by at most one position, 25 change by at most two, and the top three models remain unchanged. Scores also become more conservative. Average performance drops by 20.3 points, which indicates that weakly visual shortcuts and saturated items in the raw pool inflated scores. The drop does not flatten the leaderboard: the coefficient of variation rises from 0.15 to 0.28 (\(+\)​78%), and the gap between the top-5 and bottom-5 models widens from 26.9 to 32.3 points. MMGist therefore keeps the original ranking signal while spending more of the evaluation budget on items that separate models.

3pt

5.4 Capability-Level Findings↩︎

Figure 4 shows that score inflation in the raw pool varies sharply by capability. Expert Knowledge drops from 72.9% to 40.3% (\(-\)​32.6%), followed by Diagram & OCR (\(-\)​25.4%) and STEM Reasoning (\(-\)​22.5%). These dimensions are more exposed to textual priors, world knowledge, or saturated items. Visual Logic behaves differently: it drops only from 31.1% to 22.5% (\(-\)​8.6%), which suggests that many of its raw-pool items were already visually grounded and unsaturated. After curation, the remaining difficulty sits mostly beyond direct visual extraction. Visual Logic has the lowest cross-model average at 22.5%, and even the best model reaches only 40.4%; Medical and STEM Reasoning also remain difficult, averaging 32.1% and 36.8%. Spatial Understanding and Diagram & OCR are higher, at 60.4% and 56.4%. This suggests that structured perception, OCR, and spatial localization are comparatively mature, while Visual Logic, Medical, and STEM Reasoning remain the main shared bottlenecks for current LVLMs.

5.5 Cost Efficiency↩︎

MMGist improves evaluation efficiency through item-level curation, not uniform subsampling. It retains 7,262 of the original 23,250 items (31.2%), reducing the number of evaluated items by 69%. It also preserves the raw-pool ranking with Spearman \(\rho = 0.98\) and increases cross-model discrimination. In practice, future evaluations can use far fewer model calls while getting rankings that remain stable and easier to separate.

6 Analysis↩︎

6.1 Error Analysis↩︎

To understand where the strongest model still fails, we classify errors made by Gemini-3.1-Pro on MMGist into five categories (Figure 5). Reasoning failure is the dominant error type (41.5%), where the model perceives visual content correctly but produces flawed logical inference, consistent with Visual Logic being the hardest dimension (§5.2). The remaining errors divide among visual perception (22.8%), primarily misidentifying symbols or fine-grained visual details; knowledge gaps (17.9%), mainly in Expert Knowledge and Medical dimensions; misunderstanding (10.3%); and calculation errors (7.5%). These results suggest that improving logical reasoning remains the most impactful direction.

6.2 Effect of Reasoning Effort↩︎

We evaluate GPT-5, GPT-5-Mini, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, and Claude-Sonnet-4.6, at three effort levels (Low, Medium, High) following the protocol in §5.1.

As shown in Table 4, increasing effort yields a Macro gain of \(+\)​7.6% from Low to High, but the benefit is concentrated in reasoning-intensive dimensions: STEM gains the most (\(+\)​26.2% relative), followed by Visual Logic (\(+\)​14.7%) and Expert Knowledge (\(+\)​11.5%). In contrast, perception-oriented dimensions (Visual Perception, Diagram & OCR, Spatial) remain nearly flat (\(\leq\)​3.1% gain), and Medical even declines slightly at High. These results suggest that current models’ perception performance is bounded by visual encoding quality rather than reasoning depth.

7 Conclusion↩︎

We present MMGist, a curated vision-language benchmark of 7,262 items across seven capability dimensions, distilled from 18 existing benchmarks through a systematic audit that identifies three pervasive quality issues: weak visual dependency, item saturation, and pseudo-hard questions. We address these issues with a reusable pipeline integrating text-only ablation, cross-model saturation detection, anomaly adjudication, and expert review. Evaluation on 27 LVLMs confirms that MMGist preserves model rankings while substantially improving discriminability. We release MMGist and the curation pipeline to support higher-quality multimodal evaluation.

Limitations↩︎

7.0.0.1 Benchmark and Language Coverage.

MMGist is constructed from 18 predominantly English benchmarks using closed-form formats (multiple-choice and fill-in-the-blank). This design choice ensures fully reproducible, scorer-consistent quality control, which is a prerequisite for our filtering pipeline. Extending the pipeline to open-ended generation tasks or multilingual settings would require adapting the scoring and filtering protocols accordingly, which we leave to future work.

7.0.0.2 Model Panel Dependency.

Both saturation filtering and anomaly detection are conditioned on the 12-model panel used in this study. As future models improve, items currently retained may become saturated, and items currently flagged as anomalous may be resolved by more capable models. We mitigate this by covering models from five providers across three capability tiers, but the curated suite reflects the capability frontier at the time of construction. Our pipeline makes periodic re-curation with updated model panels straightforward.

7.0.0.3 Subjectivity in Anomaly Review.

The anomaly detection stage combines rule-based candidate recall, multi-model adjudication by five LVLMs, and human expert review. While majority voting across independent judges reduces individual bias, the process retains inherent subjectivity: LLM judges may share systematic blind spots on certain domains, and human reviewers may disagree on genuinely ambiguous borderline cases. We report inter-annotator agreement statistics in Appendix [app:agreement] to quantify this effect, but acknowledge that a different expert panel could reach different conclusions on a subset of items.

Ethical Considerations↩︎

7.0.0.4 Data Provenance.

All evaluation items in MMGist are sourced exclusively from previously published and publicly available benchmarks. We have reviewed the licensing terms of each source benchmark and confirmed that our use is consistent with their original intended purposes.

7.0.0.5 Sensitive Content.

MMGist retains items from two medical benchmarks, SLAKE [11] and MedXpertQA [12]. These items are included solely for evaluating model capabilities in the medical domain and do not constitute medical advice. Both source datasets were de-identified and released by their original creators in compliance with applicable privacy regulations; we do not introduce any additional personally identifiable information.

7.0.0.6 Potential Biases.

Our quality control pipeline relies on large models for text-only ablation, saturation detection, and multi-model adjudication. Although we mitigate single-model bias by aggregating judgments from multiple models across different providers, the filtering process may still reflect shared biases of current models, potentially favoring certain question types or penalizing others. Similarly, the human expert review stage, while serving as a final safeguard, may introduce annotator-specific biases. We encourage future work to investigate and quantify these effects.

7.0.0.7 Intended Use.

MMGist is designed for academic evaluation of large vision-language models. Benchmark scores reflect model performance on curated evaluation items under controlled settings and should not be interpreted as indicators of real-world deployment readiness. Rankings may vary under different evaluation protocols, prompting strategies, or domain-specific configurations.

7.0.0.8 Computational Cost.

Constructing MMGist requires substantial API calls and GPU computation for multi-model sampling and adjudication. However, we note that the resulting curated suite substantially reduces ongoing evaluation costs for future researchers: by retaining only high-quality, discriminative items, MMGist achieves comparable evaluation fidelity with significantly fewer items than evaluating on the full set of source benchmarks.

References↩︎

---

1. winston_yuan@sjtu.edu.cn↩︎

2. For each stage, the number of samples filtered from each benchmark is reported in the Appendix.↩︎

3. This threshold is intentionally conservative: an item must be consistently solved by almost all models, from lightweight to flagship ones, before being discarded, ensuring that truly challenging items remain even if one or two strong models solve them.↩︎