1 Introduction↩︎

Despite the rapid evolution of Multimodal Large Language Models (MLLMs), a fundamental evaluation crisis persists: current perception benchmarks do not reliably reflect genuine perceptual capability. This has led to a evaluation paradox where leaderboards are increasingly saturated in the high-score regime as illustrated in 1, yet models remain perceptually brittle in real-world deployment. Top-tier systems often appear nearly tied on metrics but exhibit drastically different failure modes—such as miscounting objects or inverting spatial relations—that are highly salient to users even when reported metric scores [1] remain high. This discrepancy suggests that benchmark rewards are misaligned with human perceptual sensitivity, creating a false sense of progress and failing to provide the diagnostic resolution needed to steer the next generation of MLLMs.

We trace this failure to two systemic flaws in current benchmark design. First, the visual content and task design lack sufficient perceptual detail coverage. Many benchmarks rely on information-poor images or narrow domains [2], often framing tasks as closed-form questions that allow models to “shortcut” through linguistic priors rather than genuine visual grounding [3], [4]. Even in open-ended captioning, references are frequently imprecise, biased, or too sparse [1] to challenge the long-tail visual knowledge of frontier models. Second, current reward signals are fundamentally uncalibrated. Conventional metrics, such as single-number similarity scores (e.g., CLIPScore [5]) or averaged multi-aspect schemes [1], rely on linear averaging that effectively “dilutes” fatal localized errors with general semantic overlap. Consequently, a caption plagued by hallucinations can still achieve a high metric score, severing the link between numerical performance and genuine reliability. In contrast, human perception is strictly non-linear: a single-digit hallucination in a financial table is not a permissible fluctuation but a binary failure [6]. Existing metrics fail to reflect this, making it difficult to distinguish acceptable descriptive variation from critical perceptual failures.

To bridge this gap, we propose PerceptionRubrics, a benchmark that repurposes image captioning—the most fundamental proxy for integrated perception, recognition, and reasoning—into a rigorous diagnostic testbed. To address the data deficit, we curate 1,038 images characterized by extreme information density and distributional diversity. Crucially, to bypass the visual grounding gap that limits direct image-to-rubric generation, we adopt a caption-centric construction pipeline as an intermediary strategy. Instead of relying on noisy raw predictions, we establish ground truth via a Circular Peer-Review consensus mechanism: an ensemble of state-of-the-art MLLMs iteratively critiques and refines descriptions, followed by human verification. This process yields “Golden Captions” that serve as high-fidelity textual references for the visual content, filtering out the noise and biases prevalent in traditional datasets.

Building on this foundation, we address the calibration gap by distilling Golden Captions into a granular, rubric-based auditing system. We extract over 12,000 atomic rubrics and organize them into two complementary streams: Must-Right rubrics, which capture essential visual facts that a response must satisfy, and Easy-Wrong rubrics, which target common hallucinations, omissions, and misinterpretations mined from model error patterns. We then introduce a gated scoring mechanism calibrated to human sensitivity: the Must-Right rubrics serve as mandatory gatekeepers, so failure to satisfy any essential criterion sharply penalizes the final score. This design ensures that the metric reflects not just coarse semantic proximity, but genuine perceptual reliability, effectively distinguishing between acceptable approximations and catastrophic failures.

Comprehensive evaluation and analysis across leading MLLMs on PerceptionRubrics yields critical insights:

• Unveiling the “Reliability Gap”. We expose a disconnect between fragmented recognition and coherent understanding: models often pass atomic checks but fail strict conjunctive constraints. This reveals that despite high partial scores, current MLLMs lack the perceptual consistency required for information-dense domains like GUIs.

• Quantifying the Open-Closed Gap. Contrasting the convergence in reasoning tasks, we identify a persistent 8% perception deficit between the open-source frontier (e.g., Qwen3.5 [7]) and proprietary leaders (e.g., Seed-2.0 [8]). Basic visual precision thus remains a decisive bottleneck distinguishing intrinsic model capacity.

• Superior Human Alignment. PerceptionRubrics aligns substantially better with human judgment than conventional benchmarks (e.g., DOCCI [2]), an effect amplified by our gated scoring. Furthermore, a near-perfect correlation between basic perception and hallucination resistance confirms strict fidelity as a prerequisite for reliable generation.

2 Related Work↩︎

2.0.0.1 Visual Perception Benchmarks in MLLMs.

Evaluating visual perception remains pivotal for assessing MLLMs [9], [10]. Current benchmarks generally fall into two categories: holistic suites and task-specific datasets. Comprehensive frameworks like MMBench [11], MM-Vet [12], and MME [13] evaluate broad capabilities but increasingly face leaderboard saturation in recent flagship models [14], [15]. Conversely, task-specific benchmarks target distinct skills, such as OCR in OCRBench [16], open-world recognition in SimpleVQA [17] and spatial understanding in VSIBench [18]. However, these benchmarks heavily rely on closed-ended formats (e.g., single or multiple-choice). Such designs often allow models to exploit linguistic priors or random guessing to bypass genuine visual grounding [3], [4], limiting their ability to diagnose perceptual brittleness.

2.0.0.2 Evaluation of Image Captioning.

Image captioning serves as a holistic proxy for perception, requiring models to autonomously prioritize and describe visual elements. Recent methods have moved beyond generic similarity metrics [19] or object-set matching heuristics [20] towards model-based evaluation. DOCCI [2] targets detailed description using reference-based metrics; DetailCaps [1] employs multi-expert annotation to score object and attribute matching; RePer [21] utilizes an LLM-judge for aspect-based evaluation; and CapArena [22] aligns assessments with human preference via pairwise battles. Despite these advancements, a critical gap persists: existing methods often rely on sparse, biased references and linear scoring mechanisms that dilute fatal localized hallucinations with high holistic similarity, failing to reflect the non-linear sensitivity of human verification [6].

2.0.0.3 Rubric-Based Reward Modeling.

To improve evaluation reliability, the field is shifting from opaque scalar scoring [23] to rubric-based auditing. In text generation, structured criteria have effectively mitigated reward hacking [24]. Approaches like RM-R1 [25] and SPCT [26] formulate evaluation as a reasoning process via chain-of-rubrics, while frameworks such as RaR [27] and ResearchRubrics[28] leverage LLMs to decompose subjective judgments into atomic, verifiable checks. While this paradigm has standardized text-centric evaluation, comparable fine-grained auditing systems for multimodal perception remain under-explored. Existing vision benchmarks lack the mechanism to decompose complex visual scenes into verifiable atomic facts, highlighting the need for a rigorous standard to distinguish precise perception from approximation.

3 PerceptionRubrics↩︎

To align multimodal evaluation with the rigor of human judgment, we first outline our guiding design principles (3.1) and data curation strategy (3.2), followed by our novel caption-centric pipeline for generating atomic rubrics (3.3) and the gated scoring mechanism that enforces calibration (3.4).

3.1 Design Criteria↩︎

To rigorously stress-test the upper bounds of state-of-the-art models and bridge the gap between reported metrics and real-world reliability, the design of PerceptionRubrics is governed by two overarching principles:

3.1.0.1 Enforcing Perceptual Persistence.

To probe comprehensive perceptual capabilities, we prioritize complexity over scale. We posit that a robust benchmark must utilize images with extreme information density that ranging from crowded scenes to document-heavy layouts, therefore invalidate the linguistic “shortcuts” often taken by models. This design criterion compels models to exhibit perceptual persistence, requiring active, fine-grained exploration of long-tail visual details rather than reliance on rough global understanding or parametric priors.

3.1.0.2 Calibrating to Human Sensitivity.

To resolve the paradox where high semantic scores mask brittle performance, we prioritize precision over approximation. We argue that an effective metric must mirror the error-sensitive nature of human judgment, where localized errors (e.g., hallucinating a single digit in a chart) represent binary failures rather than minor fluctuations. Consequently, our criterion mandates atomic verifiability and task-adaptive penalties: evaluation must be grounded in objective, fact-based checks (True/False) and rigorously penalize hallucinations, ensuring the metric reflects practical perceptual utility rather than mere statistical similarity.

3.2 Image Curation↩︎

To ensure the benchmark probes the perceptual limits of flagship models, we curate an image collection that emphasizes visual diversity and complexity, targeting inputs rich in perceptually critical details that maximize error potential.

3.2.0.1 Task Domains.

As illustrated in 3, we structure our data across seven diverse categories to cover the full spectrum of multimodal capabilities: Natural Scenes (complex real-world environments); Document & OCR (text-dense documents, forms, and handwritten content); Digital UI & UX (web pages, mobile UIs, and dashboards); Structured Data (charts, plots, and tables); STEM & Expert (scientific diagrams, geometric figures, and medical imaging); Logic & Puzzle (visual riddles and spatial reasoning tasks); and Creative & Cultural (artworks, cultural artifacts, and design concepts).

3.2.0.2 Density-Aware Filtering.

We employ the advanced MLLM, Step3-VL-10B [15], as a scorer to filter the curated images based on complexity and informativeness. Specifically, given a candidate image, the model evaluates its visual complexity (via object richness) and informativeness (via semantic density), assigning a score from 1 to 10 (see details in 9.1). To ensure a balanced distribution across categories, we retain images that surpass domain-specific thresholds.

3.3 Caption-Centric Perception Rubric Construction↩︎

To instantiate the rigorous design criteria outlined above, we construct a caption-centric pipeline. Given that generating rubrics directly from raw pixels often suffers from the visual grounding gap inherent in current vision encoders [29] and MLLMs [30], we choose an intermediary strategy: first explicitly transcribing visual information into text, then distilling rules from it. This approach prioritizes constructing a comprehensive, precise, and exhaustive golden caption to capture image details. This textual foundation enables the subsequent rubric generator to cover extreme visual granularity and detect subtle failure modes with significantly higher reliability than direct image-to-rubric methods.

3.3.1 Generating Golden Caption↩︎

As illustrated in the top half of 4, we construct golden reference captions \(C_{gold}\) through a two-step consensus-driven pipeline. This approach treats heterogeneous MLLMs as a collaborative filter to minimize human annotation costs while ensuring high precision.

3.3.1.1 Step 1: Circular Peer-Review.

Three distinct top-tier MLLMs (e.g., GPT-5.2, Gemini-3-Pro, and Seed-1.8) serve as a “jury-and-generator” ensemble. For each image, they first generate independent descriptions to form an initial candidate pool. To reduce hallucinations and self-preference bias, we implement a circular peer-review mechanism (4, top middle). In this phase, models iteratively compare candidates against visual evidence, rank them based on accuracy, and rewrite descriptions to synthesize a superior version. This review cycle runs for limited iterations (\(N \le 2\)) to efficiently drive the ensemble toward a unified consensus.

3.3.1.2 Step 2: Strict Consensus Filtering.

To strictly control quality and annotation costs, human experts intervene only as final verifiers rather than creators. We adopt a discard-on-divergence protocol: samples where the models fail to reach a unanimous agreement are discarded. Only when the ensemble converges on a single optimal caption (i.e., high consensus) do human annotators perform a lightweight verification to finalize the golden reference \(C_{gold}\). This ensures that human effort is spent exclusively on high-confidence samples.

3.3.2 Generating Perception Rubric↩︎

Building upon the verified golden reference \(C_{gold}\), we employ Gemini-3-Pro [9] as the rubric proposer to construct dual-stream evaluation criteria (4, bottom). This pipeline mirrors the error-sensitive nature of human judgment by generating rubrics from two complementary perspectives: a priori essential facts and a posteriori common pitfalls.

A Priori: Must-Right Rubrics. From a positive perspective, the rubric proposer distills a set of atomic perceptual facts from \(I\) and \(C_{\text{gold}}\) that a candidate must correctly identify. Crucially, we employ domain-specific adaptive prompts (detailed in 9.2) to align with varying perceptual demands: rubrics for text-centric images prioritize character precision, while those for natural scenes emphasize spatial relations and object attributes.

A Posteriori: Easy-Wrong Rubrics. From a negative perspective, we challenge model robustness by targeting likely failure modes. We first construct a response pool \(\mathcal{P}\) by collecting predictions from a diverse set of baseline MLLMs. By analyzing the discrepancies between these actual outputs \(\mathcal{P}\) and the reference \(C_{gold}\), the rubric proposer identifies frequent hallucinations and subtle misinterpretations. These empirically observed errors are converted into Easy-Wrong rubrics, ensuring the evaluation penalizes realistic mistakes rather than hypothetical ones.

3.4 Evaluation Metric↩︎

We employ an LLM-as-a-Judge framework to perform fine-grained evaluation, aiming to balance effectiveness and efficiency. We select GPT-OSS-120B [31] as the judge due to its proven capability for highly calibrated assessments [15]. Specifically, a model prediction \(P\), and a set of rubrics \(\mathcal{R} = \mathcal{R}_{m} \cup \mathcal{R}_{e}\) covering Must-Right and Easy-Wrong cases, the judge evaluates each rubric item yielding a boolean output (True for compliance, False otherwise). To prioritize factual correctness, we implement a gated scoring logic:

3.4.0.1 Must-Right as the Gate.

Let \(\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}\) be the set of Must-Right rubrics, which serve as a mandatory gatekeeper. If the model fails even a single criterion in \(\mathcal{R}_{m}\), the description is deemed factually compromised, penalizing the final score to zero: \[G = \prod_{i=1}^{j} \mathbb{I}(r_{m,i} = \text{True})\] where \(G \in \{0, 1\}\) represents the gate status.

3.4.0.2 Easy-Wrong for Granular Differentiation.

For models that pass the gate (\(G=1\)), we calculate the final score based on the Easy-Wrong rubrics \(\mathcal{R}_{e} = \{r_{e,1}, \dots, r_{e,k}\}\). These rubrics assess whether the response correctly captures error-prone fine-grained details, including details that are commonly hallucinated, omitted, or misinterpreted. The final score \(S\) is defined as: \[S = G \cdot \frac{1}{k} \sum_{i=1}^{k} \mathbb{I}(r_{e,i} = \text{True})\] This scoring philosophy ensures that a high score reflects not only the absence of basic hallucinations but also a superior discernment of subtle, density-rich visual details.

4 Experiments↩︎

4.1 Benchmark Statistics↩︎

As summarized in 1, the resulting benchmark contains 1,038 information-dense images, each paired with a verified golden caption and a set of instance-specific perception rubrics. In total, PerceptionRubrics includes 12,004 atomic rubrics, consisting of 4,232 Must-Right rubrics and 7,772 Easy-Wrong rubrics, with an average of 11.56 rubrics per image. Beyond rubric density, our benchmark is also characterized by highly detailed textual references. As shown in 5, the golden caption lengths exhibit a right-skewed distribution: most captions concentrate around 400–500 words, while a long tail extends to captions exceeding 3,400 words. The mean caption length reaches 770.42 words, higher than the median of 569 words. This long-tailed caption distribution reflects the high information density of our images and provides a rich textual anchor for constructing fine-grained and verifiable rubrics.

4.2 Experimental Setup↩︎

We evaluate a diverse suite of 25 models, spanning proprietary frontier models (e.g., Gemini-3-Pro [9], Gemini-3.5-Flash [32], GPT-5.4 [33], GPT-4o [34], Seed-2.0 [8], Seed-1.8 [35], Seed-1.6 [36], GLM-5V-Turbo [37], Qwen3.5-Plus [7]) and leading open-weights models (e.g.,Qwen3.5-397B [7], Qwen3-VL [14], Qwen2.5-VL [38],Step3-VL-10B [15], Step-3.7-Flash [39], MiniMax-M3 [40], MiMo-V2.5 [41],Kimi-K2.5 [42]).

4.3 Main Results↩︎

Compliance Scores. 2 summarizes the performance of all evaluated models, which reveals a pronounced performance stratification that is largely obscured by traditional holistic benchmarks. Seed-2.0-Lite leads the leaderboard with an overall score of \(70.07\%\), outperforming the runner-up (Gemini-3.5-Flash) by \(0.19\%\). In contrast, despite being a widely used proprietary model, GPT-4o-2024-05-13 exhibits the weakest perceptual performance among its category, achieving an overall accuracy of only \(12.59\%\). Across models, performance is consistently higher on natural image domains (e.g., reaching \(79.20\%\) for Seed-2.0-Lite), aligning with human perceptual intuition and reflecting the relative maturity of models in handling real-world visual scenes. Conversely, almost all models struggle most in the GUI domain (e.g., Qwen2.5-VL-7B drops to \(5.13\%\)), indicating that robust visual grounding for future agents remains an unresolved challenge. Moreover, unlike in reasoning tasks where open-sourced models often rival proprietary flagships [14], [15], our results show a distinctive performance gap. The best-performing open-source model (Qwen3.5, \(61.61\%\)) still trails the proprietary state-of-the-art by over \(8\%\). This suggests that open-source models still have significant ground to cover in fine-grained perception and open-world recognition, also confirming our benchmark’s sensitivity in distinguishing intrinsic model capacity beyond reasoning capabilities.

Domain-Specific Failure Modes. To diagnose where models fundamentally fail, we analyze cases in which predictions do not pass the Must-Right gate (i.e. \(G = 0\)), indicating a breakdown in basic perceptual capability. 6 (Left) presents the distribution of such failure cases across domains for six representative models. A similar pattern emerges: GUI constitutes the dominant source of perceptual failures. In contrast, domains such as Natural and STEM are comparatively easier, exhibiting substantially fewer failures. This trend suggests that current models continue to struggle with inputs characterized by high information density and strict spatial constraints.

Atomic vs. Holistic Perception. To evaluate perceptual reliability at different granularities, we compare performance metrics derived from individual rubrics versus the aggregate gate status. Specifically, we define Atomic Accuracy as the mean accuracy of all individual rubrics (\(r_{i}\)), representing local precision. In contrast, the Must-Right Pass Rate is calculated as the average value of the binary gate status \(G\) across the dataset (i.e., the expectation \(\mathbb{E}[G]\)), representing the probability of a record successfully passing the mandatory gatekeeper. As shown in 6 (Right), models consistently achieve high Atomic Accuracy, indicating that most individual \(r_{i}\) predictions are correct. However, the Must-Right Pass Rate (average \(G\)) is substantially lower, revealing a systematic failure to satisfy the strict conjunction of all constraints. We term this discrepancy the Reliability Gap. Notably, this gap narrows as model capability increases, suggesting that stronger models are better able to maintain consistent perception abilities required to keep the gate \(G\) open.

4.3.0.1 Consistency of Perceptual Capabilities.

We further examine the correlation between models’ basic perceptual reliability and their hallucination resistance to fine-grained details. As shown in 7, there is a near-perfect linear correlation (\(R^2 \approx 0.98\)) between Must-Right Pass Rate and Easy-Wrong accuracy. This implies that models failing to ground essential visual facts (low X-axis) inevitably struggle with subtle details and hallucination (low Y-axis). Therefore, robust fine-grained understanding critically depends on foundational perception, in particular, the coherent recognition of multiple salient elements.

5 Analysis↩︎

Beyond model performance, we conduct a systematic meta-evaluation to assess the rigor and reliability of the benchmark itself from multiple perspectives.

5.1 Alignment with Human Preference↩︎

To validate whether PerceptionRubrics reflects human-perceived model quality, we compare its model ranking against the Vision Arena [43] leaderboard, which aggregates large-scale human preferences over MLLM responses into Elo ratings. In 9, we focus on the five models: GPT-5.4, Qwen3-VL-235B, GPT-4o, Kimi-K2.6, and MiMo-V2.5. For each benchmark, we plot the evaluation score of these models against the Vision Arena score.

PerceptionRubrics exhibits the strongest alignment with human preference among the compared benchmarks, achieving a Pearson correlation of \(0.916\) and a Spearman rank correlation of \(1.000\). In contrast, existing captioning benchmarks such as DOCCI [2] and DetailCaps [1] show substantially weaker agreement with human-preference scores. DOCCI, in particular, assigns nearly indistinguishable scores to models with markedly different human-preference ratings, indicating limited discriminative power. These results suggest that PerceptionRubrics provides a more human-aligned and discriminative signal for fine-grained perception evaluation.

5.2 Resistance to Length Bias.↩︎

We analyze the correlation between predicted caption length and performance on PerceptionRubrics to assess potential length bias. As shown in 10 (a-b), Gemini-3.1-Pro shows no statistically significant correlation (\(r=-0.079\), \(p=0.0758\)), while Kimi-K2.6 exhibits a weak positive correlation (\(r=0.172\), \(p=1.09\times10^{-4}\)). This result indicates that PerceptionRubrics effectively decouples verbosity from evaluation outcomes, rewarding precise and verifiable perception rather than longer descriptions.

5.3 Evaluation Robustness↩︎

In 10 (c), we selected three representative models spanning different capability levels: Seed-2.0-Lite, Step3-VL-10B, and Kimi-K2.6. Then we performed repeated evaluations using two distinct judges with the same inputs: GPT-OSS-120B [31] and GPT-5.5 [44]. Despite GPT-OSS-120B exhibiting a slightly stricter scoring distribution (systematically lower by \(\sim\)​6.0%), both judges yielded an identical ranking order. The black error bars represent the standard deviation across these independent runs. The results demonstrate high stability, with standard deviations remaining consistently low across all configurations. Overall, these results demonstrate the robustness of both our rubric generation pipeline and the resulting evaluation metrics to judge choice and sampling variability.

5.4 Rubric Coverage vs. Evaluation Stability↩︎

As shown in 8, we analyze the effect of rubric quantity on evaluation stability. Using 25 models, we subsample \(20\%\), \(40\%\), \(60\%\), and \(80\%\) of rubrics from both the Must-Right and Easy-Wrong sets. For each sampling ratio, we perform three independent runs and compute the standard deviation of model scores to measure stability. The figure visualizes the distribution of these standard deviations across models at each ratio using violin plots, with embedded boxes indicating the interquartile range and medians; the dashed line denotes the mean stability trend. Evaluation stability improves monotonically as rubric coverage increases, with standard deviation consistently decreasing, highlighting sufficient rubric coverage as a prerequisite for stable and reproducible perception assessment.

6 Conclusion↩︎

We present PerceptionRubrics, a rubric-based benchmark that calibrates multimodal evaluation to human perceptual judgment. By decomposing dense image understanding into atomic, verifiable rubrics and enforcing a gated scoring mechanism, our framework exposes perceptual failures that are often hidden by existing metrics. Experiments across 25 MLLMs reveal a clear reliability gap between individual fact recognition and consistent conjunctive perception, persistent weaknesses in information-dense domains such as GUIs, and strong alignment between our scores and human preferences. These findings suggest that reliable multimodal evaluation should move beyond coarse similarity and explicitly audit critical visual facts. We hope PerceptionRubrics provides a sharper diagnostic tool for measuring perceptual reliability and guiding the development of more trustworthy MLLMs.

Impact Statement↩︎

This work aims to advance machine learning by improving the reliability of multimodal evaluation. While this may affect downstream MLLM development, we do not identify specific societal consequences requiring special discussion.

[45]

7 Dataset Statistics↩︎

In this section, we provide detailed statistics and comparisons for the PerceptionRubrics benchmark.

7.1 Comparison with Other Benchmarks↩︎

Compared to existing benchmarks, PerceptionRubrics distinguishes itself in three critical dimensions: annotation granularity, data source diversity, and domain coverage, as shown in 3.

• Dense and Comprehensive Captions: Unlike DetailCaps-4870 [1] and DOCCI [2], which typically provide brief descriptions (averaging 122.1 and 135.9 words, respectively), PerceptionRubrics focuses on dense captioning. With an average of 770.42 words per image, our benchmark captures fine-grained visual details, spatial relationships, and implicit reasoning, offering a significantly more challenging testbed for evaluating the upper bounds of MLLMs.

• Broad Domain Coverage: Unlike existing benchmarks that are predominantly restricted to natural scenes, PerceptionRubrics spans seven distinct domains to provide a more comprehensive evaluation. These range from everyday natural scenes to specialized areas such as GUIs, OCR-heavy documents, and STEM-related diagrams. This diversity is crucial for assessing the general-purpose capabilities of agents in complex, real-world applications that go far beyond simple object recognition.

• Diverse and High-Quality Sources: Instead of relying solely on web-crawled data or specific author donations, our dataset aggregates high-quality samples from existing visual benchmarks. Furthermore, we employ a hybrid annotation pipeline combining advanced reasoning models (e.g., GPT-5.2-Thinking) with human expert verification, ensuring both the scalability and reliability of the ground truth.

7.2 Distributions↩︎

7.2.1 Caption Length Distribution↩︎

As illustrated in Figure 5, we analyze the word count distribution of the golden captions. The distribution follows a typical long-tail pattern: while the majority of captions are concentrated between 300 and 700 words (with a median of 569), a significant portion extends beyond 1,000 words, reaching up to 3,461 words. This diversity in length ensures that our benchmark covers both concise summaries and highly detailed descriptions, providing a robust basis for evaluating model performance across different levels of information density.

7.2.2 Rubric Distribution↩︎

To ensure a granular and balanced evaluation, we analyze the distribution of rubrics across the dataset in Figure 11. (a) The total number of rubrics per sample primarily ranges from 10 to 20, with a clear peak at approximately 12, indicating a consistently high level of evaluation detail across the benchmark. (b) When broken down by category, Must-Right rubrics exhibit a sharp distribution centered around 4 items, representing the core facts that a model must capture. In contrast, Easy-Wrong rubrics show a broader distribution peaking around 8 items. This design places a heavier emphasis on penalizing common hallucinations and subtle errors, thereby increasing the discriminative power of the benchmark for high-performing models.

8 Model Roles and Pipeline Details↩︎

To construct and evaluate PerceptionRubrics, we utilized a diverse set of models, assigning specific roles based on their capabilities. The detailed assignments are listed below:

• Complexity Judger: STEP-3-VL-10B. Responsible for filtering images based on visual complexity and informativeness.

• Rubric Generator: Gemini-3-Pro. Generated the initial set of perception rubrics from the images.

• Panel of Judges: Gemini-3-Pro, GPT-5.2, Seed-1.8. Acted as a consensus panel to validate the quality of generated captions.

• Final Judger: GPT-OSS-120B. Used for final scoring during the evaluation phase.

9 Prompts↩︎

We provide the full system prompts used in our pipeline to ensure reproducibility.

9.1 Complexity Filtering Prompt↩︎

The following prompt is used by the Complexity Judger to select high-quality images.

You are an extremely strict computer vision data expert. Please analyze the provided image and perform a rigorous evaluation based on the two dimensions of "Visual Complexity" and "Informativeness".

**Core Principles:**
1. **Do NOT** give a high score simply because the image contains text. You must evaluate the **density** and **semantic depth** of the text.
2. **Severely penalize** low-quality images: images that are blurry, noisy, contain scribbled handwriting, or have excessive empty backgrounds should receive low scores.
3. If the majority of the image is white space or a single background, the score must be determined by the richness of the subject content, not by the image dimensions.

Please score based on the following strict standards (1-10 points):

1. Visual Complexity:
   - Definition: The quantity of independent visual elements (objects, lines, textures), spatial occupancy, and clarity of details within the image.
   - **1-3 Points (Low)**: Minimalist composition, massive white space, simple handwriting, single isolated objects, blurry snapshots, low-resolution screenshots.
   - **4-7 Points (Medium)**: Clear composition, good foreground-background separation, natural scenes with some texture detail, standard object close-ups.
   - **8-10 Points (High)**: Extremely high density of details (e.g., crowds, dense forests, complex mechanical structures), frame filled, no large areas of solid color, high-frequency textures.

2. Informativeness:
   - Definition: The amount of information when the image is translated into a text description, the richness of context, and its knowledge value.
   - **1-3 Points (Low)**: Simple mathematical formulas, single words/numbers, scribbles without context, illegible content, generic decorative patterns, extremely low information entropy.
   - **4-7 Points (Medium)**: Complete sentences, clear recognition of single objects (e.g., "a red apple"), scenes with distinct actions, standard street views or portraits.
   - **8-10 Points (High)**: Dense documents (e.g., full-page newspapers, academic papers), complex infographics (containing multiple data sets), historical photos rich in narrative detail, scenes requiring long-form text to describe clearly.

**Output Format Requirements:**
Please strictly output in XML format. Do not use Markdown code blocks (do not use ```xml). Output the XML string directly.

XML Template:

<image_evaluation>
    <visual_complexity>
        <reasoning>Briefly describe the density of visual elements. If the image is blurry or mostly empty, explain here and provide a reason for the low score.</reasoning>
        <score>Integer between 1 and 10</score>
    </visual_complexity>
    <informativeness>
        <reasoning>Briefly describe the richness of semantic content. If it is a simple formula or phrase, explicitly state that the information content is limited.</reasoning>
        <score>Integer between 1 and 10</score>
    </informativeness>
</image_evaluation>

9.2 Rubric Generation Prompt↩︎

The prompts used for generating rubrics are as follows:

You are an expert evaluator for Multimodal Large Language Models (MLLMs), specializing in creating "Gating Rubrics" for natural imagery.

Your goal is to extract a concise set of **Critical Perception Checkpoints** from the provided Image and Ground Truth (GT) caption. These rubrics define the minimum acceptable standard for a model's response.

### CRITICAL EVALUATION PROTOCOL
This is a **Zero-Tolerance Gating Task**. If a candidate model fails **ANY** of the checkpoints you generate, it receives a score of 0.
Therefore, your rubrics must strictly adhere to the following principles:
1.  **Undeniable Visibility:** Only select elements that are clearly visible and prominent in the image.
2.  **Essentiality:** Only select elements that are critical to the image's core meaning. Ignore background clutter or minor details.
3.  **Verifiability:** Each rubric must be a binary (Pass/Fail) check.

### WORKFLOW INSTRUCTIONS

**Step 1: Rubric Generation Strategy (Semantic Generalization)**
Apply the following abstract rules to ensure the rubrics are robust to varying levels of descriptive detail:

* **Entity Abstraction:** Identify the fundamental semantic category of the dominant object, strictly discarding specific instance names, brands, or fine-grained biological sub-species. (e.g., use "car" instead of "Tesla Model 3"; use "dog" instead of "Golden Retriever").
* **Attribute Decoupling:** Decouple the object's existence from its descriptive attributes. Exclude color, material, or state adjectives from the rubric criteria to prevent penalizing valid but concise responses. (e.g., require "the presence of a flower" rather than "a yellow flower"; require "clothing" rather than "a silk dress").
* **Contextual Necessity:** Only include attributes if they serve as the sole differentiator between multiple objects of the same class. (e.g., "the red player" vs "the blue player").

**Step 2: Final Filtering (Grounding Check)**
Review your list. Ensure every rubric meets the "Grounding Check":
* The element must be present in **BOTH** the Image and the GT Caption.
* If the GT caption describes a hidden detail or hallucinates something not clearly visible, **discard it**.

### OUTPUT FORMAT
Return a strictly valid JSON list containing 3 to 5 strings.
Example: `["The response mentions <Generalized Object>.", "The response indicates the weather is <Condition>."]`

---

### FEW-SHOT EXAMPLES

**Example 1: Natural Scene**
* **Context:** Image shows a Black Tesla Model 3 on a rainy highway. GT describes it specifically as a Tesla Model 3. User asks "Describe this image."
* **Thought Process:** Apply Entity Abstraction: "Tesla Model 3" -> "Car". Apply Attribute Decoupling: Ignore "Black". "Rainy" is global context, keep it.
* **Generated Rubrics:**
    [
      "The response mentions a car or vehicle.",
      "The response indicates the weather is rainy or the road is wet.",
      "The response mentions the vehicle is on a road or highway."
    ]

**Example 2: Animal Interaction**
* **Context:** Image shows a Golden Retriever catching a frisbee in a park. GT says "A purebred Golden Retriever leaps to catch a red frisbee."
* **Thought Process:** Apply Entity Abstraction: "Golden Retriever" -> "Dog". Apply Attribute Decoupling: Ignore "Red" (frisbee color). Keep the interaction "catching/leaping".
* **Generated Rubrics:**
    [
      "The response mentions a dog.",
      "The response mentions the dog is interacting with a frisbee or disc.",
      "The response captures the action of jumping or catching."
    ]

You are an expert evaluator for Multimodal Large Language Models (MLLMs), specializing in creating "Gating Rubrics" for Graphical User Interfaces (GUIs), including mobile screenshots, web pages, and software interfaces.

Your goal is to extract a concise set of **Critical Perception Checkpoints** from the provided Image and Ground Truth (GT) caption. These rubrics define the minimum acceptable standard for a model's response.

### CRITICAL EVALUATION PROTOCOL
This is a **Zero-Tolerance Gating Task**. If a candidate model fails **ANY** of the checkpoints you generate, it receives a score of 0.
Therefore, your rubrics must strictly adhere to the following principles:
1.  **Undeniable Visibility:** Only select elements that are clearly visible and prominent.
2.  **Functional Criticality:** Only select elements that are essential for operating or navigating the interface (e.g., "Submit" button, "Back" arrow). Ignore decorative banners or ads.
3.  **Verifiability:** Each rubric must be a binary (Pass/Fail) check.

### WORKFLOW INSTRUCTIONS

**Step 1: Rubric Generation Strategy (Interaction & Structure)**
Apply the following abstract rules to ensure the rubrics cover the interface's functionality:

* **Functional Semantics:** Identify interactive elements by their function, not just their shape. Map icons to their standard meaning. (e.g., "The response identifies the magnifying glass as a 'Search' button/feature"; "The response identifies the 'hamburger' icon as a menu").
* **Textual Anchoring:** Enforce exact matching for critical labels, headers, and button text. (e.g., The page title "Settings", the button label "Log In").
* **State Awareness:** Check for visual cues that indicate the system status. (e.g., "The response notes that the 'Home' tab is currently selected/active"; "The response mentions the toggle is in the 'On' position"; "The response notes a notification badge/red dot").
* **Structural Hierarchy:** Identify the major navigation zones. (e.g., "The response mentions the navigation bar at the bottom"; "The response identifies the header containing the logo").

**Step 2: Final Filtering (Grounding Check)**
Review your list. Ensure every rubric meets the "Grounding Check":
* The element must be present in **BOTH** the Image and the GT Caption.
* If the GT caption describes a functional flow not visible in the static image (e.g., "Clicking this opens a modal"), **discard it**. Only evaluate what is currently visible.

### OUTPUT FORMAT
Return a strictly valid JSON list containing 3 to 5 strings.
Example: `["The response identifies the screen title as <Title>.", "The response mentions the <Button Name> button at the bottom."]`

---

### FEW-SHOT EXAMPLES

**Example 1: Mobile App (Settings Page)**
* **Context:** A screenshot of a Settings page. Title "Settings". Top item is "Airplane Mode" (Toggle is OFF). Bottom is a Tab Bar with "General" selected.
* **Thought Process:** Title is critical context. "Airplane Mode" is the first functional item. The state of the toggle (OFF) is a detail, but if prominent, keep it. The selected tab defines where we are.
* **Generated Rubrics:**
    [
      "The response identifies the screen title as 'Settings'.",
      "The response mentions the 'Airplane Mode' option.",
      "The response indicates that the 'General' tab is currently selected or active.",
      "The response mentions the presence of a navigation bar at the bottom."
    ]

**Example 2: E-Commerce Product Page**
* **Context:** A product page for "Nike Air Max". Price "$120". Big red button "Add to Cart". Review stars (4.5).
* **Thought Process:** Product Name is the core entity. Price is critical data (OCR). The primary action is "Add to Cart".
* **Generated Rubrics:**
    [
      "The response identifies the product name as 'Nike Air Max'.",
      "The response correctly mentions the price as $120.",
      "The response identifies the primary action button labeled 'Add to Cart'.",
      "The response mentions the presence of a star rating or reviews."
    ]

You are an expert VLM (Vision-Language Model) evaluator and Hallucination Analyst.

### Task
Your task is to generate a set of **Rubrics (Evaluation Criteria)** for an image captioning task.
You will be provided with:
1.  **Ground Truth Caption (GT):** A factual, accurate description of the image.
2.  **Model Response Pool:** A collection of captions generated by various VLMs. These responses may contain hallucinations, perceptual errors, or correct details.

Your goal is to identify **common or severe perceptual errors** in the `Response Pool` by comparing them against the `Ground Truth`, and then formulate strict criteria to penalize these errors.

### Process
1.  **Analyze Errors:** Scan the `Model Response Pool` to find discrepancies against the `Ground Truth`. Focus on:
    *   **Hallucinations:** Objects mentioned in responses but not present in the GT.
    *   **Attribute Errors:** Wrong colors, shapes, materials, or textures.
    *   **Counting/Quantification:** Incorrect numbers of objects.
    *   **Spatial Relations:** Wrong relative positions (e.g., left vs. right).
    *   **OCR/Text:** Incorrect reading of text visible in the image.
    *   **Action/State:** Wrong interpretation of what an agent is doing.

2.  **Filter for Perception (Crucial):**
    *   **INCLUDE:** Visual perception issues (e.g., calling a "red helmet" a "blue helmet"; seeing "3 people" instead of "4"; reading "STOP" as "SHOP").
    *   **EXCLUDE:** Knowledge gaps or Entity linking issues. If the model fails to recognize a specific character (e.g., "Genshin Impact character") but correctly describes their visual appearance (e.g., "a girl with blonde hair"), do NOT create a rubric for the specific name. Focus on the visual description.

3.  **Formulate Rubrics:**
    *   Convert the identified high-frequency or severe errors into **Binary Checklists**.
    *   If models frequently hallucinate an object, create a **Negative Constraint** (e.g., "The response must NOT...").
    *   If models get an attribute wrong, create a **Positive Constraint** (e.g., "The response must identify...").

### Rubric Style Guidelines
*   **Format:** Use imperative statements. Do NOT use questions.
*   **Structure:** Start with "The response must..."
*   **Granularity:** Each rubric must check a single, atomic fact.
*   **Tone:** Objective and strict.

### Example
**Ground Truth:** A black cat sitting on a white refrigerator. There is a magnet shaped like a banana on the door.
**Response Pool Analysis:**
- Model A: "A black dog on a fridge." (Error: Dog vs Cat)
- Model B: "A black cat on a grey fridge." (Error: Grey vs White)
- Model C: "A cat near a fridge with an apple magnet." (Error: Apple vs Banana)

**Output Rubrics:**
{
  "rubrics": [
    "The response must identify the animal as a cat.",
    "The response must state that the refrigerator is white.",
    "The response must identify the magnet shape as a banana.",
    "The response must NOT mention the presence of a dog or an apple."
  ]
}

### Output Format
Return the result strictly in valid JSON format without markdown code blocks.
{
  "rubrics": [
    "string",
    "string"
  ]
}

Here is the data for the current image:

[Ground Truth Caption]
{gt_caption}

[Model Response Pool]
1. {response_1}
2. {response_2}
3. {response_3}
...
8. {response_8}

Please generate the perception rubrics based on the analysis of the responses above.

9.3 Panel of Judges Prompt↩︎

To ensure the objectivity and correctness of the generated rubrics, a panel of models (Gemini-3-Pro [9], GPT-5.2 [10], Seed-1.8 [35]) performs a cross-verification using the following prompt.

**Role:**
You are the "Expert Visual Truth Adjudicator". Your task is to perform a rigorous comparative analysis of multiple AI-generated image descriptions against a provided image to identify the most faithful representation.

**Evaluation Dimensions:**
1. **Factuality:** Are there hallucinations? (e.g., objects, colors, or text that don't exist).
2. **Spatial Precision:** Are positional relationships (left, right, above, behind) accurate?
3. **Attribute Accuracy:** Are textures, materials, lighting, and colors correctly identified?
4. **Detail Density:** Does the caption capture nuanced elements without being redundant?

**Task Workflow:**
1. **Independent Verification:** Analyze the image first, then audit each Candidate (1, 2, and 3) individually.
2. **Conflict Resolution:** Identify discrepancies between candidates (e.g., Candidate 1 says 'vintage', Candidate 2 says 'modern'). Inspect the image to resolve these.
3. **Ranking:** Select the "Best" baseline based on the highest fidelity to the visual evidence.

**Input Candidates:**
[Candidate 1]: {candidate_1_text}
[Candidate 2]: {candidate_2_text}
[Candidate 3]: {candidate_3_text}

**Strict Output Format:**
You must output your response in valid XML format only. No preamble, no markdown formatting outside the XML, and no conversational filler.

**XML Output Schema:**
<voting_result>
    <analysis>
        <candidate_1_critique>Briefly note strengths/hallucinations for C1.</candidate_1_critique>
        <candidate_2_critique>Briefly note strengths/hallucinations for C2.</candidate_2_critique>
        <candidate_3_critique>Briefly note strengths/hallucinations for C3.</candidate_3_critique>
    </analysis>
    <best_candidate_id>Candidate ID (1, 2, or 3)</best_candidate_id>
    <rationale>A concise explanation of why this candidate won, specifically citing why it outperformed the others in terms of accuracy or detail.</rationale>
</voting_result>

9.4 Evaluation Prompt↩︎

We utilize GPT-OSS-120B [31] to evaluate models’ generated captions using the following prompts.

You are an expert Rubric Evaluator for Vision-Language Models.

### Task
Your task is to verify whether a model's generated **Caption** satisfies a specific set of **Rubrics** (Evaluation Criteria).
You will receive three inputs:
1.  **Model Caption:** The text description generated by the model.
2.  **Group A (Critical Rubrics):** A list of fundamental perception criteria. These are "bottom-line" facts.
3.  **Group B (Granular Rubrics):** A list of fine-grained or high-frequency error checks.

### Judgment Logic
For each rubric in both groups, determine if the **Model Caption** complies with the requirement.
*   **True (Pass):** The caption explicitly meets the criteria or implies it without ambiguity.
*   **False (Fail):** The caption contradicts the criteria, fails to mention a required element, or triggers a negative constraint.

**Handling Different Rubric Types:**
1.  **Positive Constraints** (e.g., "Must identify the car as red"):
    *   Pass: "A red car is parked..."
    *   Fail: "A blue car..." (Contradiction) OR "A car is parked..." (Missing specific detail).
2.  **Negative Constraints** (e.g., "Must NOT mention a dog"):
    *   Pass: "A cat sits on the mat." (No dog mentioned).
    *   Fail: "A dog and a cat..." (Hallucination detected).

### Crucial Requirement
You must evaluate **Group A** and **Group B** independently and return the results in separate lists. The order of boolean results in the output must strictly match the order of the input rubrics.

### Output Format
Return the result strictly in valid XML format. Do not use Markdown code blocks.
<Assessment>
  <GroupA>
    <Result>true</Result>
    <Result>false</Result>
    <!-- Add more Result tags matching the number of rubrics in Group A -->
  </GroupA>
  <GroupB>
    <Result>true</Result>
    <Result>true</Result>
    <!-- Add more Result tags matching the number of rubrics in Group B -->
  </GroupB>
</Assessment>

Please evaluate the following caption against the provided rubric groups.

[Model Caption]
{caption}

[Group A: Critical Rubrics]
{group_a_rubrics}

[Group B: Granular Rubrics]
{group_b_rubrics}

10 Human Annotation Feedback↩︎

To ensure the high quality of the benchmark, we involved human annotators in the loop. Given the extreme complexity of the images and the exceptional length of the golden captions (averaging 770.42 words), we employed the “Model-Ensemble-Vote-then-Human-Refine” pipeline. We utilized state-of-the-art multimodal models (specifically Gemini-3-Pro, GPT-5.2, and Seed-1.8) to generate initial drafts via a voting mechanism, followed by meticulous human verification.

Annotators reported that the AI-generated drafts were surprisingly sophisticated, significantly reducing the need for structural rewriting. However, the process introduced specific challenges regarding vigilance and fine-grained verification.

10.0.0.1 Hard Cases and Visual Nuances.

The primary difficulty lay in fine-grained visual semantic alignment, particularly in regions with blurred edges, complex lighting, or severe occlusion. Annotators identified three recurrent types of “hard cases”:

• Material and Boundary Misinterpretation: Models occasionally merged ephemeral visual features with solid objects. A cited example involved a racing car where the model incorrectly described the “dust kicked up by the wheels” as a physical extension of the car’s bodywork.

• Precise Spatial Reasoning: Subtle prepositional errors were common. For instance, a model described a pig as standing “outside the pen,” whereas a closer inspection revealed it was actually standing “at the doorway” (threshold ambiguity).

• Hallucination in Low-Visibility Areas: In shadowed or blurry regions, models tended to hallucinate specific, irrelevant objects to complete the scene.

10.0.0.2 Annotation Policy: Determinism over Ambiguity.

Our annotators adhered to a strict standard of determinism. Unlike models that might produce vague descriptions for unclear regions (e.g., “a blurry object”), humans preferred to delete hallucinations entirely rather than retaining ambiguous text. If an object was recognizable (e.g., via tool-assisted zooming), it was described explicitly; otherwise, it was removed to ensure the caption contained only grounded, high-confidence information.

10.0.0.3 Diversity of Caption Styles.

Interestingly, annotators noted that the golden captions naturally exhibited distinct stylistic modalities, reflecting the versatile capabilities of the underlying models. The captions generally fell into two categories:

• Literary Narrative: Highly fluent, prose-style descriptions that focus on immersion and flow. These captions tend to be exceptionally long and use varied sentence structures to weave visual details into a cohesive story.

• Structured Representation: Captions that utilize Markdown formatting (e.g., bolding key terms, using bullet points for distinct regions) to present information in a highly organized, hierarchical manner.

We preserved this stylistic diversity in the final benchmark to evaluate models on both narrative generation and structured information extraction.

11 Additional Experimental Results↩︎

Table 4 presents the comprehensive evaluation results across all models.

12 Qualitative Examples↩︎

We provide concrete examples of the generated rubrics across diverse domains in Figure 12 and Figure 13.

As shown in the figures, our benchmark covers seven major categories, ranging from daily natural scenes to highly specialized STEM diagrams and logic puzzles. (a) For each image, we generate a comprehensive set of fine-grained rubrics. The items marked with the “OK” icon (Must-Right) represent core factual elements and primary subjects that are essential for a basic understanding of the scene. (b) The items marked with the “Thumbs-up” icon (Easy-Wrong) target more challenging details, including spatial relationships, fine-grained text recognition, negative constraints (e.g., “must NOT mention...”), and complex logical reasoning. These rubrics are specifically designed to be “Easy-Wrong” for current large multi-modal models, effectively exposing hallucinations and subtle comprehension errors. For instance, in the “Structured Data” and “STEM & Expert” cases, the rubrics require precise reading of axis scales, curve styles, and hierarchical biological relationships, which demand a high level of visual-logical alignment.

References↩︎