1 Introduction↩︎

Visual counting [1]–[3], the ability to quantify instances of specific categories in an image, is an important skill that reflects a key aspect of visual intelligence. For modern Multimodal Large Language Models (MLLMs) [4]–[9], counting is not merely a numerical task but a rigorous test of fine-grained grounding [10], [11] and spatial reasoning [12]. While MLLMs excel at qualitative scene understanding such as Visual Question Answering (VQA) [13], their limited quantitative precision remains a critical barrier for applications such as logistics [14], safety monitoring [15], and retail analytics [16], where numerical errors carry high costs.

Despite the rapid evolution of MLLMs [4]–[7] such as the GPT series [17] and the Qwen-VL series [5], [6], their ability to perform reliable counting remains surprisingly fragile [1], [2]. Although these models exhibit strong zero-shot performance in general scene description, they often struggle with precision when tasked with enumerating objects. This counting limitation frequently manifests as numerical hallucinations, where the model confidently reports a count that bears little resemblance to the true quantity, undermining reliability in count-sensitive environments. Recent benchmarking efforts [1]–[3], including CountBench [2] and CountQA [1], have begun to address this challenge but remain constrained by limited object diversity and a focus on simple perceptual tasks. They often fail to reveal complex failure modes that arise when models face property constraints, logical constraints, or adversarial visual conditions.

To address these gaps, we introduce HoloCount, a comprehensive benchmark designed to rigorously evaluate the counting competence of MLLMs. Unlike prior benchmarks [2], [3], [18] (see detailed comparison in Table 1), HoloCount is organized around a three-level hierarchical taxonomy that shifts the focus from mere superficial perception in idealized settings to the advanced cognitive processing and environmental resilience required for true numerical grounding. These three distinct pillars evaluate equally vital facets of counting competence: (1) Semantic Counting assesses foundational skills through atomic counting (basic enumeration) and property-based filtering, testing the model’s ability to isolate target objects by fine-grained attributes such as object chromatic and material properties. (2) Analytical Counting examines the interplay of vision and logic through logical composition, requiring models to execute spatial-based reasoning, set-based operations, and conceptual calculus, such as multi-set aggregation, exclusions, and relational comparisons. (3) Robustness Testing probes the systemic limits of model reliability under adverse perceptual domains (e.g., heavy occlusion, high object density, and aerial perspectives) while explicitly evaluating the model’s resistance to deceptive counter-priors, including linguistic biases, null targets, and visual illusions. To realize this multi-dimensional diagnostic pipeline, HoloCount comprises a meticulously curated dataset of 2,480 highly precise QA pairs partitioned across 20 fine-grained subsets, as shown in Fig. 1. By structuring this extensive scale around distinct cognitive axes rather than a flat, unstructured mixture of queries, our benchmark provides a decoupled evaluation framework. Consequently, HoloCount can comprehensively review the deep-seated counting vulnerabilities of MLLMs, systematically isolating whether a model’s failure stems from a visual alignment oversight, a logical reasoning flaw, or a fragile reliance on superficial patterns.

Through an exhaustive evaluation of over 20 leading MLLMs [5], [6], we uncover a sobering reality: model performance deteriorates markedly as tasks progress from basic perception to analytical reasoning and robustness challenges. The experimental results also reveal valuable insights and provide a clear roadmap for enhancing the visual counting capabilities of MLLMs.

Our contributions are summarized as follows:

• The HoloCount Benchmark: We present HoloCount, a novel benchmark designed to rigorously evaluate the counting competence of MLLMs. Unlike prior flat benchmarks, HoloCount is organized around a hierarchical three-level taxonomy—Semantic Counting, Analytical Counting, and Robustness Testing—that shifts focus from superficial perception to advanced cognitive processing and environmental resilience.

• Fine-grained Tasks: We construct a meticulously curated dataset of 2,480 high-precision QA pairs spanning 20 fine-grained counting tasks and 1,481 visual concepts, making it the most diagnostically rich counting benchmark to date in terms of task diversity, visual concepts, and structured evaluation axes.

• Extensive Evaluation & Actionable Insights: We conduct an exhaustive empirical study of state-of-the-art models (including the Qwen-VL series, the GPT series, etc.), revealing that performance deteriorates markedly from basic perception to analytical reasoning and robustness challenges, thereby providing a sobering reality check on current MLLM counting capabilities. The benchmark and evaluation framework identify root causes of model failures, offering a clear roadmap for developing truly grounded, numerically reliable systems.

2 Related Work↩︎

Visual Counting in VLMs. Traditional visual counting research [18], [20], [24] was largely confined to specialized domains with fixed, narrow categories, such as crowd counting [18] or cell enumeration [25], [26]. Early methodologies often relied on task-specific architectures, utilizing normalized attention weights [27] or relation networks [28] to infer object quantities. With the emergence of Large Vision-Language Models (VLMs) [29], research shifted toward leveraging the zero-shot capabilities of foundational models like CLIP [30] to count objects in an open-vocabulary manner [1], [2]. However, despite their strong contrastive alignment, these dual-encoder models often struggle to parse complex semantics or reason through free-form counting queries. The transition to Multimodal Large Language Models (MLLMs) has significantly expanded the scope of counting, allowing for more conversational and multi-step interactions. Recent works [3] have attempted to improve these capabilities through targeted data and architectural innovations. For instance, PixMo (Pixels for Molmo) [3] introduced a novel 2D pointing dataset that allows models to "count by pointing," significantly enhancing grounding and numerical accuracy through point-based supervision.

Counting Benchmarks. Early visual counting benchmarks were predominantly domain-specific, focusing on high-density scenarios such as crowd analysis (e.g., ShanghaiTech [18], NWPU-Crowd [19]) or vehicle monitoring (e.g., CARPK [21]). These datasets typically formulate counting as a density map regression task restricted to a closed set of categories. To achieve category-agnostic counting, few-shot benchmarks such as FSC-147 [23] and FSCD-LVIS [31] were introduced, requiring models to count arbitrary objects given a few visual exemplars. Some works [32]–[35] focus on counting tasks in more modalities such as audio and video. With the emergence of MLLMs, the paradigm has shifted toward open-vocabulary counting guided by natural language instructions [1]–[3], [36], [37]. CountBench [2] and CountQA [1] pioneered this space by evaluating zero-shot counting via VQA formats, while PixMo-Count [3] utilized point-based supervision to improve numerical grounding. However, these benchmarks largely focus on basic perception and lack the depth to diagnose failures in complex reasoning or robustness under deceptive conditions. In this work, HoloCount bridges this gap by introducing a three-level hierarchical evaluation that systematically decouples semantic, analytical, and adverse robustness.

3 The HoloCount Benchmark↩︎

3.1 Task Formulation↩︎

The objective of visual counting within the framework of Multimodal Large Language Models (MLLMs) is to quantify a specific subset of objects in an image \(\mathcal{I}\) based on a natural language instruction \(\mathcal{Q}\), e.g., “How many cats are there in the image”. Formally, given the input pair \((\mathcal{I}, \mathcal{Q})\), the model \(\mathcal{M}\) aims to generate a textual response \(\mathcal{A}\) that contains the predicted count \(\hat{N}\). This process is defined as: \[\mathcal{A} = \mathcal{M}(\mathcal{I}, \mathcal{Q}; \theta)\] where \(\theta\) represents the model parameters. The success of the task is measured by the alignment between the extracted numerical value \(\hat{N} \in \mathcal{A}\) and the ground-truth cardinality \(N\).

3.2 Overview↩︎

Traditional counting benchmarks [18], [20], [24] typically evaluate counting models on a limited set of fixed visual concepts, such as vehicle monitoring in CARPK [21], where models count cars in parking lots. With the emergence of MLLMs, many VQA benchmarks [13] have been proposed to evaluate the visual understanding capability of these models, including a few examples of counting questions. Recently, CountBench [2] and CountQA [1] have been specifically proposed to focus on the counting problem. However, these benchmarks remain limited by narrow object diversity (see comparison of visual concepts in Table 1) and a focus on simple perception. They often fail to capture the complex failure modes that occur when models are subjected to property constraints, logical constraints, or adversarial visual conditions. Most importantly, these benchmarks adopt a flat structure; the evaluation only yields a single accuracy score, providing no insight into where or why the model fails. Consequently, they cannot offer effective guidance for improving the counting capability of the model.

To bridge this gap, we propose HoloCount, a comprehensive and rigorous benchmark designed to holistically assess the counting ability of MLLMs. HoloCount is structured around a three-level hierarchical taxonomy that shifts the focus from perception to cognitive counting ability and adverse scenarios, including: (1) Semantic Counting assesses foundational skills through atomic counting (basic enumeration) and property-based filtering, testing the model’s ability to isolate target objects by fine-grained attributes such as object chromatic and material properties. (2) Analytical Counting examines the interplay of vision and logic through logical composition, requiring models to execute spatial-based reasoning, set-based operations, and conceptual calculus, such as multi-set aggregation, exclusions, and relational comparisons. (3) Robustness Testing probes the systemic limits of model reliability under adverse perceptual domains (e.g., heavy occlusion, high object density, and aerial perspectives) while explicitly evaluating the model’s resistance to deceptive counter-priors, including linguistic biases, null targets, and visual illusions. In particular, HoloCount contains 20 subsets and 2,480 QA samples. As shown in Table 2, Semantic Counting includes 6 subsets covering atomic counting, chromatic, material, scale, action and state attributions. Analytical Counting evaluates models from the perspectives of spatial-based reasoning (Visual-Prompt Region Grounding, Coordinate-Prompt Region Grounding, Relative Canonical Orientation) and set-based logic reasoning (Differential Comparison, Joint-Set Aggregation, Complementary Exclusion, Categorical Cardinality). Robustness Testing focuses on adverse conditions (High-Density Enumeration, Aerial Perspective Shift, Small-Scale Enumeration, Partial Object Occlusion) and hallucination (Null-Target Prompting, Linguistic Prior Conflict, Visual Distractor Illusion). We also present example tasks in Fig. 2 for better understanding of the HoloCount benchmark. HoloCount not only provides a more diverse set of evaluations but also serves as an effective tool for analyzing the failure modes of counting models.

3.3 Data Curation Process↩︎

To construct a high-quality and reliable benchmark, we adhere to two core principles during the development of HoloCount: (1) Ambiguity Elimination: Visual counting often suffers from semantic or boundary ambiguity (e.g., whether to count a highly occluded object). We prune samples with vague instructions or ambiguous visual evidence, ensuring that each query has a singular, well-defined numerical answer. (2) Category and Scenario Diversity: We aim for maximum coverage across diverse object categories and environmental contexts to test the true generalization of MLLMs.

General Data Pipeline. To ensure the diversity, difficulty, and diagnostic depth of HoloCount, we implement a multi-stage data curation pipeline, transitioning from large-scale collection to rigorous human-in-the-loop verification. The general data pipeline is shown in Fig. 3 and contains four core steps: (1) Data Collection/Generation. To meet different evaluation purposes and ease the data construction process, we adaptively collect images from the internet, detection datasets, specialized benchmarks, and manual synthesis for specific purposes. (2) Data Cleaning and Filtering. We exclude images with extreme sizes or aspect ratios. Moreover, we also exclude images where target objects occupy extreme scales (too small for clear identification or too large to be reliably perceived). (3) QA Generation and Filtering. We use advanced MLLMs to generate QA pairs and employ another model to check the correctness of the generated QA pairs. (4) Rigorous Human Verification. The final and most critical stage involves exhaustive human audit. Each data sample undergoes manual review by expert annotators. The annotators are tasked with correcting any numerical inaccuracies and ensuring that the queries are linguistically natural and unambiguous.

Note that the data pipeline described above represents the general strategy for HoloCount benchmarks. Some steps may be deliberately broken for specific subsets to serve particular evaluation purposes. For example, we intentionally retain small visual objects in the subset of Small-Scale Enumeration dedicated to small object counting. We will provide detailed introductions for these subset-specific designs later.

3.3.1 Semantic Counting↩︎

Semantic Counting aims at evaluating whether the model has the ability to bind the semantics of objects in an image (see Fig. 2 for example). Existing counting benchmarks belong to this taxonomy.

Atomic Counting. This is the most common counting problem, involving basic concepts such as “how many apples are there in the image?” This subset is very close to CountBench [2], PixMo Count [3], and CountQA [1]. To construct this subset, we first sample images from the validation set of LVIS [38]. We constrain the maximum count number to 20 and filter out samples where the objects occupy an area of the image less than \(0.1\%\) or more than \(85.0\%\). Since detection datasets often contain miss-annotated objects that significantly influence the counting training signal, we utilize MLLMs to first check the correctness of the annotations. Then, human experts double-check the final annotations to ensure the quality of the dataset.

Attribution: Chromatic, Material, Scale, Action & State, and General Semantic. Attributes represent core semantic dimensions for counting. In this work, we consider chromatic, material, size, action and state, and other general semantic attributes as key factors for visual object counting. To construct attribute-based counting QA pairs, we prompt a strong MLLM, Gemini-3-Flash-Preview, to generate candidate questions and answers conditioned on each attribute type. We then employ another model as a verifier to check the correctness of the generated QA pairs, filtering out inconsistent or invalid ones. Finally, we conduct manual verification by human annotators to ensure high quality and reliability of the final QA dataset.

3.3.2 Analytical Counting↩︎

Existing counting benchmarks [1]–[3] primarily assess perceptual counting, i.e., directly enumerating objects of a given category. However, real-world visual reasoning often requires more than direct perception: it demands analytical counting, where the model must integrate visual information with logical operations, spatial constraints, or set-based reasoning. For example, answering “How many more red blocks than blue blocks?” involves not only counting each color but also performing a relational comparison. Conventional benchmarks fail to evaluate such cognitive counting abilities because they assume a flat, direct mapping from image to count. They neither introduce compositional logic nor require relational thinking. As a result, a model that performs well on atomic counting may still fail catastrophically when faced with analytical counting tasks—a failure mode that flat accuracy scores cannot reveal. HoloCount explicitly targets this gap by introducing Analytical Counting, which comprises two broad categories: (i) spatial-based reasoning and (ii) set-based logical reasoning.

Spatial-based Analytical Counting. Spatial-based analytical counting requires the model to count objects within a region of interest (ROI) that is defined not by object categories but by spatial reference. This tests the model’s ability to ground counting instructions to specific visual locations. We design three subsets: (1) Visual-Prompt Region Grounding: The ROI is marked by a visual prompt (e.g., a bounding box). The model must count only the objects falling inside that region, ignoring visually similar distractors outside. This evaluates visuospatial alignment without textual coordinates. (2) Coordinate-Prompt Region Grounding: The ROI is specified via numerical coordinates (e.g., “in the region <bbox>x_min y_min x_max y_max</bbox>”). The model must translate symbolic spatial descriptions into pixel-level counting. This tests the integration of language-based geometry with visual enumeration. (3) Relative Canonical Orientation: The ROI is defined relative to object orientations (e.g., “in the bottom-left quadrant of the image”). This requires the model to recognize canonical directions and apply them as a spatial filter. These tasks are absent from prior counting benchmarks, which either assume global counting or rely on trivial region definitions. By contrast, HoloCount forces models to jointly perform spatial reasoning and enumeration, revealing weaknesses in grounding and attention. Obviously, the construction of visual grounded counting requires the help of bounding boxes of objects. We therefore start from LVIS [38] and utilize an MLLM to check the annotations. Then, we build the three subsets according to the bounding boxes. After that, human annotators further check the generated QA pairs.

Set-based Analytical Counting. Set-based analytical counting involves operations over multiple object sets, requiring the model to reason about intersections, unions, complements, or cardinality comparisons. We design four subsets: (1) Joint-Set Aggregation: The question asks for the total count of objects from multiple categories (e.g., “How many A and B together?”). The model must count each category separately and then sum, a simple but often overlooked compositional operation. (2) Complementary Exclusion: The model must count objects that belong to one set but not another. This tests the ability to apply logical negation in the visual domain. (3) Categorical Cardinality: The model counts the number of distinct object types or categories present in the image, independent of their instance frequencies. For example, “How many kinds of fruits are in the image?” This requires the model to recognize and categorize objects at a type level, then determine the cardinality of the set of categories, a higher-level cognitive skill beyond mere instance counting. (4) Differential Comparison: The model computes the numerical difference between two sets (e.g., “How many more apples than oranges?”). This goes beyond comparison to exact arithmetic on perceived quantities. Existing benchmarks rarely include such logical compositions. When they do, they treat them as incidental rather than systematic. HoloCount provides controlled subsets for each operation, enabling fine-grained diagnosis of where models fail—whether in visual perception, logical parsing, or arithmetic execution. The building pipeline of these subsets is similar to that of semantic counting.

3.3.3 Robustness Testing↩︎

While Semantic Counting evaluates attribute-based filtering and Analytical Counting examines spatial and set-based logical reasoning, both operate under relatively clean, well-structured visual conditions. In real-world deployments, however, MLLMs face adverse perceptual domains (e.g., high density, occlusion, and aerial views) and deceptive priors (e.g., language biases, null targets, and visual illusions). To thoroughly assess the robustness of MLLMs’ counting ability under such challenges, we construct seven additional subsets within HoloCount that explicitly target adverse visual conditions and counter-grounded linguistic priors.

High-Density Enumeration. For dense scenarios, we initially planned to sample images from existing high-density benchmarks (e.g., ShanghaiTech [18] for crowd counting). However, we found that many of those samples are ambiguous and difficult to verify. Therefore, we developed a script to generate synthetic symbols on a blank canvas, mimicking dense object distributions.

Aerial Perspective Shift. Most existing counting benchmarks focus on natural, ground-level viewpoints (e.g., street scenes, indoor environments). To evaluate counting under out-of-distribution (OOD) visual conditions, we sample images from CA44 [39], where one subset provides aerial drone-view perspectives. These top-down views introduce significant changes in object appearance, scale, and spatial layout compared to typical counting datasets. The original data contain some noise and inaccurate QA pairs, so we manually filter out these problematic samples to ensure correctness.

Small-Scale Enumeration. Counting tiny objects is particularly challenging for MLLMs because such objects occupy few pixels, offering limited visual features for recognition and enumeration [40]. Most existing benchmarks focus on reasonably sized objects, leaving small-scale counting under-evaluated. To address this gap, we sample images that contain objects smaller than 32\(\times\)​32 pixels from the Object365 [41] validation set. We then use an MLLM to filter out incorrect counting samples and employ human annotators to verify the final QA pairs.

Partial Object Occlusion. In real-world scenes, objects are often partially occluded, requiring models to infer hidden parts or count based on visible evidence [42], [43]. However, current counting benchmarks rarely consider occlusion as a systematic variable. To evaluate counting under occlusion, we start from the validation set of Object365 [41] and retain samples where objects exhibit heavy occlusion (IoU \(>\) 0.85). We then use an MLLM to filter out incorrect counting samples and employ human annotators to verify the final QA pairs.

Null-Target Prompting. This subset evaluates whether a model hallucinates when asked about a concept absent from the image. In real-world applications, models may confidently produce a positive count for objects that do not exist, revealing a critical failure in visual grounding. To systematically probe this issue, we construct queries targeting categories that are either semantically related to the scene or completely out of context. Every query is carefully verified to have zero ground-truth instances. A robust model should correctly answer zero, whereas a hallucinating model may produce a non-zero response, exposing its reliance on language priors rather than genuine visual perception.

Linguistic Prior Conflict. In some cases, models may answer based on language priors rather than image content [44]–[46]. For example, a lobster typically has two claws. If we provide an image where a lobster has only one claw, does the model follow the visual evidence or the linguistic prior? Similarly, we probe common-sense knowledge such as the number of legs on a cat. This subset tests whether the model can override strong language priors when they conflict with the visual input. To construct this dataset, we first manually design a prompt set describing counterfactual objects (e.g., a lobster with one claw, a three-legged cat). We then use Gemini Image to generate corresponding images and build QA pairs, followed by human verification to ensure visual plausibility and correct ground-truth.

Visual Distractor Illusion. In this subset, objects are visually similar but represent different concepts, such as variations in material (plastic vs. real fruit). These subtle distinctions challenge the model’s fine-grained discrimination ability, requiring it to count only the target category while ignoring visually distracting confounders. We collect samples from [47], then convert them to QA format with carefully designed instructions that specify the target attribute. For example, an image containing both real apples and plastic decorative apples may prompt the question: “How many real apples are in this image?” All QA pairs undergo rigorous manual verification to ensure unambiguous grounding.

4 Experiments↩︎

4.1 Setup↩︎

Models. We evaluate a comprehensive collection of state-of-the-art Multimodal Large Language Models (MLLMs) on our HoloCount benchmark. Our evaluation spans both open-source and proprietary frontier models to provide a holistic view of current counting capabilities. (1) Open-Source Models. We include widely adopted open-source MLLMs from leading families. Specifically, we evaluate the Qwen-VL series [5], including Qwen2.5-VL [48], Qwen3-VL [5], and Qwen3.5 [49], InternVL series [50], Gemma3 series [51], MiniMax-M3, and Kimi series [52]. All open-source models are evaluated with their official checkpoints under zero-shot settings without any task-specific fine-tuning. (2) Proprietary API Models. To benchmark against proprietary state-of-the-art systems, we evaluate leading commercial models via their official APIs. These include GPT-4.1, GPT-5.4, and GPT-5.5 from the GPT series [17], Gemini-2.5-Pro, Gemini-3-Flash-Preview and Gemini-3.1-Pro-Preview from the Gemini family, as well as the Claude family. For proprietary models, we use default generation parameters and follow each vendor’s recommended best practices for visual question answering.

Evaluation Details. For each model, we perform zero-shot inference on all 2,480 QA pairs across the 20 subsets of HoloCount. Models receive the input image and the counting question as a plain text prompt without any few-shot examples or chain-of-thought instructions; the system prompt can be seen in the Appendix. We extract the numerical answer from the model’s generated response using regular expression matching. If the response contains no valid number, we record an incorrect prediction. Following prior work [1], [2], we report exact match accuracy as the percentage of samples where the predicted count exactly matches the ground-truth count. For open-source models, all experiments are conducted on NVIDIA H800 GPUs.

4.2 Main Results and Analysis↩︎

2pt

Overall Performance Comparison. Table 3 reveals that the most advanced open-source models now rival or even surpass proprietary systems on the HoloCount benchmark. The top open-source entry, Qwen3.5-397B-A17B, achieves a macro average of 76.9%, closely followed by Qwen3.5-27B (75.9%) and Qwen3.5-122B-A10B (75.4%). Among closed-source models, Gemini-3-Flash-Preview and Gemini-3.1-Pro-Preview lead with 74.8% and 74.7%, respectively. This near tie indicates that the performance gap between well-trained open-source families and commercial APIs has substantially narrowed. Scaling remains effective: within the Qwen3.5 family, moving from 4B to 27B yields a gain of \(7.3\%\), and the largest 397B MoE model further pushes the state-of-the-art.

The Catastrophic Failure Mode of High-Density Contexts. Counting in high-density scenes (Dense) remains a universal weakness. Even the strongest models show dramatic degradation. For example, Qwen3.5-27B achieves only 20.0% exact accuracy on this subset, while its average across all other subsets exceeds 80%. Most open-source models score below 10%, and several (e.g., Gemma3-27B, Qwen2.5-VL-72B) fall to 0–1%. The best closed-source performer, Gemini-3.1-Pro-Preview, reaches 49.0% but is still far from reliable. To further diagnose this breakdown, Table 4 reports Mean Absolute Error (MAE) and Relative Accuracy at 5% and 10% thresholds. Open-source models exhibit MAE values of 32–50, meaning their predictions deviate by roughly one-third to half of the true count. RA@5% stays below 20% for all but Gemini-3.1-Pro-Preview, which achieves 79.0% RA@5% and a low MAE of 10.46. These results confirm that dense scenes cause systematic under-counting, likely because current vision encoders cannot maintain fine-grained spatial distinctions when objects heavily overlap or crowd the canvas.

Logical Execution vs. Baseline Perception Disconnect. A clear dissociation exists between atomic counting and higher-order analytical tasks. Nearly all models perform strongly on Atomic Counting (Atom.), often exceeding 85%. However, performance drops sharply on tasks that demand set operations or comparisons. For instance, InternVL3.5-38B falls from 79.7% on atomic counting to 35.0% on Differential Comparison (Diff.); Qwen3.5-27B drops from 90.7% to 76.0%. Even the best models show a consistent gap, confirming that the primary bottleneck is not visual perception per se but the symbolic reasoning and arithmetic composition required after objects are perceived.

The Inverse Performance Paradox in Null-Target Presentation. The Null.T. subset reveals a striking inversion: smaller or older open-source models often outperform advanced proprietary engines when asked to count objects that are entirely absent. Lightweight models such as Qwen3-VL-8B-Instruct (96.4%) and Qwen2.5-VL-7B-Instruct (95.6%) achieve near-perfect scores. In contrast, Gemini-3-Flash-Preview drops to 62.4%, and Gemini-3.1-Pro-Preview collapses to 55.2%—the lowest among all models. This paradox suggests that aggressive optimization for detailed scene description and object retrieval inadvertently increases over-confidence in the presence of any visual signal. Larger models become reluctant to output zero, even when no target exists, whereas simpler models faithfully follow the instruction.

Resisting the Siren Call of Linguistic Priors. The Linguistic Prior Conflict (Ling.P.) subset tests whether models can override strong language priors (e.g., “a lobster has two claws”) when the image contradicts them. Most models perform poorly, with scores in the 20–40% range for Gemma3, Qwen2.5, and early Qwen3 families. This indicates a general “lazy” reliance on textual statistics rather than visual verification. However, the Gemini-3 series stands out: Gemini-3-Flash-Preview achieves 69.9%, and Gemini-3.1-Pro-Preview achieves 73.6%, the highest among all evaluated models. This gap implies that the Gemini architecture incorporates a stronger vision-first arbitration mechanism that can suppress incompatible linguistic priors when they conflict with observed pixels.

Vulnerability to Visual Distractors and False-Positive Susceptibility. The Visual Distractor Illusion (Vis.D.) subset measures resilience against visually similar but semantically distinct distractors (e.g., real vs. plastic fruit). All models suffer a significant drop compared to atomic counting. Smaller models are hit hardest: Gemma3-4B scores only 38.0%. Among open-source models, the best performer is Qwen3.5-397B-A17B (80.0%), while leading closed-source models like Gemini-3-Flash-Preview (67.5%) and GPT-5.5 (76.0%) also show substantial room for improvement. This pattern indicates that although modern vision encoders capture global scene semantics, their attention mechanisms often fail to perform fine-grained attribute filtering, leading to over-counting of distractors that share a coarse visual match.

Impact of Thinking Mode. We investigate the effect of extended thinking on counting performance by comparing Qwen3.5 models in thinking mode versus non-thinking (instruct) mode across six model scales. As shown in Fig. 4, thinking mode consistently improves counting accuracy across all scales, with gains ranging from 10.6 to 15.4 absolute percentage points. Notably, smaller models benefit more (4B: +15.4, 9B: +13.9), but even the largest 397B model gains +13.1 points. This suggests that counting tasks require deliberate step-by-step reasoning, e.g., enumerating objects, filtering by attributes, or performing set operations, which is poorly approximated by direct pattern matching. Thinking mode effectively compensates for the lack of built-in counting primitives in standard MLLMs.

5 Conclusion↩︎

Existing counting benchmarks suffer from flat structures, narrow visual concepts, and a focus on perceptual enumeration, leaving complex failures largely undiagnosed. To bridge this gap, we introduced HoloCount, a hierarchical benchmark organized into three core pillars: Semantic Counting, Analytical Counting, and Robustness Testing. With 20 fine-grained subsets and 2,480 carefully curated QA pairs spanning 1,481 visual concepts, HoloCount provides a decoupled evaluation framework that isolates where and why models fail. Our extensive evaluation of over 20 MLLMs confirms that current models remain far from reliable counting, underscoring the urgent need for architectures that prioritize cognitive reasoning and resilience over superficial perception.

6 Benchmark Statistics↩︎

In this section, we conduct a thorough statistical analysis of our benchmark to provide a holistic understanding of its composition and inherent properties. A comprehensive examination of these statistics is indispensable, as it not only reveals the diversity and potential biases of the dataset but also establishes a foundational context for interpreting the experimental results in subsequent sections. Our analysis is structured along three key perspectives: the overall count distribution, the semantic coverage of visual concepts, and the numerical scale range for Small-Scale Enumeration tasks.

6.1 Count Distribution↩︎

Fig. 5 illustrates the distribution of ground-truth count values across all 2,480 samples in HoloCount. The counts span a wide range from 0 to 149, with a mean of 8.06 and a median of 3. The distribution exhibits a long-tailed pattern: 64.8% of samples fall within the [1, 5] range, reflecting common real-world counting scenarios, while 11.9% of samples have a count of 0, mainly corresponding to the Null-Target Presentation subset where queried objects are absent from the image. Beyond 20, the tail extends to 149, primarily contributed by the dense and aerial counting subsets. This deliberate design covers both low-count fine-grained discrimination and high-count estimation, enabling comprehensive evaluation of counting capabilities across diverse difficulty levels.

6.2 Visual Concept Word Cloud↩︎

Visual Concept Extraction. To quantify the semantic diversity of HoloCount, we extract the core visual concepts from each counting question using GPT-3.5-turbo (see Fig. 7). Specifically, we prompt the model with a system instruction and few-shot examples to parse noun phrases from questions while normalizing to singular form, preserving distinguishing modifiers (e.g., “yellow flower”, “wooden chair”), and removing spatial or locational phrases (e.g., “on the wall”, “in the foreground”). For composite questions involving comparison or aggregation (e.g.., “How many more X than Y?”), all compared concepts are extracted independently. This process yields 1,481 unique visual concepts in 2,480 samples.

Visual Concept Distribution. Fig. 6 visualizes the extracted concepts as a word cloud. The distribution reveals broad semantic coverage spanning common objects (car, person, cup, plate), animals (zebra, bird, elephant, duck, cat), body parts (leg, tail, wheel), and fine-grained attributes. Notably, the distribution is highly long-tailed: 74.7% of concepts (1,105 out of 1,481) appear only once, indicating that HoloCount avoids over-concentration on a few dominant categories and provides broad coverage of the visual world.

6.3 Scale Distribution for Small-Scale Enumeration↩︎

Fig. 8 shows the object scale distribution in the small-scale enumeration subset (101 samples, 995 bounding boxes), where all targets are constrained to an area of \(\leq\)​1024 pixels\(^2\) (equivalent to 32\(\times\)​32). The \(\sqrt{\text{area}}\) ranges from 4.2 to 31.9 pixels with a median of 19.3 pixels. The distribution is concentrated between 8–32 pixels: 30.8% of objects fall in the [8, 16) pixel range, 38.0% in [16, 24), and 26.7% in [24, 32), while 4.5% are extremely small (\(<\)​8 pixels). This subset specifically targets the challenging regime where objects occupy less than 0.16% of the image area on average, testing whether models can detect and enumerate targets that are barely perceptible at typical display resolutions.

7 Data Construction Details↩︎

7.1 QA Generation Prompts↩︎

Unlike basic object counting, which can be derived from existing detection datasets with bounding box annotations, attribute-based semantic counting and analytic counting lack readily available datasets. To construct these subsets, we leverage Gemini-3-Flash-Preview to generate question-answer pairs. This LLM-assisted generation paradigm enables rapid and scalable construction of counting datasets: by simply modifying the prompt templates, one can efficiently produce large-scale QA pairs covering diverse counting dimensions without labor-intensive manual annotation. The prompts used for generation are shown in Fig. 9 and Fig. 10. Note that the sub-capability names in the prompts may differ slightly from those used in the final benchmark taxonomy, as the prompts were designed to guide QA generation rather than to define the evaluation taxonomy.

7.2 Linguistic Prior Conflict Image Generation Pipeline↩︎

In some cases, models may answer based on language priors rather than image content. For example, a lobster typically has two claws — if we provide an image where a lobster has only one claw, does the model follow the visual evidence or the linguistic prior? To systematically evaluate this behavior, we construct a linguistic prior conflict subset through the following pipeline. We first manually curate a triplet (object, part, prior_count) encoding strong common-sense associations (e.g., dog/legs/4, spider/legs/8). For each triplet, we generate abnormal counts that deviate from the prior (typically \(\pm\)​1 to \(\pm\)​3), then fill them into prompt templates that explicitly request photorealistic images depicting the specified abnormal quantity with all parts fully visible. We use Gemini to generate images from these prompts, followed by manual verification to retain only images where the ground-truth count is visually unambiguous. The resulting evaluation question takes the form “How many {part} does the {object} in the image have?”, a model relying on linguistic priors would answer with the prior count, while a faithful visual perceiver would report the actual depicted count.

7.3 Counting System Prompt↩︎

During evaluation, all models receive the same system prompt to ensure a fair and consistent output format across different architectures. The prompt is designed to elicit a single integer response, minimizing ambiguity in answer extraction:

This strict integer-only constraint avoids the need for post-hoc answer parsing (e.g., extracting numbers from natural language responses) and ensures that evaluation metrics directly reflect the model’s counting ability rather than its instruction-following capability.

8 Failure Case↩︎

We provide qualitative failure cases of Gemini-3.1-Pro across all subsets of HoloCount in Figures 11, 12, and 13. These examples illustrate representative error patterns including attribute misinterpretation, spatial grounding failure, logical reasoning breakdown, language prior conflict, and perception degradation under adverse conditions.

9 Future Work↩︎

In this work, we present HoloCount, a comprehensive benchmark for evaluating the counting capabilities of multimodal large language models across semantic counting, analytic counting, and robustness testing. Our evaluation reveals significant gaps in current models, particularly in analytical reasoning, small-scale enumeration, and resistance to linguistic priors. Building on these findings, we identify several promising directions for future research:

Video Counting. HoloCount currently focuses on static images. However, real-world counting often occurs in dynamic environments where objects enter, exit, or become temporarily occluded across frames. Extending the benchmark to video sequences would enable evaluation of temporal counting abilities, such as tracking cumulative arrivals, counting unique individuals across frames, and maintaining accurate counts despite motion blur or transient occlusions.

Grounded Counting with Localization. Our current evaluation only requires models to output a single integer. A natural extension is to require models to jointly count and localize — producing not only the total count but also the spatial location (e.g., bounding boxes or points) of each counted instance. This would enable deeper diagnosis of counting errors: whether a model counts incorrectly because it misidentifies objects, double-counts the same instance, or fails to detect certain targets entirely.

Multi-step Compositional Counting. While HoloCount includes basic analytic counting (comparison, aggregation, exclusion), future work can explore more complex compositional chains that require multiple reasoning steps. For example, questions like “How many objects satisfy condition A but not condition B, and are larger than the average size of category C?” demand sequential constraint evaluation, arithmetic operations, and cross-category comparison — pushing models toward genuine multi-step visual reasoning rather than pattern-matched responses.

Counting with World Knowledge. Current counting tasks define targets by visual attributes or explicit category names. A cognitively richer direction involves counting that requires external world knowledge. For instance, “How many mammals are in this image?” requires taxonomic knowledge to classify each visible animal, while “How many edible items are on the table?” demands functional knowledge about object affordances. This direction bridges visual counting with knowledge-grounded reasoning, testing whether models can integrate encyclopedic knowledge with perceptual evidence.

References↩︎