1 Introduction↩︎

Artificial Neural Networks have advanced significantly in visual reasoning since the development of deep Convolutional Neural Networks (CNNs) over a decade ago [1]. Figure 1a shows a glimpse of the early years of deep CNNs, with models that could outperform humans in tasks like object recognition. Deep Recurrent Neural Networks [2], which process and generate sequential data like text, combined with deep CNNs and image-text datasets [3]–[7] led to the rise of task-specific multi-modal models that can either provide captions [8], answer questions [9], visually reason [10], [11] based on images or generate images based on text [12]. The development of transformer architectures [13] and improvement in pre-training methods [14] gave way to task-agnostic multi-modal models that can perform all vision language tasks with minimal fine-tuning required [15], [16]. In addition to these developments, the growth of data and computational power has accelerated the development of Multi-Modal Large Language Models (MLLMs), which are claimed to achieve human-level reasoning capabilities.

Often, images in standard datasets are simple scenes (e.g., an image of a kitchen, desk, or beach scene) that do not include complex human actions or social interactions. Thus, the datasets may not capture the human behaviors and interactions that are critical to everyday visual inferences. Describing these complex scenes accurately might be more challenging for models than describing simple scenes from typical datasets. Figure 1b shows that the reported performance of the then state-of-the-art vision-language models (based on values reported in the corresponding model papers [17]–[20] published between 2017 and 2022) using the BLEU-4 [21] metric approaches human performance in such datasets. However, Figure 1c highlights the inability of the 2017 model to capture complex social interactions. The figure showcases depictions provided by a 2017 state-of-the-art image captioning model (UDC [17]) and a current MLLM (GPT-4o1 [22]) for two scenes: one from the traditional MS-COCO [3] dataset (left) and another from a movie scene portraying complex social interactions (right). Notably, the 2017 model performs reasonably well on the MS-COCO dataset, comparable to current MLLMs, but it fails to capture the complex social interactions in the CSB dataset. This indicates that evaluating the model’s progress in scene understanding requires the use of more challenging scene datasets that depict complex human behaviors and interactions.

In addition, although vision-language models have improved in their ability to describe scenes, there has been limited work examining which types of errors have decreased over time. Recent research has evaluated properties of these models, such as robustness, bias, fairness, and toxicity [23]. Other studies have analyzed how factors like training data, model architecture, and training or inference procedures influence model decision-making [24], have examined the shortcomings of VLMs in describing scenes using controlled image perturbations, revealing errors such as misclassification, omission, and incorrect quantity when new objects are inserted into an image [25], and identify factual errors in chart descriptions [26]. In this study, we introduce a new taxonomy of visual–cognitive errors to improve understanding of where models struggle with visual reasoning and to assess which errors have been overcome and which persist. The categories include errors in object detection, object recognition [27], hallucinations [28], scene understanding, and spatial dependence [29].

Building on this, we have developed the Complex Social Behavior (CSB) dataset, which comprises scenes featuring intricate social interactions and behaviors. Using this dataset, we assess how vision–language models have evolved over the past decade. We also classify the types of errors and examine how they have evolved over the past decade.

We compare the performance of nine vision language models developed in the last decade (2017-2025) relative to humans: Up-Down Captioner (UDC) [17], Meshed Memory Transformer(M2M) [18], OSCAR ([19]), OFA [20] are four pre-MLLMs, while GPT-4 [30], Gemini [31], GPT4-o1 [22], GPT-5.1, and Gemini3 [32] are five MLLMs. Some early models are no longer functional on current platforms, making our evaluation, which is the result of cumulative work beginning in 2019, unique in documenting progress in the evolution of MLLMs.

As a comparison with our CSB dataset, we conducted the same performance evaluation and error analysis on a separate set of images from the widely used MS-COCO dataset [3]. We evaluate the image descriptions of models and humans against a gold-standard description. Historically, for vision–language models, studies have used automatic metrics to evaluate model descriptions that use sentence structures [21], [33], [34] (BLEU, CIDER, ROUGE). More recently, improved metrics using language embeddings, such as BERTScore [35], have been developed to compare model and human descriptions. Here, we use Gemini/GPT Similarity Index (Gemini-SI/GPT-SI) based on the cosine similarity of description embeddings (similar to BERTScore), which we show correlates better with human ratings than BERTScore (Supplementary Figure 9).

2 Methods↩︎

2.1 Images↩︎

We used 200 images in this study. Of these, 100 were randomly selected from the MS-COCO dataset. The other 100 images were collected by ten research assistants (RAs). The RAs were instructed to select frames from movie scenes in which they could formulate questions that require reasoning about social interactions or complex human behaviors. Figure 2a shows examples of images in the CSB and MS-COCO datasets used in our study. The small sample of images reflects the time-consuming labor required to analyze each model’s error types (see section 2.5). We use resampling bootstrap statistics to quantify the variability of model accuracy due to the image sample size.

2.2 Models↩︎

We used nine different models. Five of them utilize state-of-the-art MLLM architectures (GPT4 [30], Gemini [31], GPT4-o1 [22], GPT-5.1, and Gemini3 [32]). These models employ transformer architectures and are trained on vast datasets of image-text pairs, incorporating human feedback through reinforcement learning techniques [36]. However, limited details about their internal architecture are publicly available.

The remaining four models are pre-MLLMs. Among them, three (M2M [18], OSCAR [19], and OFA [20]) use a ResNet-101 CNN [37] to extract image features. These features are processed by a Transformer encoder, which represents a joint vision-language space, followed by a Transformer decoder that generates text. The UDC [17] model also consists of a ResNet-101 CNN image backbone but uses an LSTM (a type of recurrent neural network [38]) for both encoding and decoding, rather than Transformers. The UDC model is explicitly trained for describing scenes and cannot perform other tasks.

M2M, OSCAR, and OFA are task-specific models, but their encoders are pre-trained on image-text pairs, enabling easy fine-tuning across tasks. Additionally, the OFA model includes language input, with text features processed by a language network (BERT). This allows for minimal fine-tuning across different tasks, as the text input provides additional context about the task at hand.

2.3 Human descriptions of scenes↩︎

2.3.1 Collection↩︎

We used Amazon Mechanical Turk to collect 20 descriptions for each image. Participants were instructed to describe the scene’s semantic content. Prior to beginning their scene description task, we provided example images and descriptions to ensure that participants understood the expected format.

2.3.2 Procedure to establish gold standard description↩︎

These descriptions were ranked by a separate group of 50 Mechanical Turk workers. In each trial, participants were shown one image along with five of its 20 descriptions. To ensure attentiveness, two descriptions from other images were randomly included as distractors. Participants were asked to select the description that best matched the image’s semantic content.

Each participant completed 800 trials (200 images × four description splits). This meant that each participant selected four descriptions per image. To prevent bias from description splits, each participant received a unique split of the 20 descriptions.

Descriptions were ranked by the number of votes received as the "best" description. The most frequently chosen description for each image was designated as the gold standard. To reduce comparisons while capturing the variability in human descriptions, we selected the top four (top-ranked) and bottom four (bottom-ranked) descriptions from the remaining 19 descriptions. These were used to compare models and human-generated descriptions against the gold standard description. The similarity comparisons were conducted using LLM-based metrics, as discussed in the results section below.

2.4 Input text for Multi-Modal models↩︎

Six (one Pre-MLLM (OFA) and all MLLMs) of the nine models require input text and an associated image, on which the model generates its output. OFA was finetuned to perform scene description using the input text "Describe Image". For the MLLMs, the following input text was used to prompt the models to generate descriptions of the given image.

Input Text Prompt: "Make your best guess of what might be happening in this scene in one sentence. Avoid mentioning objects that do not aid in understanding the context of the scene."

2.4.1 Gemini-SI/GPT-SI metrics to evaluate description similarity to gold standard description↩︎

Gemini-SI and GPT-SI are LLM-based similarity (LLM-SI) metrics that use language embeddings (3,072 vector size) produced by MLLMs such as Gemini and GPT [30], [31] to represent the semantic content of sentences, and compute cosine similarity between these embeddings to quantify semantic similarity.

2.4.2 Comparing Gemini-SI/GPT-SI metrics with human raters of scene descriptions↩︎

To evaluate LLM-based similarity (LLM-SI) metrics, we collected human ratings to compare model-generated descriptions (UDC, M2M, and OSCAR) against the gold standard.

16 Mechanical Turk workers participated in this experiment. In each trial, participants rated the similarity between three model-generated descriptions and the gold standard. These human ratings were then correlated with the LLM-SI (Gemini-SI/GPT-SI) metrics and BERTScore [35]. Supplementary Figure 9a presents the results of these comparisons. Figure 2b depicts the flowchart explaining the description ranking and rating procedure.

2.5 Classification of model errors↩︎

2.5.1 Pre-MLLMs↩︎

Since all pre-MLLMs rely on a ResNet-101 architecture to extract image features, detection and recognition errors were determined based on the ResNet-101 model’s bounding box predictions and object classifications.

Detection Errors: Objects of relevance that are not mentioned in the description and lack an associated bounding box are classified as detection errors.

Recognition Errors: Objects of relevance that are misclassified, or irrelevant objects that are misidentified and included in the description, are classified as recognition errors.

Scene Understanding Errors: Objects of relevance that are correctly recognized but omitted from the description are considered scene understanding errors.

Hallucinations: Objects that are not present in the scene or wrongly classified but are mentioned in the description of the model constitute hallucinations [28].

2.5.2 MLLMs↩︎

Since the architectures of MLLMs are not publicly disclosed, we identified images where model-generated descriptions deviated from human descriptions and then posed targeted questions to categorize the error types.

Detection Errors: If an object of relevance is missing from the description, we referenced nearby landmarks and asked whether the model could see an object near a specific landmark. If the model failed to detect it, this was categorized as a detection error. Example question: "Is there an object in the hands of the person standing next to a wall?"

Recognition Errors: We asked the model to name the missed object. If it misclassified the object, it was marked as a recognition error. Example question: "What is the name of the object in the hands of the person standing next to a wall?"

Scene Understanding Errors: If the model correctly recognized an object but still omitted it from its description, it was categorized as a scene understanding error.

Hallucinations: If the model described an object that was not present in the scene and was not simply a misclassification of another object, it was considered a hallucination. To verify whether an object was wrongly recognized, we analyzed the model’s description, identified potential locations based on landmarks, and asked questions similar to those used for detection and recognition errors.

2.5.3 Spatial dependence errors using masking paradigm:↩︎

The last error type measured the degree to which models and humans agreed on which image regions contributed to the description. If the model and humans relied on different parts of the images for the scene description, we classified this as a spatial dependence error.

To assess the dependence of the scene description on different image regions, we used a masking paradigm. Occlusion masks consisted of a black opaque spatial window with a soft edge, covering approximately one-third of each image. These masks varied in position. We created a total of 600 masked images (200 images × 3 masking positions).

Descriptions for these masked images were collected from 60 Mechanical Turk workers (20 per masking condition). Each worker saw only one masked version of each image to prevent memory effects from influencing their descriptions.

The masked descriptions were compared with the gold standard (unmasked) descriptions to assess which image regions were most important for human perception. For the models, we generated descriptions for each masked version and compared them to descriptions generated for the unmasked images. Figure 3 illustrates an example of this masking procedure used to analyze which image regions influence the scene descriptions for models and humans. Similarity evaluations were performed using the LLM-based metrics discussed above.

If the image regions most critical for descriptions differed significantly between models and humans, we categorized this as a Spatial Dependence Error (SDE). We quantified this by correlating similarity ratings between model-generated and human-generated descriptions for the three masked regions and comparing them to the average inter-human correlation for these regions. If the model-human correlation was two standard deviations below the inter-human correlation, it was classified as a spatial dependence error.

2.6 Statistical analysis↩︎

Given the small dataset (200 images), we employed bootstrapping to estimate the variability in our measures (scene description similarity to the gold standard and error rates) across images and thus their generalization to the population of images. We used a total of 10,000 image bootstrap resamples per analysis. Since model performance comparisons involved more than 15 statistical comparisons, we corrected significance levels for the False Discovery Rate (FDR) using the Benjamini-Hochberg method [39]. The adjusted p-values (q-values) were reported accordingly.

3 Results↩︎

We selected a dataset of 100 images depicting complex social interactions and behavior (CSB) and 100 images from the widely used MS-COCO dataset [3]. Figure 2a presents examples of images used in our study. We utilized nine vision-language models: four pre-MLLMs (UDC, OSCAR, M2M, OFA) and five MLLMs (GPT-4, Gemini, GPT-4o1, GPT5.1, and Gemini3). All pre-MLLMs evaluated contain an object detection component and employ a fully trained, faster R-CNN model. OSCAR, M2M, and OFA use variants of the Transformer architecture to encode a combined vision-language feature space, from which descriptions of a given input image are generated after fine-tuning. UDC, on the other hand, uses LSTMs for this purpose. The Multi-Modal Large Language Models (MLLMs) also rely on Transformer architectures, but their model details are not publicly available.

3.1 Evolution of vision-language models in providing human-level descriptions for scenes depicting complex social interaction↩︎

3.1.1 Gold standard descriptions↩︎

To obtain high-quality gold standard descriptions, we collected 20 human-generated descriptions for each image in our dataset. A separate group of 50 participants then ranked these descriptions based on how well they matched with each scene. The top-ranked description for each image was designated as the gold standard and used to compare with model-generated descriptions and other human descriptions. Among the remaining 19 human descriptions, the top four were classified as top-ranked, and the bottom four as bottom-ranked. Figure 2b(left) illustrates an example image along with human and model-generated descriptions.

3.1.2 Metric of semantic similarity for description comparisons↩︎

To compare model performance with human descriptions on the CSB and MS-COCO datasets, we measured the similarity between model-generated descriptions and the gold standard using the cosine similarity of text embeddings. These embeddings were obtained from LLMs such as Gemini (Gemini-Similarity Index, Gemini-SI) [31] and GPT (GPT-Similarity Index, GPT-SI) [30]. Text embeddings are numerical vectors representing the semantic meaning of a sentence in a high-dimensional space. In this space, sentences with similar meanings are represented by vectors that are closer to each other.

We validated the use of GPT-SI and Gemini-SI by comparing them with human ratings. To this end, we collected similarity ratings from 20 human raters for scene description pairs comparing three models (UDC, M2M, and OSCAR) to the gold standard. We found that the average correlation of GPT-SI and Gemini-SI with individual human raters was higher than the average correlation among human raters (bootstrap, p \(<\) 0.001) for all three models (see Supplementary Figure 9a). Figure 2b(right) illustrates the ratings provided by humans and LLM-SI (Gemini-SI and GPT-SI) for a model-generated description compared to the gold standard.

3.1.3 Human-model descriptions’ semantic similarity, 2017-2025↩︎

Figure 4a shows example images from the MS-COCO dataset and the CSB dataset, along with the description provided by the models developed in the last ten years. The example illustrates that, even in 2017, image caption models could describe many scenes in the MS-COCO dataset, and that the introduction of MLLMs has improved these descriptions. However, the more challenging CSB dataset shows significant improvements in visual common-sense reasoning capabilities with the introduction of MLLMs. Figure 4b shows the model performances as evaluated by Gemini-SI (left) and GPT-SI (right) in both datasets. On average, the pre-MLLMs perform significantly worse in the CSB images than the MS-COCO images (bootstrap, p \(<\) 0.0001), while the MLLMs perform similarly in both image sets. All the MLLMs’ performances are close to the top-ranked human performances in both datasets (dotted lines). The pre-MLLMs perform at the level of the bottom-ranked humans on the MS-COCO dataset and are significantly lower than the bottom-ranked humans (bootstrap, p \(<\) 0.0001) on the CSB dataset.

One possible explanation for the MLLMs’ high performance is that images in the CSB dataset, which consisted of movie scenes available online, may have been part of the MLLM’s vast training data. To discount such an explanation, we also tested all the MLLMs and one pre-MLLM (OFA) on an in-house dataset of 100 images (see Supplementary Figure 4c) not posted online (as of these models’ release), and thus unseen by the models during training. We found similar trends for the in-house images as for the movie scenes, with MLLM performance close to that of humans, while the OFA model performed significantly lower (bootstrap, p \(<\) 0.0001) than humans.

3.2 What kind of errors do models make?↩︎

To better understand the evolution of the models, we analyzed the types of errors each model made when generating descriptions. A typical vision-language model consists of two main components: an object detector and an encoder-decoder architecture, both of which can introduce errors in their descriptions. We categorized these errors based on concepts from vision science and cognitive psychology.

3.2.1 Detection errors↩︎

Detection errors occur when a model fails to detect relevant objects in an image. An object is considered relevant if it appears in the gold standard human description and is present in the scene. As illustrated in Figure 5a, humans identified a flashlight in their descriptions, making it a relevant object. However, the UDC (pre-MLLM) model failed to detect it, whereas the Gemini (MLLM) model correctly identified it. Pre-MLLMs made significantly more detection errors in the CSB dataset compared to the MS-COCO dataset (bootstrap, p \(<\) 0.001), whereas MLLMs detected objects accurately in most cases.

3.2.2 Recognition errors↩︎

Recognition errors occur when a model: 1) incorrectly classifies a relevant object, or (2) misidentifies an irrelevant object and incorrectly mentions it in the description. Figure 5b provides an example: the UDC (pre-MLLM) model detected a doorway but misidentified it as a mirror, whereas the Gemini model misidentified a lamp as a second gun. Pre-MLLMs exhibited significantly higher recognition errors in the CSB dataset than in the MS-COCO dataset (bootstrap, p \(<\) 0.001), with a decreasing trend in errors over time. All models performed well at recognizing objects in the MS-COCO dataset.

3.2.3 Scene understanding errors↩︎

Scene understanding errors occur when a model recognizes relevant objects but fails to mention them in its description. As shown in Figure 6a, the UDC model recognized people in the scene but did not mention them in its description, while the Gemini model recognized the hospital ward but failed to include it in its description. Pre-MLLMs had significantly higher scene understanding errors in the CSB dataset than in the MS-COCO dataset (bootstrap, p \(<\) 0.01), whereas MLLMs exhibited few such errors across both datasets.

3.2.4 Hallucinations↩︎

Hallucinations [28] occur when a model describes an object that is neither recognized by the model nor present in the image. These objects can be described as ‘imaginations’ of the model, which may occur when similar images in the model’s pretraining dataset contain the hallucinated object. Figure 6b shows an example where the UDC model incorrectly mentioned a frisbee. Pre-MLLMs made significantly more hallucination errors in the CSB dataset than in the MS-COCO dataset (bootstrap, p \(<\) 0.0001), whereas MLLMs had very few hallucinations.

3.2.5 Spatial dependence errors↩︎

Spatial dependence errors (SDE) occur when humans and models rely on different image regions to generate descriptions. To estimate this, we masked one-third of the image at a time. We collected descriptions from humans and the models after masking each image region and compared the similarity of these descriptions to the corresponding gold-standard descriptions for the models/humans (see methods). Figure 3 shows a detailed overview of the masking procedure. A lower similarity rating for a masked region indicates that masking that region significantly affects the generated description. When the critical regions used by humans and models do not align, an SDE is recorded. LLM-SIs (GPT-SI and Gemini-SI) are used to assess these similarities.

In the example shown in Figure 7a, heatmaps illustrate the regions important for description generation. Humans and the UDC (pre-MLLM) model rely on different regions, whereas the Gemini (MLLM) model relies on regions similar to those chosen by humans. Figure 7b demonstrates that SDE errors are considerably higher for all models in the CSB dataset. However, over the years, errors in this dataset show a downward trend across models. In contrast, SDE errors in the MS-COCO dataset remain relatively stable across all models.

3.2.6 Impact of model errors on similarity rating↩︎

To quantify how different error types affect the similarity between human and model descriptions, we computed the average Gemini-SI score across all model–image pairs and grouped the results by the error type present. In Figure 8, the dashed line indicates the baseline average similarity score of 0.71 for images with no identified errors. Detection, recognition, and hallucination errors produced the largest reductions in similarity, lowering the average Gemini-SI to 0.405, 0.411, and 0.381, respectively. In contrast, scene understanding errors had a significantly smaller impact on description similarity than detection, recognition, and hallucination errors (Gemini-SI = 0.467, bootstrap, p < 0.04), while spatial dependence errors showed the least impact overall (Gemini-SI = 0.560, bootstrap, p < 0.0001).

Overall, these findings indicate a significant difference in error rates between the MS-COCO and CSB datasets. In particular, pre-MLLMs on the MS-COCO dataset showed lower detection, recognition, and hallucination error rates than scene understanding error rates. For the CSB dataset, the pre-MLLMs’ detection, recognition, and hallucination error rates were closer to the scene understanding error rates and might be related to the critical role of smaller objects in the CSB dataset. In addition, the MLLMs have reduced rates across the board for all error types for both datasets. The only error type that remains significant for MLLMs is the spatial dependence error, suggesting that MLLMs can sometimes rely on different image regions than humans to describe scenes.

4 Discussion↩︎

Comparisons between humans and models span a wide range of tasks, including contour detection, recognition performance under visual distortions [40], visual reasoning with synthetic tasks [41], abstraction and reasoning [42], language representation [43], visual search in real-world scenes [44], aerial scene understanding [45], image captioning [46], [47], perception of gaze [48] and lesion detection in medical images [49], [50].

The present study assessed the evolution of vision-language models over the last decade and compared them with human capabilities for understanding scenes. We selected four pre-Multi-Modal Large Language Models (pre-MLLMs) (UDC [17], M2M [18], OSCAR [19], and OFA [20]) and five MLLMs (GPT4 [30], Gemini [31], GPT4-o1 [22], GPT-5.1, and Gemini3 [32]) that cover the range of state-of-the-art models over the last decade. Traditional datasets used in the computer science field [3]–[7] often contain images that are simple to understand, and all the models over the last ten years perform relatively well, even though it is evident that the current MLLMs are much superior to the other models. We created the Complex Social Behavior (CSB) dataset, depicting complex human behaviors and social interactions. Additionally, we selected 100 representative images from the MS-COCO dataset [3] and evaluated the models’ performance at describing images in both datasets.

Overall, our findings indicate that the assessment of how models have evolved over the last decade varies substantially depending on the image set: MS-COCO versus CSB. Assessment using the CSB images shows the significant improvements between pre-MLLMs and MLLMs for images depicting complex human behavior and social interactions. The assessment with MS-COCO images shows a more modest improvement. Furthermore, our findings highlight that MLLMs have narrowed the performance gap between the MS-COCO and CSB images. Pre-MLLMs exhibit a large performance gap between the two datasets. MLLMs’ performance is nearly identical across the simpler MS-COCO and CSB datasets. Comparison with humans on the CSB dataset also shows that the performance of pre-MLLMs is below that of the bottom-ranked humans. In contrast, MLLMs’ performance is close to the top-ranked humans.

Aside from performance assessment, it is important to understand across image sets to facilitate future model improvements and help users gain trust in model outputs. To this end, the field of explainable AI (XAI) [51] has introduced many approaches for understanding how models make decisions. Some methods examine how changes in certain features affect the model’s output, while others estimate how much each feature contributes to the final prediction [52]–[54]. There are also approaches that organize examples in a dataset into prototypes, which represent typical cases, and criticisms, which highlight where the model struggles [55]. Most of these approaches focus on explaining individual inputs and therefore do not provide an easy way to obtain statistical summaries of explanations across an entire dataset. Moreover, since the architectures of current MLLMs are not publicly available, it is especially important to develop techniques that do not rely on access to internal model components. To understand the shortcomings of these models, we developed a novel technique to classify model errors in image description generation. These errors included object detection errors, object recognition errors [27], hallucinations [28], scene understanding errors, and spatial dependence errors [29].

We found that for images depicting complex social behaviors, there is a significant reduction across all error types, and errors are almost negligible for current MLLMs. MLLMs have reduced errors for these CSB images to comparable levels to the MS-COCO, while pre-MLLMs show much larger error rates for all error types for the CSB images relative to MS-COCO. The one exception is the spatial dependence errors, which have reduced for the CSB dataset but remain a source of error for the MLLMs. This result suggests that humans and MLLMs can sometimes rely on different parts of the image to generate the scene description.

Another result of our analysis is that detection, recognition, and hallucination errors contribute the most to reducing semantic similarity to the gold standard. The one error type remaining in MLLMs (spatial dependence) contributes the least to reducing semantic similarity between the model description and the gold standard. This persistent spatial dependence error aligns with recent work by [56], who found that affordance-related tasks requiring spatial and agent-centered reasoning remained uniquely resistant to improvement across 18 VLMs, even with prompt engineering and newer model releases. Their findings suggest this reflects a deeper difficulty in reasoning about embodied agents within three-dimensional scenes from two-dimensional images.

A limitation of the current study is that we only evaluated 200 images. This might be considered a small sample compared to large dataset evaluations. Our main bottleneck is the analysis of error types, which requires manually probing the model for each image in the dataset. Detection, recognition, and scene understanding errors are all handled manually for pre-MLLMs. The MLLMs were asked a series of questions to determine the object detection, recognition, vision-language, and hallucination errors. Still, our analysis accounted for variability across images by using a bootstrap resampling method to assess statistical differences between the CSB and MS-COCO datasets and other effects.

Our findings should not be interpreted as implying that MLLMs have achieved human-level performance across all perceptual tasks. Others have revealed the remaining limitations of MLLMs. Tong et al. [57] introduced a framework using question answering to analyze the errors made by MLLMs that are induced by their visual components, while Zhang et al. [58] introduced a new dataset to study shortcomings of MLLMs in reasoning and cognitive understanding of sarcasm in image-associated text, a capability that requires more than understanding the visual details of a scene to achieve good performance. Tong et al. [57] showed that although these models perform well on reasoning tasks, they still lack robust visual grounding and struggle with fundamental perceptual patterns such as orientation, counting, and spatial relations. Similarly, Zhang et al. [58] demonstrated that most state-of-the-art MLLMs exhibit a significant gap in sarcasm understanding, suggesting that visual detection of elements is not equivalent to understanding the scene. Our technique, like those of Tong et al. [57] and Zhang et al. [58], is model-agnostic and operates solely on model inputs and outputs, enabling us to analyze how shortcomings in different model components contribute to errors in generated descriptions.

Overall, we have developed a novel technique to assess a vision-language model’s capacity by analyzing the sources of errors in the visual-cognitive processes it employs when providing descriptions. A unique aspect of the presented work is that it could apply the technique to models spanning a decade, even though many of the early models cannot be easily made functional in current hardware. These errors might account for the actual gap in human and model visual reasoning capabilities in the pre-MLLMs and also help us probe humans and current MLLMs in scenes that require more complex reasoning.

Declaration of Generative AI and AI-assisted technologies in the writing process↩︎

The author(s) wrote a draft of the paper. The 1st author used Grammarly and ChatGPT in order to improve the readability and grammar throughout the manuscript. Author(s) further edited the content for clarity and tone. Author(s) take(s) take full responsibility for the content of the published article.

Declaration of competing interests↩︎

The authors declare no competing interests.

Supplementary↩︎

References↩︎