1 Introduction↩︎

Faces and humans are always the most crucial elements of photos and videos in our everyday lives. Consequently, they are also critical focuses in multi-modal AI applications. In the past two years, ChatGPT [1] and GPT-4 [2] have achieved great success with impressive instruction-following and multi-modal understanding capabilities respectively. Numerous excellent works [3]–[6] from the open-source community have followed, collectively presenting the immense potential of multi-modal assistants. Since faces and humans are central to social interaction, a deep understanding of this information can make multi-modal assistants achieve improved response quality and broadened application scope. For instance, in movie understanding [7]–[9], identifying characters is a prerequisite for multi-modal assistants to describe the plot accurately. In multi-modal human-computer interaction [10], perceiving expressions and body language can help multi-modal assistants accurately understand the context, generating more personalized and humanized responses. In media forensics [11]–[13], determining whether deepfake artifacts exist on a face is crucial for multi-modal assistants to detect misinformation.

Comprehensive and scientific evaluation is the foundation for researching applications of multi-modal assistants related to “faces and humans." Existing benchmarks [14]–[16] for large multi-modal models typically involve limited abilities of face and human understanding, such as celebrity recognition, action recognition, identity reasoning, and social relation, leaving many important abilities unexplored. On the other hand, since face and human understanding is one of the earliest research topics in artificial intelligence, there are numerous datasets available for evaluating the performance of specialist models. The images and annotations from these datasets can serve as original material to evaluate multi-modal assistants.

As the starting point of our evaluation, we propose a hierarchical ability taxonomy, as shown in 1. This taxonomy consists of three levels. Level-1 (L1) has two perspectives to study: from the target perspective, L1 includes face understanding and human understanding; from the cognitive process perspective, L1 includes perception and reasoning. Subsequently, we incorporate finer-grained abilities into the taxonomy and categorize them into 10 Level-2 (L2) and 18 Level-3 (L3) ability dimensions. Then, based on this taxonomy, we collect datasets from the face and human community and use a semi-automatic data pipeline to transform original images and annotations into multi-modal QAs. The final obtained benchmark called Face-Human-Bench, including a development set and a test set, each with 1800 problems, supporting evaluations in both English and Chinese. For ease of evaluation, we adopt multiple-choice as the problem format following MMBench [16] and SEED-Bench [15].

Our study aims to provide readers with the following insights: Q1: How do existing Multi-modal Large Language Models (MLLMs) perform in the face and human understanding? Specifically, we focus on (a) the performance of 25 mainstream MLLMs, (b) the correlation between abilities at different levels, (c) the impact of the relative position of targets on performance, and (d) the impact of Chain of Thought (CoT) [17] prompting on performance. Q2: In the field of face and human understanding, which tasks’ specialized models achieve significantly better performance than current MLLMs? With these insights, researchers in the multi-modal community can strategically strengthen the deficient abilities in their general-purpose models. Moreover, researchers in the face and human understanding community can use our evaluation results to select suitable MLLMs as initialization for downstream application scenarios. Knowing the specific abilities that MLLMs lack, researchers can also strategically leverage the outputs of specialized models to construct multi-modal agents [18], [19] that enhance responses.

In response to Q1, our main findings are as follows: (a) The Face-Human-Bench can effectively distinguish the abilities of MLLMs in faces and human understanding. Under the zero-shot setting, the best-performing closed-source model, GPT-4o [20], does not perform as well as the best open-source model, InternVL-Chat-v1.2-Plus [21]. (b) The correlation coefficients can reveal correlations between abilities at different levels. At L2 and L3, there are some ability groups in which the ability dimensions exhibit significant positive correlations between each pair. (c) Many models show substantial performance differences on the same task with different relative positions of targets. We use a new metric called the relative position sensitivity score (RPSS) to measure this phenomenon. On this metric, InternLM-XComposer2-VL-7B [22] performs the best, indicating that its performance is almost unaffected by the relative position of targets. (d) Introducing hints and CoT instructions into the prompts significantly improves the performance of the closed-source model GPT-4o, but has no effect on the open-source model, InternVL-Chat-v1.2-Plus. In response to Q2, we find that in tasks of deepfake detection, crowd counting, and face recognition (under challenging scenarios), the performance of MLLMs is significantly inferior to that of corresponding specialist models.

Our contributions can be summarized as follows:

• We propose the Face-Human-Bench, the first benchmark dedicated to evaluating multi-modal assistants’ face and human understanding abilities. It is based on a three-level ability taxonomy and supports both English and Chinese.

• Utilizing the Face-Human-Bench, we conduct a comprehensive evaluation of mainstream MLLMs, revealing the correlation between abilities, and exploring the impact of the relative position of targets and CoT prompting on the performance of MLLMs.

• We explore which face and human understanding tasks see specialist models significantly outperform MLLMs, providing recommendations for downstream application scenarios.

2 Face-Human-Bench↩︎

2.1 Hierarchical Ability Taxonomy↩︎

As shown in 1, the proposed ability taxonomy includes three levels. Level 1 (L1) has two research perspectives. From the target perspective, L1 includes face understanding and human understanding. From the cognitive process perspective, L1 includes perception and reasoning. In our evaluation, perception involves direct comprehension of only one target, while reasoning requires synthesizing information from multiple targets and environments to conclude. There are ten abilities in total at Level 2 (L2). Five are focused on faces: facial attribute recognition, age estimation, facial expression recognition, face attack detection, and face recognition, and five are focused on humans: human attribute recognition, action recognition, spatial relation understanding, social relation understanding, and person re-identification. It should be noted that at L2, there are 6 abilities under perception and 4 abilities under reasoning. Level 3 (L3) further refines the ability dimensions at L2. Facial expression recognition can be categorized into basic and compound types. Face attack detection includes deepfake detection and face anti-spoofing. Face recognition involves five scenarios: basic, cross-pose, cross-age, similar-looking, and occluded. Spatial relation understanding concerns relative position and count. Social relation understanding includes social relationship recognition and identity reasoning. Please refer to 6.1 for detailed definitions and examples of these abilities.

2.2 Semi-Automatic Data Pipeline↩︎

Based on the hierarchical ability taxonomy defined in 2.1, we collect 16 public datasets from the face and human community, covering each L3 ability. Then, we employ a semi-automatic data pipeline to produce problems for the Face-Human-Bench.

An original sample \(S_i\) from public datasets can be represented as a binary tuple \((I_i, L_i)\), where \(I_i\) denotes an original image set and \(L_i\) denotes an original label set. Note that we use “image set” and “label set” to describe the composition of one sample because, in some datasets, a single sample may consist of multiple images or labels. For instance, in face recognition, a sample includes a pair of face images to verify identity, and in facial attribute recognition, a sample may involve 40 attribute labels.

For ease of evaluation, we adopt multiple-choice as the problem format in our Face-Human-Bench. Each problem \(P_i\) corresponds to a quadruple \((V_i, Q_i, O_i, A_i)\). Here, \(V_i\) refers to the images obtained via the image processing pipeline \(p_{image}:\mathbb{I}\rightarrow\mathbb{V}\). \(p_{image}\) performs an operation such as cropping, concatenating, adding boxes, or leaving the original images unchanged, depending on the ability to test. \(Q_i\) denotes the question. Each L3 ability includes a set of pre-written questions that share the same semantics but exhibit diversity. When producing samples, a question \(Q_i\) is randomly selected from this question set. \(O_i\) is the set of \(n\) options \((o_1, o_2, ..., o_n)\), where \(2 \leq n \leq 4\). These options are obtained through the text processing pipeline \(p_{text}: \mathbb{L} \rightarrow \mathbb{O}\). \(p_{text}\) converts the original labels into one correct option and \(n-1\) incorrect options. For some tasks, ChatGPT [1] is used within \(p_{text}\) to assist in generating incorrect options or adjusting options at the sentence level (fixing grammar or re-wording sentences for fluency). \(A_i\) is the correct answer to the problem. The produced \(P_i\) will be checked by data reviewers to ensure that options are unambiguous and there is one and only one correct answer. Problems that do not meet the requirements will be removed.

In summary, our semi-automatic data pipeline leverages image and text processing pipelines, \(p_{image}\) and \(p_{text}\), to transform original samples into multiple-choice format problems. These problems are then manually checked to ensure quality. We obtain a benchmark with a development set of 1800 problems for the MLLM community to evaluate during training and a test set of 1800 problems for the formal evaluation in our paper. Additionally, the English problems are translated into Chinese to create a Chinese version of the benchmark. For more details on data sources, statistics, and the semi-automatic data pipeline, please refer to [asec:data-sources-and-statistics,asec:more-details-on-the-semi-automatic-data-pipeline].

3 Experiment↩︎

3.1 Experimental Setup↩︎

We use the weighted accuracy of multiple-choice problems as the evaluation score. As shown in 1, the proportion of the sectors represents the weight of the corresponding abilities in the overall score on the Face-Human-Bench. Note that we set equal weights for each L2 ability.³ To prevent models from favoring certain option letters over others, we shuffle the options to ensure the correct answers are evenly distributed across all option letters. During the testing, we add some constraint instructions to ensure MLLMs output only option letters as much as possible.⁴ After obtaining the MLLM’s response, we use regular expressions to extract the option letters. If this fails, we follow the implementation of MMBench [16] using ChatGPT [1] to extract the choices.⁵ We evaluate 25 MLLMs (as shown in 1) in different sizes from 13 model families. For more details on these models, please refer to 7.1.

3.2 Main Results↩︎

1 shows the performance of all evaluated MLLMs at different levels of abilities on the Human-Face-Bench (English)⁶ under the zero-shot setting. Overall scores range from 27.9% to 76.4%, demonstrating the effectiveness of the Face-Human-Bench in distinguishing the abilities of MLLMs in face and human understanding. We visualize the overall scores of MLLMs in 2. The findings can be summarized as follows: (1) The top 3 performing open-source models in terms of the overall score are InternvL-Chat-v1.2-Plus, LLaVA-Next-34B, and InternVL-Chat-v1.5. These models’ LLMs have the largest number of parameters among all open-source models we evaluate. (2) Generally, open-source models within the same series tend to show improved performance with increasing parameter scale. However, there are exceptions; for instance, the 13B version of LLaVA-1.5 and LLaVA-Next perform slightly worse than their 7B counterparts. (3) Under the zero-shot setting, the best closed-source model, GPT-4o, does not surpass the performance of the top-performing open-source models. We believe this is because GPT-4o does not fully realize its potential under the zero-shot setting. The experiments in 3.5 confirm our hypothesis. (4) Newer models show significant improvements compared to earlier models. Among MLLMs with 7B parameters within LLM, LLaVA-OneVision-7B performs best. Impressively, LLaVA-OneVision-0.5B, with only 0.5B parameters within LLM, outperforms the earlier InstructBLIP-13B.

L2 and L3 Performance⁷ (1) At L2 and L3, the best performance among open-source models is usually achieved by one of InternvL-Chat-v1.2-Plus, LLaVA-Next-34B, and InternVL-Chat-v1.5. Specifically, GLM-4V-9B achieves the best results in compound expression recognition (L3), facial expression recognition (L2), and action recognition (L2) and CogVLM2-19B-Chat achieves the best result in relative position understanding (L3). (2) At L2 and L3, the best performance among closed-source models is usually achieved by GPT-4o or GPT-4V. Notably, Gemini-1.5-Pro demonstrates outstanding face recognition ability (L2), achieving the best performance among all models with a score of 85.6%.

3.3 Correlation Between Abilities↩︎

In this section, we examine whether improving one ability in a model will enhance another by calculating the Pearson Correlation Coefficient between abilities at different levels, using the evaluation scores from 3.2.

At L1, the correlation coefficient of face and human understanding is 0.94 and the correlation coefficient of perception and reasoning is 0.79, both indicating significant positive correlations, as shown in 3 (a) and (b). We further investigate the correlations between L2 abilities, resulting in the correlation coefficient matrix shown in 3 (c). For clarity, we have drawn this as a lower triangular matrix. Our findings can be summarized as follows: (1) For the three face understanding abilities—facial attribute recognition, age estimation, and facial expression recognition—there are high positive correlations between each pair. (2) For the four human understanding abilities—human attribute recognition, action recognition, spatial relation understanding, and social relation understanding—there are high positive correlations between each pair. (3) For the three face understanding abilities and four human understanding abilities mentioned above, there are high positive correlations between each pair. (4) The two identity recognition tasks—face recognition and person re-identification—show a high positive correlation. (5) The correlation between face attack detection and any other ability is low. In 8.3, we further present the correlations between L3 abilities.

3.4 Relative Position of Targets↩︎

We investigate the impact of the relative position of targets on performance in four L3 abilities: facial attribute recognition, age estimation, basic expression recognition, and human attribute recognition. As shown in 4, for the three face understanding abilities, we provide both the original and cropped versions, where only one person is included but the relative position varies. For human attribute recognition, we offer box-added and cropped versions. In the box-added version, multiple people are included, with the target to be discussed indicated by a red box. 5 illustrates the performance differences between the two versions across various models. Our findings can be summarized as follows.

Face Understanding Abilities. (1) Preferences for either version depend on the model and the ability, with no overarching trend observed. (2) A model’s preference can vary across different face understanding abilities. For example, Yi-VL-6B shows no significant preference for facial attribute recognition, prefers the original images for age estimation, and favors cropped images for basic expression recognition. We think that this phenomenon may occur because MLLMs have been trained using images with different target relative positions when aligning visual information for different facial features.

Human Attribute Recognition. The majority of models perform better on the cropped version. This indicates that these models still struggle to accurately understand a specific individual when there are multiple people in the image.

We define the relative position sensitivity score (RPSS) as the sum of the absolute differences in scores between the two versions across the four tasks. This metric can serve as an effective reference for training MLLMs with more robust visual alignment for face and human understanding. We observe that InternLM-XComposer2-VL-7B, LLaVA-OneVision-7B, InternVL-Chat-v1.5, LLaVA-NeXT-34B, and InternVL-Chat-v1.2-Plus not only perform well in the four tasks but also exhibit low sensitivity scores. Among them, InternLM-XComposer2-VL-7B has the lowest sensitivity score of only 3.7%.⁸

3.5 CoT prompting↩︎

In this section, we select InternVL-Chat-v1.2-Plus and GPT-4o to explore whether incorporating hints and Chain-of-Thought (CoT) instructions in the prompts can enhance the MLLMs’ performance. These two models have achieved the best overall performance in the main experiment among open-source models and closed-source models respectively. A hint involves tips on how to answer the question. For example, the hint for person re-identification is “if two people have significant differences in posture and their faces are relatively blurry, the main basis for determining whether they are the same person is their clothing characteristics." CoT instructions, on the other hand, guide MLLMs in articulating the reasoning process that leads to the answer. The vanilla CoT instruction simply requires the model to”analyze the question and options step by step", whereas task-specific CoT instructions provide more tailored guidance based on the task. For example, for the deepfake detection task, the prompt might instruct the model to “analyze whether there are any artifacts in the facial image." Following Multi-modal CoT [17], we also conduct ablation experiments with both 1-stage and 2-stage frameworks. In the 1-stage framework, MLLMs are required to sequentially output the relevant analysis (rationale) and the answer in one round of dialogue. In the 2-stage framework, MLLMs first output the relevant analysis (rationale) in the first round and then provide the answer in the second round. Hints and task-specific CoT instructions for each L3 ability can be found in 7.2.3.

2 presents the performance of InternVL-Chat-v1.2-Plus and GPT-4o after incorporating hints and three different CoT settings. The results indicate that including hints and CoT instructions does not improve the performance of the open-source model; in fact, it may even cause a slight performance decline. By analyzing the outputs, we find that the open-source model does not provide rationales in its responses after adding CoT instructions to prompts. We believe this could be due to the model’s insufficient generalization capabilities, preventing it from understanding the CoT instructions. Specifically, the reasons may include the following aspects: (1) The scarcity of vision-language training samples aligned with the evaluation tasks of face and human understanding. (2) The insufficient exposure to samples of the complex reasoning paradigm during the training process. (3) The inadequate model capacity, which stems from substantially smaller parameter sizes compared to those of closed-source MLLMs. In contrast, the closed-source GPT-4o shows significant performance improvements. Adding hints leads to a 3.4% improvement compared to the zero-shot setting. Building upon this, vanilla CoT, 1-stage task-specific CoT, and 2-stage task-specific CoT further improve performance by 5.2%, 6.5%, and 5.7%, respectively. Ultimately, the combination of hints and 1-stage task-specific CoT instructions emerge as the best setting for overall performance.

In 6, we further explore the main reasons for the performance improvements of GPT-4o in each ability at L2. Hints significantly improve performance in face attack detection, face recognition, and person re-identification, while CoT instructions significantly improve performance in facial attribute recognition, face attack detection, human attribute recognition, and action recognition. For the reasons behind the performance improvements in each ability at L3, please refer to 8.5.

3.6 Specialist Models Significantly Outperforming MLLMs↩︎

In this section, we explore whether specialist models can significantly outperform MLLMs for the evaluation of 13 L3 abilities.⁹ We directly test the performance of MLLMs using original datasets from the face and human community to facilitate comparison with specialist models. We design a set of prompt templates to transform the classification problems into multiple-choice problems and the regression problems (age estimation and crowd counting) into fill-in-the-blank problems.¹⁰ Specialist models are generally trained and tested on data from the same distribution. They can achieve high performance even if the test labels contain noise. However, the visual information learned by MLLMs and the original datasets used for testing may exhibit data distribution bias. To enable an effective comparison, we utilize early specialist models (which emerged after the widespread adoption of deep learning) as a reference to judge the performance of MLLMs on these tasks.¹¹

We further define the relative performance score \(S\) to normalize performances across different tasks: \(S = (P_m - P_r)/(P_s - P_r)\), where \(P_m\) is the performance of the MLLM. Here, we take the highest-performing model among InternVL-Chat-v1.2-Plus, LLaVA-Next-34B , and InternVL-Chat-v1.5 (the top three models in the main experiment). \(P_r\) is the performance of random responses, and \(P_s\) is the performance of the early specialist model. This metric typically ranges from 0 to 1, where a higher relative score indicates stronger abilities of MLLMs on the corresponding task. A relative score below 0 stands for even worse results than random responses, whereas a score above 1 indicates the performance surpassing the corresponding specialist models for reference.

As shown in 3, MLLMs perform well in age estimation, facial expression recognition, face anti-spoofing, action recognition, and person re-identification, eliminating the need to introduce specialist models to enhance response quality. In contrast, for deepfake detection and crowd counting tasks, the MLLM significantly underperforms specialist models. Moreover, for face recognition, MLLMs can approach the specialist model under the basic scenario but indicate poor performance under more challenging scenarios, such as cross-pose, cross-age, similar-looking, and occluded. We recommend incorporating the corresponding specialist models into multi-modal assistants for applications where deepfake detection, crowd counting, and accurate face recognition are required. 11 provides a demonstration of how to enhance multi-modal assistant responses with specialist models.

4 Related Work↩︎

Evaluation of MLLMs for Face and Human Understanding. Currently, there is no dedicated benchmark evaluating the face and human understanding abilities of MLLMs. Some efforts aim at comprehensively benchmarking MLLMs, containing some ability dimensions about face and human understanding. LAMM [31] evaluates 9 different 2D vision tasks using 11 existing public datasets. Among these, the facial classification task utilizes the CelebA [32] dataset to evaluate the accuracy of smile detection and hair attribute classification. MME [14] includes the celebrity recognition ability, requiring MLLMs to respond with Yes/No answers. SEED-Bench [15] includes the action recognition ability, where the inputs consist of multiple frames taken from a video, and MLLMs are required to choose the correct answer from four descriptions. MMBench [16] includes the most extensive set of abilities related to faces and humans: celebrity recognition, action recognition, identity reasoning, and social relation, all of which are tested using multiple-choice problems. Considering the importance of faces and humans in multimedia, these evaluations are insufficient.

Face and Human Understanding. Face and human understanding is among the earliest research topics in artificial intelligence with successful applications. Numerous high-quality datasets were proposed for training and evaluating tasks of face attribute recognition [32], age estimation [33]–[35], facial expression recognition [36]–[38], deepfake detection [39], [40], face anti-spoofing [41], [42], face recognition [43]–[47], human attribute recognition [48], [49], human-object interaction detection [50], [51], crowd counting [52], social relationship recognition [53], [54] and person re-ideitification [55], [56]. Entering the 2020s, a new paradigm emerged, which initially pre-trains a task-agnostic backbone and then based on this, trains a unified face or human model [57]–[59] to simultaneously handle multiple face and human understanding tasks within a unified structure. In our evaluation, we observe that in certain tasks, MLLMs do not perform as well as specialist models. Utilizing these unified face or human models as specialist models to help MLLMs can greatly enhance responses.

5 Conclusion↩︎

In this work, we propose the hierarchical Face-Human-Bench, the first benchmark specifically designed to evaluate MLLMs’ face and human understanding abilities. We comprehensively and scientifically assess the performance of 25 mainstream MLLMs with our benchmark. We reveal the correlations between abilities and explore the impact of the relative position of targets and CoT prompting on the performance of MLLMs. Inspired by multimodal agents, we investigate which abilities of MLLMs need to be supplemented by specialist models. Our work will provide the face and human community valuable insights on how to more effectively leverage multi-modal assistants in applications related to “faces and humans."

Acknowledgement↩︎

This work was supported by the National Natural Science Foundation of China under Grants 62306043, 62076031, and 62076036.

6 More Details on Face-Human-Bench↩︎

6.1 Definition about Each Leaf Ability↩︎

We will sequentially describe the definitions of L2 abilities and the L3 abilities they encompass. We provide examples of problems in Face-Human-Bench in [atab:samples_01,atab:samples_02,atab:samples_03,atab:samples_04,atab:samples_05,atab:samples_06,atab:samples_07,atab:samples_08].

Facial Attribute Recognition: Recognize various characteristics and traits from facial images.

Age Estimation: Estimate the age of the person in the image based on facial information.

Facial Expression Recognition: Recognize the emotions of the person in the image, categorized into basic and compound types. Basic expressions include surprised, fearful, disgusted, happy, sad, angry, and neutral. Compound expressions provide more nuanced emotional descriptions, including: happily surprised, happily disgusted, sadly fearful, sadly angry, sadly surprised, sadly disgusted, fearfully angry, fearfully surprised, angrily surprised, angrily disgusted, and disgustedly surprised.

Face Attack Detection: Determine whether the face in the image involves digital manipulation or physical spoofing. The corresponding sub-abilities are referred to as Deepfake Detection and Face Anti-Spoofing, respectively.

Face Recognition Identify and verify individuals’ identities in images according to facial information. In our tests, this ability is mainly to determine whether two photos showcase the same individual. Five scenarios are involved: basic, cross-pose, cross-age, similar-looking, and occluded.

Human Attribute Recognition Recognize various characteristics and traits from human images.

Action Recognition Recognize human actions, including interactions with objects.

Spatial Relation Understanding Understand the spatial positions of people in the image, including relative position understanding (comprehending the relative positions of one person to others and objects) and crowd counting (counting the number of people in the image).

Social Relation Understanding Including social relationship recognition (inferring social relationships between people through their interactions) and identity reasoning (deducing social identity based on a person’s attributes, actions, interactions with others, and environmental information).

Person Re-Identification Identify and verify individuals’ identities in images based on full-body attributes (usually excluding the face, as facial features are often blurry).

6.2 Data Sources and Statistics↩︎

[atab:data-sources-statistics] provides information on the data sources for Face-Human-Bench, as well as the image processing pipeline, the number of problems in the development and test sets, and the weights, for each subset.

We set the weights of all 10 L2 abilities to be equal. For L2 abilities that encompass multiple L3 abilities, each L3 ability equally shares the weight of the corresponding L2 ability. For L3 abilities that encompass multiple image versions, each image version subset equally shares the weight of the corresponding L3 ability. Finally, we obtain the detailed weights of each subset, as shown in [atab:data-sources-statistics].

We sequentially provide overviews of the public datasets we used for original samples.

CelebA [32] is a large-scale facial attributes dataset released by the Multimedia Laboratory of Chinese University of Hong Kong. It contains over 200,000 celebrity images, each annotated with 40 attributes. The dataset includes a wide range of body pose variations and complex, diverse background information. It comprises 10,177 identities, 202,599 face images, and 5 landmark positions, with 40 binary attribute annotations for each image.

UTKFace [35] dataset is a large-scale facial dataset with a wide age range, spanning from 0 to 116 years. It contains over 20,000 face images, annotated with age, gender, and ethnicity labels.

RAF-DB [37] is a large-scale facial expression database consisting of 29,672 real-world images, each accompanied by a 7-dimensional expression distribution vector. It includes two different subsets: a single-label subset with 7 basic expressions (RAF-DB Basic) and a two-tab subset with 12 compound expressions (RAF-DB Compound). Additionally, the dataset provides 5 precise landmark locations, 37 automatic landmark positions, bounding boxes, and annotations for ethnicity, age range, and gender attributes for each image.

FF++ [39] consists of 1,000 original video sequences processed using four different automated facial manipulation methods: Deepfakes, Face2Face, FaceSwap, and NeuralTextures. The data in FaceForensics++ comes from 977 YouTube videos, all featuring trackable frontal faces without occlusions, allowing the automated manipulation methods to generate realistic forgeries.

SiW-Mv2 [60] collects 785 videos from 493 subjects, and 915 spoof videos from 600 subjects. The dataset includes 14 types of spoofing, ranging from typical print and replay attack, to various masks, impersonation makeup and physical material coverings. SiW-Mv2 exhibits a good variance in spoofing modes, with each mode specified and validated by the IARPA project.

LFW [61] is a commonly used test set for face recognition, comprising 13,233 face images sourced from natural scenes in everyday life. Each image is associated with a name, representing 5,749 individuals, with most people having only one image. The database randomly selected 6,000 pairs of faces to create face recognition image pairs to test the accuracy of face recognition systems, with 3,000 pairs containing two images of the same person and 3,000 pairs featuring one image of different individuals.

CPLFW [47] builds upon LFW by considering the impact of pose variations. It specifically searches for and selects 3,000 pairs of positive faces with differing poses, adding pose variation to the intra-class variance. Additionally, it includes negative pairs with the same gender and ethnicity to minimize the influence of attribute differences between positive and negative pairs.

CALFW [45] builds upon LFW by considering the impact of age variations. It specifically searches for and selects 3,000 pairs of positive faces with age differences to increase the intra-class variance associated with the aging process. Negative pairs are chosen to have the same gender and ethnicity to reduce the influence of attribute differences.

SLLFW [46] intentionally selects 3,000 pairs of visually similar faces through human crowdsourcing from the original image folder, replacing the random negative pairs in LFW.

MLFW [62] dataset is created based on CALFW and focuses on masked faces. The masks generated for the faces in the dataset maintain good visual consistency with the original faces. It includes a variety of mask templates that cover most common styles encountered in everyday life, achieving diversity of the samples.

WIDER Attribute [48] is a large-scale human attributes dataset containing 13,789 images across 30 scene categories, with 57,524 human bounding boxes. Each bounding box is annotated with 14 binary attributes, including male, long hair, sunglasses, hat, long shirt, long sleeves, formal, shorts, jeans, long pants, skirt, mask, logo, and checkered or striped patterns.

HICO-DET [51] is a commonly used dataset in the Human Object Interaction (HOI) domain, consisting of 47,776 images, with 38,118 in the training set and 9,658 in the testing set. The dataset includes 117 action (verb) categories, 80 object categories, and 600 verb-object combinations.

SpatialSense [63] is a dataset for spatial relation recognition, where the task is to determine whether a specific spatial relation holds between two given objects. The dataset contains 17,498 relations on 11,569 images, involving 3,679 unique object classes, with 2,139 of these classes appearing only once, presenting a challenging long-tail distribution.

PISC [54] is focused on the task of social relationship recognition in still images. It is used to benchmark models that analyze the relationships between people based on contextual and individual features. It contains 22,670 images with 76,568 annotated samples representing 9 types of social relationships.

ShTech [52] is focused on the task of crowd counting, where the goal is to accurately estimate the number of people in an image with varying crowd density and perspective. It contains 1,198 images with approximately 330,000 annotated heads. The dataset aims to address challenges in crowd counting that were not covered by previous datasets.

Market-1501 [56] is designed for the task of person re-identification. This dataset addresses the limitations of scale and realistic conditions found in previous datasets. The large-scale data supports training and testing models effectively for person re-identification. It includes over 32,000 annotated bounding boxes and a distractor set of more than 500,000 images.

6.3 More Details on the Semi-Automatic Data Pipeline↩︎

6.3.1 Details on Image Processing Pipeline↩︎

7 illustrates four operations of the image processing pipeline: cropping, concatenating, adding boxes, or leaving the original images unchanged. For simplicity, these four operations are denoted as Crop, Cat, AddBox, and Identity, respectively. The image processing pipeline used for each L3 ability is shown in [atab:data-sources-statistics].

6.3.2 Details on Text Processing Pipeline↩︎

We introduce the text processing pipeline for each L3 ability as follows. Facial Attribute Recognition Each option involves three attributes. At least two of the three attribute descriptions are incorrect in the incorrect options.

Age Estimation Add incorrect options at intervals of 5 years, 10 years, and 15 years, with each interval accounting for one-third of the total.

Basic Expression Recognition Incorrect options are randomly selected from the remaining 6 categories of expressions after removing the correct option.

Compound Expression Recognition Incorrect options are randomly selected from the remaining 10 categories of expressions after removing the correct option.

Deepfake Detection Set the options to “Yes" and”No". “Yes" indicates the presence of digital manipulations, while”No" indicates their absence.

Face Anti-Spoofing Set the options to “Yes" and”No". “Yes" indicates the presence of physical spoofs, while”No" indicates their absence.

Basic/Cross-Pose/Cross-Age/Similar-Looking/Occluded Face Recognition Set the options to “Yes" and”No". “Yes" indicates that the two photos are of the same person, while”No" indicates that the two photos are not of the same person.

Human Attribute Recognition Each option involves three attributes combined into a complete sentence using ChatGPT. At least two of the three attribute descriptions are incorrect in the incorrect options.

Action Recognition The incorrect options are actions generated by ChatGPT related to but not the same as the correct option.

Relative Position Understanding Each option is a sentence formed by connecting the subject and the object with a preposition. Incorrect options are generated by randomly selecting prepositions from the remaining 8 categories of relative positions after removing the correct preposition.

Crowd Counting The set includes three equally sized subsets, with the number of people in each subset being within the ranges of less than 10, 10-100, and more than 100, respectively. In the first subset, the incorrect options are also numbers within 10. In the latter two subsets, the incorrect options are numbers that are half, three times, and five times the correct option, respectively, with all options rounded to the nearest 10 and 100.

Social Relationship Recognition Incorrect options are randomly selected from the remaining 5 categories of social relations after removing the correct option.

Identity Reasoning The incorrect options are occupations generated by GPT related to but not the same as the correct option.

Person Re-Identification Set the options to “Yes" and”No". “Yes" indicates that the two photos are of the same person, while”No" indicates that the two photos are not of the same person.

6.3.3 Details on Data Checking↩︎

At the end of our data pipeline, the produced problems are checked by data reviewers. Each problem is read by three reviewers who are provided with an instruction as shown in 13. A problem is retained only if all three reviewers deem it acceptable.

7 More Details on Experiment Setup↩︎

All experiments for the open-source models were conducted on four NVIDIA A800 80G GPUs.

7.1 Overviews of Involved MLLMs↩︎

GPT-4V and GPT-4o: GPT-4V [2], released by OpenAI in September 2023, is a vision-enabled variant of the GPT-4 model, utilizing the same training process as GPT-4 for its visual capabilities. It is first trained on a large dataset of text and images, followed by fine-tuning through Reinforcement Learning with Human Feedback (RLHF). GPT-4V demonstrates the exceptional performance of a language-only system augmented with new modalities. The API we applied in our experiments is “gpt-4-turbo-2024-04-09”. GPT-4o [20] is released by OpenAI in May 2024. It accepts any combination of text, image, audio and video as input and generates any combination of text, image, and audio output. GPT-4o attains GPT-4 Turbo-level performance in text, inference, and code, while also demonstrating strong capabilities in multilingual, audio, and visual tasks. The API we applied in our experiments is “gpt-4o-2024-05-13”.

Gemini [64]: Gemini is a multimodal large model developed by Google, available in three scales: Ultra, Pro, and Nano. From its inception, Gemini was designed with a multimodal focus, excelling in tasks across image, audio, video, and text domains. In February 2024, Google released Gemini 1.5 [29], which includes Gemini 1.5 Pro and the more lightweight Gemini 1.5 Flash. In our work, we employ Gemini 1.5 Pro to conduct experiments.

Claude [65]: The Claude model is developed by Anthropic and is intended to be a useful, honest and harmless assistant. The version we applied in this paper, Claude 3.5 Sonnet [30], was released on June 2024. It is the most powerful visual model in the Claude series to date.

LLaVA [3]: LLaVA is an open-source large multimodal model that leverages multimodal language-image instruction-following data for instruction tuning. It was released in April 2023. LLaVA-1.5 [26], released in October 2023, introduced the following key improvements: the use of MLP as a vision-language connector, the use of prompt data with explicitly specified output formats, and the addition of task-specific datasets for training. Following that, LLaVA-1.6 (LLaVA-NeXT) [27] was released in January 2024, featuring improved input image resolution and enhanced visual reasoning and OCR capabilities. The model also supports better visual conversation on different scenarios and applications. SGLang was utilized for efficient deployment and inference. We apply LLaVA-13B, LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-NeXT-7B, LLaVA-NeXT-13B, and LLaVA-NeXT-34B in our experiments.

MiniGPT-4 [4]: MiniGPT-4, released in April 2023, uses a projection layer to align a frozen vision encoder with the frozen LLM Vicuna. The authors trained MiniGPT-4 in two stages: the first stage involved using a low-level dataset, and in the second stage, they curated a detailed image description dataset to fine-tune the model. In our experiments, we use MiniGPT-4-7B and MiniGPT-4-13B.

InstructBLIP [5]: InstructBLIP, released in May 2023, applies its instruction-tuning paradigm to the BLIP-2 [66] model. To be specific, InstructBLIP performs instruction fine-tuning on visual tasks to enhance model performance. In our experiments, InstructBLIP-7B and InstructBLIP-13B are used.

Qwen-VL [6]: Qwen-VL, released in August 2023, accepts images, text, and bounding boxes as inputs, and outputs text and bounding boxes. It supports multilingual and multi-image interleaved dialogue, as well as open-domain localization in Chinese. Qwen-VL is also capable of relatively fine-grained recognition and understanding. We adapt Qwen-VL-Chat in our experiments.

InternLM-XComposer2-VL [67]: InternLM-XComposer-VL, released in September 2023, is a multimodal large language model built with InternLM [68] as the language model. Later, in January 2024, InternLM-XComposer2-VL [67] was released, supporting free-form text and image composition. The authors proposed the Partial LoRA (PLoRA) method, which balances precise visual understanding with literary-inspired text generation. InternLM-XComposer2-VL-7B is used in our experiments.

Yi-VL [25]: Yi-VL, released in May 2024, excels in image-text understanding and chat generation, supporting multi-turn image-text conversations, bilingual text, and fine-grained image comprehension. Yi-VL adopts the LLaVA architecture and employs a three-stage training process to align visual information with the semantic space of Yi LLM [25].

InternVL [21]: InternVL, released in December 2023, extends its visual model to 6 billion parameters. It progressively aligns with the LLM using web-scale image-text data. InternVL-Chat-V1.2 was released in February 2024, expanding the LLM to 34 billion parameters. Shortly after, InternVL-Chat-v1.2-Plus was introduced, utilizing more supervised fine-tuning (SFT) data to further enhance its performance. Subsequently, InternVL-Chat-v1.5 [69] was released in April 2024, with improvements primarily focused on a stronger visual encoder, dynamic high-resolution capability, and a high-quality bilingual dataset. The model we use in the experiments includes InternVL-Chat-v1.2-Plus and InternVL-Chat-v1.5.

DeepSeek-VL [24]: DeepSeek-VL, released in March 2024, is designed for general multimodal understanding. It is built for real-world applications in visual and language comprehension, capable of handling tasks such as logical diagrams, web pages, formula recognition, scientific literature, natural images, etc. In the experiments, we apply DeepSeek-VL-1.3B and DeepSeek-VL-7B.

CogVLM2 and GLM-4V [28], [70]: CogVLM, released in October 2023, enables deep fusion of visual and language features without sacrificing performance on NLP tasks. In May 2024, the next generation, CogVLM2, was introduced. It inherited the visual expert architecture and improved training recipes in the pre-training and post-training stages, supporting high input resolutions. Shortly after, in June 2024, GLM-4V was released. It used the same data and training recipes as CogVLM2 but employed GLM-4-9B as the language models and removed the visual expert to reduce the model size. In our experiments, we utilize CogVLM2-19B-Chat and GLM-4V-9B.

LLaVA-OneVision [23]: LLaVA-OneVision, released in August 2024, supports three major computer vision scenarios: single image, multi-image, and video scenes. It also exhibits strong transfer learning capabilities across different modalities and scenarios. We use LLaVA-OneVision-0.5B and LLaVA-OneVision-7B in our experiments.

14 summarizes the LLMs and vision encoders used in involved MLLMs.

7.2 More Details on the Experiments for Q1↩︎

7.2.1 Prompt Templates for Different Settings↩︎

Zero-Shot (ZS) The prompt template used for the zero-shot setting is shown in 15.

Hints (H) The prompt template for experiments with hints is shown in 16.

Hints and Vanilla CoT Instructions (H+VCoT) The prompt template for experiments with hints and vanilla CoT instructions is shown in 17.

Hints and Task-Specific Instructions With One-Stage Framework (H+1TCoT) The prompt template for one-stage experiments with hints and task-specific CoT instructions is shown in 18.

Hints and Task-Specific Instructions With Two-Stage Framework (H+2TCoT) The prompt template for two-stage experiments with hints and task-specific CoT instructions is shown in 19.

7.2.2 Prompt Used for Choice Extraction↩︎

The prompt used for choice extraction is shown in 20.

7.2.3 Hints and Task-specific CoT Instructions↩︎

Hints and task-specific CoT instructions for each L3 ability are shown in 21.

7.3 More Details on the Experiments for Q2↩︎

7.3.1 Unexplored L3 Abilities↩︎

We explain the reasons for not conducting experiments on the remaining 5 L3 abilities as follows.

Face/Human Attribute Recognition These two tasks include a large number of binary classification labels (40 labels in CelebA for face and 14 labels in WIDER Attribute for human). Using evaluation protocols designed for specialist models to fully assess the performance of MLLMs would result in huge computational costs. Additionally, many attribute labels have ambiguous semantics that are difficult to define accurately, such as "attractive," "big lips," and "big nose."

Relative Position Understanding In the face and human understanding community, there are no specialist models specifically constructed to perceive the spatial relationships between one person to others and objects.

Identity Reasoning There is a lack of publicly available specialist models that perform occupation classification.

Social Relationship Recognition Existing evaluation datasets for specialist models, such as PIPA and PISC, are not suitable for directly evaluating the social relationship recognition abilities of MLLMs. Because many annotations in these datasets are ambiguous (it is often impossible to assert that people in an image belong to one relationship category rather than another) and semantically overlapping (for example, “couple" is a separate category, but a married couple is semantically also part of”family"). Fine-tuned specialist models can still learn to classify under unclear standards, but for zero-shot learning MLLMs, the lack of clear definitions greatly reduces performance. It is worth noting that in Face-Human-Bench, when we use PISC to construct problems for evaluating the social relationship recognition ability of MLLMs, we manually review and remove problems with ambiguous options or potentially non-unique answers, allowing us to assess accurately.

7.3.2 Explored L3 Abilities↩︎

We provide the prompt templates for directly evaluating L3 abilities on public datasets from the face and human community in [atab:prompt_rq2_01,atab:prompt_rq2_02,atab:prompt_rq2_03,atab:prompt_rq2_04,atab:prompt_rq2_05,atab:prompt_rq2_06,atab:prompt_rq2_07,atab:prompt_rq2_08,atab:prompt_rq2_09]. During testing, the options will be randomly shuffled.

8 Additional Results↩︎

8.1 Face-Human-Bench (English)↩︎

We provide the visualization of the L2 and L3 results in [afig:below10B,afig:above10B,afig:closed].

8.2 Face-Human-Bench (Chinese)↩︎

31 shows the performance of all evaluated MLLMs at different levels of abilities on the Human-Face-Bench (Chinese). We further compare the performance of different MLLMs on English and Chinese versions of the Face-Human-Bench, as shown in 11. Models are sorted with the ascending order of average performance.

8.3 Correlation Between Abilities↩︎

The correlation coefficient matrix for L3 is shown in 12. Pay particular attention to the ability correlations highlighted in the red boxes.

8.4 Relative Position of Targets↩︎

32 presents the performance differences of MLLMs across different relative positions of targets, under the three face understanding abilities and human attribute recognition.

8.5 CoT prompting↩︎

Based on 33, we explore the main reasons for the performance improvements of GPT-4o in each ability at L3, as shown in 13.

Abilities with performance improvements mainly due to hints include face anti-spoofing, cross-pose face recognition, cross-age face recognition, similar-looking face recognition, occluded face recognition, and person re-identification.

Abilities with performance improvements mainly due to vanilla CoT instructions include facial attribute recognition, deepfake detection, face anti-spoofing, occluded face recognition, human attribute recognition, action recognition, relative position understanding, and social relationship recognition. Comparison of outputs from H and H + VCoT settings is shown in [atab:output-compare-01,atab:output-compare-02,atab:output-compare-03,atab:output-compare-04].

Abilities with performance improvements mainly due to 1-stage task-specific CoT instructions include compound expression recognition and crowd Counting. Comparison of outputs from H + VCoT and H + 1TCoT settings is shown in 38.

Abilities with performance improvements mainly due to 2-stage task-specific CoT instructions include compound expression recognition and cross-pose face recognition. Comparison of outputs from H + 1TCoT and H + 2TCoT settings is shown in 39.

8.6 Specialist Models Significantly Outperforming MLLMs↩︎

We list the early specialist models used for comparison in 40.

8.7 Statistical Significance for Face-Human-Bench↩︎

The core contribution of this paper is the introduction of Face-Human-Bench, a benchmark designed to evaluate the performance of MLLMs. We aim to confirm that the proposed benchmark can significantly reflect performance differences across models.

Since the overall score is computed as a weighted average from several subset accuracies, it is crucial to validate the effectiveness of tests for each individual subset.

For each subset \(j\), we posit the null hypothesis \(H_0^{(j)}\): All models exhibit identical true performance on subset \(j\), with the observed variations attributed solely to measurement noise.

First, we calculate the unbiased sample variance across 25 models: \[S_j^2 = \frac{1}{24} \sum_{i=1}^{25} \left( \text{Acc}_{ij} - \bar{\text{Acc}}_j \right)^2,\] where \(\bar{\text{Acc}}_j\) represents the mean accuracy for subset \(j\). Next, we compute the expected variance under \(H_0^{(j)}\): \[\sigma_j^2 = \frac{\bar{\text{Acc}}_j (1 - \bar{\text{Acc}}_j)}{K_j}.\] Here, \(K_j\) is the size of subset \(j\). We derive the chi-squared statistic that quantifies the discrepancy between the observed and expected variances: \[\chi_j^2 = \frac{24 \cdot S_j^2}{\sigma_j^2}.\] Under \(H_0^{(j)}\), \(\chi_j^2\) follows a chi-squared distribution with 24 degrees of freedom. Across all subsets, we observe that \(\chi_j^2 \gg 24\) and the p-values are close to 0, indicating rejection of \(H_0^{(j)}\). This demonstrates that our benchmark can significantly reflect the differences in model performance.

9 Potential Bias for Demographic Characteristics↩︎

Do MLLMs contain potential biases? Specifically, do their performances vary based on the demographic characteristics of the input faces? Existing works, such as constructing the RFW [77] and BFW [78] datasets, have explored racial biases in face recognition systems. Inspired by these works, we investigate whether MLLMs exhibit different face recognition abilities across different racial groups.

We transform face pairs from the Caucasian, African, Asian, and Indian subsets of the RFW dataset into face recognition problems similar to those in Face-Human-Bench. The test results of the three best-performing open-source models in our main experiments are presented in 41, revealing the racial bias of MLLMs in face recognition ability. The performance of Caucasians is the best for each model, significantly surpassing that of other racial groups. In our future work, we will systematically evaluate the performance variations of MLLMs on samples with different demographic characteristics.

10 Societal Impacts↩︎

Our work aims to inspire the community to build multi-modal assistants with improved response quality and broadened application scope by providing a comprehensive and scientific evaluation of MLLMs’ face and human understanding abilities. This reflects the positive societal impacts of our research.

At the same time, we recognize that in some instances, personal privacy must be adequately protected. Therefore, it is important to proactively limit MLLMs’ ability to understand facial and bodily information to prevent the improper extraction of private information.

Our proposed Face-Human-Bench can also be used to evaluate privacy protection. In such scenarios, we want MLLMs to refuse to answer certain questions related to faces and humans. In such cases, lower performance on the Face-Human-Bench indicates a higher success rate in privacy protection on this information. 42 presents a comparison of the performance between APIs provided by OpenAI and Azure OpenAI. Note that Azure OpenAI primarily offers security and enterprise-grade services. GPT-4V and GPT-4o from Azure OpenAI show significant performance degradation in age estimation and expression recognition. Here are some example outputs:

• “I cannot determine the age of the person in the photo with the information provided."

• “I’m sorry, but the image is too blurry to make an accurate assessment of the person’s age."

• “I don’t have enough visual information from the image provided to accurately determine the emotion being expressed by the person."

• “I’m unable to determine the person’s expression due to the blurred face. Based on the available data, I cannot select a correct answer from the provided options."

From these outputs, it can be observed that Azure OpenAI might employ security strategies such as refusing to answer or blurring images.

11 A demonstration of How to Enhance Multi-Modal Assistant Responses with Specialist Models↩︎

In 14, we use media forensics as an application scenario to demonstrate how specialist models can improve the response quality of a multimodal assistant. Path 1 directly uses the MLLM to generate responses, while Path 2 introduces a well-trained specialist model for deepfake detection to determine whether there are digital artifacts on the faces in the image. By using the output of the specialist model to enhance the prompt, Path 2 ultimately allows the MLLM to provide more accurate responses.

12 Limitations↩︎

Despite the rich findings, there are still some limitations in this study. (1) This is the first work to comprehensively evaluate the face and human understanding abilities of MLLMs, mainly focusing on perception and simple reasoning. It does not involve tasks that require complex reasoning by integrating multiple face and human information. We plan to explore this in future work. (2) Considering the languages supported by existing mainstream MLLMs, Face-Human-Bench currently includes only English and Chinese. The capabilities of MLLMs in understanding face and human information in more languages remain to be further explored.

13 Ethics Statement↩︎

Our work does not involve reproducing, duplicating, copying, selling, trading, reselling, or exploiting any images from the original public datasets of the face and human community for any commercial purposes. Additionally, our work does not involve further copying, publishing, or distributing any portion of the images from the original public datasets. We fully comply with the agreements of all used original public datasets.

We will only open-source the JSON files containing our test problems and the data preprocessing scripts. You need to download all the original images from the involved public datasets yourself and organize the folders according to our instructions. The data preprocessing scripts will produce images for multi-modal QAs only during testing.

In our semi-automatic data pipeline, we provide adequate compensation to all participating data reviewers and ensure that this process complies with laws and ethical guidelines. Data reviewers only remove erroneous problems and thus do not involve the impact of regional or cultural differences among reviewers.

Face-Human-Bench is intended solely for academic and research purposes. Any commercial use or other misuse that deviates from this purpose is strictly prohibited. We urge all users to respect this provision to maintain the integrity and ethical use of this valuable resource.

References↩︎