Knowing You at First Glance: Inferring Apparent Personality from Faces
July 16, 2026
Abstract
Inferring apparent personality from facial images is important in social scenarios for embodied agents in human-robot interaction. Unlike inferring intrinsic personality traits via conversation, this task models first-impression personality perception based solely on facial appearance before interaction begins. Existing studies mainly focus on the Big Five personality model and often rely on language or multimodal inputs. As a result, it remains unclear whether facial cues alone can support meaningful associations with perceived personality traits. This question is particularly relevant for MBTI types, which are widely used in practice and more readily interpretable by large language models. To this end, we propose GlanceFace, an end-to-end framework for apparent personality inference leveraging vision-language models to introduce semantic priors and a semantic-enhanced facial representation module to capture subtle personality-related cues, together with an uncertainty-aware learning strategy to handle noisy and subjective annotations. Extensive experiments demonstrate strong performance on MBTI-based apparent personality benchmarks and reveal relationships between facial characteristics and perceived personality traits, highlighting its potential to support adaptive initial interaction strategies for embodied agents. The code and dataset are available at https://github.com/MrHuan3/GlanceFace.
1 Introduction↩︎
Suppose a social interactive robot encounters a user for the first time. Prior to interaction, it has no access to conversational signals and must rely on visual cues to form an initial impression and guide its behavior. This ability to perceive personality before interaction is crucial in real-world social scenarios, where early impressions influence subsequent interaction dynamics. For instance, an agent such as OpenClaw [1] can construct an initial user profile from this first impression to initialize its dialogue strategy. Despite recent progress in modeling personality-related traits [2]–[4], most existing works rely on multimodal inputs [5], [6], including audio, video, and text, which are typically available only after sufficient interaction. While effective, such approaches are less applicable in early-stage scenarios where only facial images are accessible. In contrast, MBTI personality types [7], widely used in practice and interpretable by large language models, remain underexplored in this prior-to-interaction setting.
This gap raises a fundamental question: can facial appearance alone provide sufficient cues to learn meaningful associations with perceived MBTI personality traits? As illustrated in Fig. 1, we study the practical yet underexplored setting of face-only apparent personality inference. This setting is particularly challenging due to subtle visual cues and subjective annotations. Specifically, the associations between facial characteristics and perceived personality traits are often weak, implicit, and fine-grained, making them difficult to model from static images alone. Meanwhile, personality labels may vary across annotators and contain uncertainty, introducing noisy supervision that degrades prediction accuracy. These challenges call for a model that can effectively exploit subtle semantic facial cues while explicitly accounting for annotation uncertainty.
To this end, we propose GlanceFace, an end-to-end framework for apparent personality inference from facial images. Our framework leverages vision-language models (VLMs) to introduce semantic priors and a semantic-enhanced facial representation module to capture subtle cues associated with perceived MBTI personality traits. To address subjectivity and noise in personality annotations, we further develop an uncertainty-aware learning strategy for reliable training. Extensive experiments demonstrate the effectiveness of GlanceFace on MBTI apparent personality prediction benchmarks, and further analyses provide insights into the associations between facial characteristics and perceived personality traits. Our contributions are summarized as follows:
We present GlanceFace, an end-to-end framework for apparent personality inference from facial images, and study the underexplored prior-to-interaction face-only setting for perceived MBTI personality prediction.
We develop a Semantic-Enhanced Facial Representation module that leverages VLM-derived semantic priors to capture subtle personality-relevant facial cues, along with an Uncertainty-Aware Personality Learning strategy to improve accuracy against subjective and noisy annotations.
Extensive experiments validate the effectiveness of GlanceFace on MBTI-based apparent personality prediction benchmarks, while additional analyses reveal associations between facial cues and perceived personality traits.
2 Related Work↩︎
2.1 Apparent Personality Inference↩︎
Apparent personality inference aims to model how personality traits are perceived from observable cues and has attracted increasing attention in computer vision [2], [6], [8], [9]. Most existing studies focus on the Big Five personality model and rely on multimodal inputs, such as facial appearance, voice, and text [3], [10]–[12]. While these approaches have achieved promising results, their reliance on multimodal signals limits their applicability in scenarios where only facial images are available [13]. In contrast to the widely studied Big Five traits, MBTI organizes personality preferences into four binary dimensions: Introversion-Extraversion (I/E), Sensing-Intuition (S/N), Feeling-Thinking (F/T), Judging-Perceiving (J/P), whose combinations define 16 personality types. This formulation provides a structured yet discrete description of personality preferences. Although MBTI-based personality prediction has been investigated in prior work, relatively few studies consider the face-only setting, and direct prediction from facial images remains underexplored [14]–[16]. Therefore, in this work, we focus on the underexplored yet practically important setting of face-only apparent personality inference for perceived MBTI traits.
2.2 VLMs for Facial Understanding↩︎
VLMs have shown strong capability in learning high-level semantic representations and have been widely applied to visual understanding tasks [17]–[19]. Compared with conventional facial features, VLM-based semantic priors are more effective at capturing subtle and abstract cues [20]–[22]. However, their potential for apparent personality inference from facial images remains largely underexplored. This problem is challenging due to weak and implicit associations between facial appearance and perceived personality traits, as well as the inherent subjectivity and noise in personality annotations. Motivated by these challenges, we leverage VLMs to enhance facial representations and incorporate uncertainty-aware learning to better handle noisy supervision during training.
3 Methods↩︎
Our framework, termed GlanceFace, is illustrated in Fig. 2. Given a face image, we first employ a modified ResNet backbone [23] to extract hierarchical facial features, while a VLM produces semantic embeddings as high-level priors. The semantic embeddings are progressively injected into the ResNet features through the proposed Semantic-Enhanced Facial Representation (SEFR) module, leading to enriched facial representations for apparent personality inference. Based on the final fused feature, we use a linear layer and a classification head to predict personality logits, together with an L2-normalized embedding branch for identity-preserving representation learning.
All face images are detected and aligned using MTCNN [24]. To preserve consistency across different images of the same identity, we follow margin-based face recognition methods [23], [25], [26] and encourage identity-consistent facial embeddings, which are optimized with the AdaFace loss [27]. Meanwhile, personality prediction is supervised by the proposed Uncertainty-Aware Personality Learning (UAPL), which leverages soft labels and vote-based confidence cues for reliable learning under noisy and subjective annotations. The detailed designs of SEFR, UAPL, and the gated FFN are shown in Fig. 2 (b)-(d).
3.1 Semantic-Enhanced Facial Representation↩︎
Recent VLMs [17]–[19] provide rich semantic priors for cross-modal understanding. Motivated by this capability, we enhance facial representation learning by incorporating VLM-derived semantic embeddings for apparent personality inference. Considering both effectiveness and efficiency, we adopt Qwen3-VL-Embedding-2B [28] to extract a semantic embedding from each face image. Compared with conventional face recognition features, these semantic embeddings capture higher-level cues beyond identity-related information, which may be informative for perceived personality understanding. Nevertheless, such semantic priors may also contain noise and uncertainty. To selectively filter and integrate them into facial representations, we propose the SEFR module.
Given a VLM embedding \(\mathbf{E}_{\textrm{V}} \in \mathbb{R}^{D_{\textrm{V}}}\), directly applying standard attention is not appropriate, not only because \(\mathbf{E}_{\textrm{V}}\) is a global one-dimensional embedding rather than a token sequence, but also because it may contain substantial noise and task-irrelevant semantic components. Inspired by the differential idea in [29], but tailored to the non-sequential and noisy nature of VLM embeddings in our setting, we design a lightweight differential gated attention mechanism based on element-wise interactions. Specifically, we first project \(\mathbf{E}_{\textrm{V}}\) into \[\mathbf{Q}_0, \mathbf{Q}_1, \mathbf{K}_0, \mathbf{K}_1, \mathbf{V} \in \mathbb{R}^{d},\] and compute the differential semantic response as
\[\label{eq:diffattn} \begin{align} \mathbf{DA} = \underbrace{\left(\mathrm{Softmax}\left(\frac{\mathbf{Q}_0 \odot \mathbf{K}_0}{\sqrt{d}}\right) - \mathrm{Softmax}\left(\frac{\mathbf{Q}_1 \odot \mathbf{K}_1}{\sqrt{d}}\right)\right)}_{\mathbf{A}_{\textrm{diff}}} \odot \mathbf{V}, \end{align}\tag{1}\]
where \(\odot\) denotes element-wise multiplication. Since the projected representations are one-dimensional vectors, we use element-wise query-key interactions for efficient semantic estimation. The differential response \(\mathbf{A}_{\textrm{diff}}\) suppresses shared or less informative activations while emphasizing more discriminative semantic cues, and the resulting feature \(\mathbf{DA}\) is further modulated by \(\mathbf{V}\).
Although the differential response alleviates part of the noise, the resulting semantic interactions may still be entangled. Inspired by gated feed-forward designs [30], we further refine \(\mathbf{DA}\) through a gated FFN:
\[\label{eq:gateffn} \begin{align} \mathbf{F}_{\textrm{mid}} &= \mathrm{LN}\!\left( \mathbf{W}_{\textrm{down}} \mathbf{DA} \odot \mathrm{GELU}(\mathbf{W}_{\textrm{gate}} \mathbf{DA}) \right), \\ \mathbf{F}_{\textrm{Qw}} &= \mathrm{LN}\!\left( \mathbf{W}_{\textrm{up}} \mathbf{F}_{\textrm{mid}} \right), \end{align}\tag{2}\]
where \(\mathrm{LN}\) denotes layer normalization, \(\mathrm{GELU}\) denotes the GELU activation function, and \(\mathbf{W}_{\textrm{down}}\), \(\mathbf{W}_{\textrm{gate}}\), and \(\mathbf{W}_{\textrm{up}}\) are learnable linear projection matrices. This gated refinement adaptively enhances informative semantic channels while suppressing residual noisy responses.
We insert one SEFR module after each ResNet block to progressively inject semantic priors into facial representations. Let \(\mathbf{F}_{\textrm{RN}}^{i} \in \mathbb{R}^{C \times H_i \times W_i}\) denote the feature map produced by the \(i\)-th ResNet block. We first apply global average pooling to obtain \(\mathbf{F}_{\textrm{P}}^{i} \in \mathbb{R}^{C}\), which is then concatenated with \(\mathbf{F}_{\textrm{Qw}}^{i}\) to form \(\mathbf{F}_{\textrm{cat}}^{i}\). Another gated FFN transforms \(\mathbf{F}_{\textrm{cat}}^{i}\) into a channel-wise modulation vector \(\mathbf{F}_{\textrm{g}}^{i}\), which is fused with the ResNet feature as
\[\label{eq:fuse} \begin{align} \mathbf{F}_{\textrm{fuse}}^{i} = \mathbf{F}_{\textrm{RN}}^{i} + \mathbf{F}_{\textrm{RN}}^{i} \odot \mathrm{Tanh}(\mathbf{F}_{\textrm{g}}^{i}), \end{align}\tag{3}\]
where \(\mathrm{Tanh}\) denotes the hyperbolic tangent activation function, and \(\mathbf{F}_{\textrm{g}}^{i}\) is broadcast along the spatial dimensions. The fused feature \(\mathbf{F}_{\textrm{fuse}}^{i}\) is then fed into the next ResNet block for further representation learning. Rather than directly concatenating VLM embeddings with facial features, SEFR performs noise-aware semantic refinement and progressively injects the refined semantic priors into hierarchical facial representations, leading to more stable facial representations.
3.2 Uncertainty-Aware Personality Learning↩︎
A key challenge in apparent personality prediction arises from the noisy and subjective nature of personality annotations. To address this issue, we propose UAPL, which leverages the soft personality distributions and vote statistics provided by the dataset to construct more reliable supervision.
For each sample, the dataset provides a soft distribution for each MBTI dimension, denoted by \(ST^{i}(t) \in \mathbb{R}^{2}\), where \(t \in \{\mathrm{I/E}, \mathrm{S/N}, \mathrm{F/T}, \mathrm{J/P}\}\) and \(i\) indexes the sample. Each sample is also associated with a vote count \(N_{\mathrm{vote}}^{i}\), which serves as an indicator of annotation reliability. Intuitively, samples receiving more votes tend to be more reliable. Accordingly, we define the vote-based reliability weight for the \(i\)-th sample as
\[\label{eq:vote} \begin{align} w_{\mathrm{vote}}^{i} = \frac{\log(1 + N_{\mathrm{vote}}^{i})}{\frac{1}{B}\sum_{j=1}^{B}\log(1 + N_{\mathrm{vote}}^{j}) + \epsilon}, \end{align}\tag{4}\]
where \(B\) denotes the batch size and \(\epsilon\) is a small constant for numerical stability. However, vote count alone is insufficient, since two samples with the same number of votes may still differ substantially in label certainty. For instance, a sample with a \(95{:}5\) vote distribution is more informative than one with a \(55{:}45\) distribution. To quantify this uncertainty, we further define an entropy-based confidence weight for the \(t\)-th dimension of the \(i\)-th sample:
\[\label{eq:entropy} \begin{align} w_{\mathrm{ent}}^{i}(t) = 1 - \frac{1}{\log 2} \left(-\sum_{c=1}^{2} ST_{c}^{i}(t)\log\!\big(ST_{c}^{i}(t)+\epsilon\big) \right). \end{align}\tag{5}\]
This term is the complement of normalized entropy, assigning larger weights to samples with sharper personality distributions. To further reduce the effect of ambiguous supervision, we introduce a confidence indicator:
\[\label{eq:th} \begin{align} s_{t}^{i} = \begin{cases} \max\!\big(ST_0^i(t), ST_1^i(t)\big), & \text{if } \max\!\big(ST_0^i(t), ST_1^i(t)\big) > \tau,\\ 0, & \text{otherwise}, \end{cases} \end{align}\tag{6}\]
where \(\tau=0.75\) is a predefined threshold based on the dataset. This indicator discards highly ambiguous samples and preserves only those with sufficiently confident personality annotations. Combining the above factors, we define the final uncertainty-aware weight as
\[\label{eq:finalweight} \begin{align} w^{i}(t) = s_{t}^{i}\, \big(w_{\mathrm{vote}}^{i}\big)^{\alpha} \big(w_{\mathrm{ent}}^{i}(t)\big)^{\beta}, \end{align}\tag{7}\]
where \(\alpha\) and \(\beta\) control the contributions of the vote-based reliability and entropy-based confidence, respectively. Let \(\mathbf{z}^{i}(t)\in\mathbb{R}^{2}\) denote the predicted logits for the \(t\)-th MBTI dimension of the \(i\)-th sample, and let
\[\label{eq:logit2prob} \begin{align} \mathbf{p}^{i}(t)=\mathrm{Softmax}\big(\mathbf{z}^{i}(t)\big) \end{align}\tag{8}\]
denote the corresponding predicted probability distribution. We then match \(\mathbf{p}^{i}(t)\) to the soft target \(ST^{i}(t)\) using the Jensen-Shannon divergence:
\[\label{eq:jsd} \begin{align} \mathrm{JS}\big(\mathbf{p}^{i}(t)\,\|\,ST^{i}(t)\big) = \frac{\mathrm{KL}\big(\mathbf{p}^{i}(t)\,\|\,\mathbf{M}^{i}(t)\big) + \mathrm{KL}\big(ST^{i}(t)\,\|\,\mathbf{M}^{i}(t)\big)}{2}, \end{align}\tag{9}\]
where
\[\begin{align} \mathbf{M}^{i}(t)=\frac{\mathbf{p}^{i}(t)+ST^{i}(t)}{2}. \end{align}\]
In addition, we encourage the predicted distribution to more closely match the soft target through a margin term:
\[\label{eq:margin} \begin{align} m^{i}(t)=\left\|\mathbf{p}^{i}(t)-ST^{i}(t)\right\|_{1}^{\gamma}, \end{align}\tag{10}\]
where \(\gamma\) controls the penalty strength. The overall UAPL loss is defined as
\[\label{eq:loss} \begin{align} \mathcal{L}_{\mathrm{UAPL}} = &\frac{\sum_{i=1}^{B}\sum_{t} w^{i}(t)\left[\mathrm{JS}\big(\mathbf{p}^{i}(t)\,\|\,ST^{i}(t)\big) + \lambda\, m^{i}(t)\right]}{\sum_{i=1}^{B}\sum_{t} w^{i}(t)+\epsilon} \\ &+ \lambda_{\mathrm{FR}}\mathcal{L}_{\mathrm{FR}}(\mathbf{e}, y_{\mathrm{FR}}), \end{align}\tag{11}\]
where \(\lambda\) denotes the margin coefficient, \(\lambda_{\mathrm{FR}}\) controls the face recognition regularization term, \(\mathcal{L}_{\mathrm{FR}}\) is the AdaFace loss [27], and \(\mathbf{e}\) is the facial embedding extracted by the ResNet backbone. In this way, UAPL places greater emphasis on reliable and confident personality annotations while reducing the influence of noisy or ambiguous supervision. As shown in Fig. 2 (c), UAPL integrates vote-based weighting, entropy-based confidence, and soft-target matching to provide more reliable supervision under noisy personality annotations.
3.3 Optimization Objective and Personality Prediction↩︎
For each of the four MBTI dimensions, we apply a Softmax function to the corresponding logit vector \(\mathbf{z}^{(t)}\) to obtain a binary probability distribution \(\mathbf{p}^{(t)} \in \mathbb{R}^2\) according to Eq. 8 . Following a factorized formulation over the four MBTI dimensions, the probability of the \(k\)-th MBTI type is approximated as the product of the corresponding marginal probabilities:
\[\label{eq:mbti} \mathbf{p}_k^{(\mathrm{mbti})} = \prod_{d=1}^{4} P\!\left(X_d = x_d^{(k)}\right), \quad k \in \{1, \dots, 16\},\tag{12}\]
where \(X_d\) denotes the random variable for the \(d\)-th MBTI dimension, and \(x_d^{(k)}\) denotes the corresponding trait of the \(k\)-th MBTI type. Each marginal probability \(P(X_d = x_d^{(k)})\) is obtained from the corresponding entry of \(\mathbf{p}^{(d)}\). Based on the resulting \(\mathbf{p}_k^{(\mathrm{mbti})}\), we rank all 16 MBTI types in descending order and report the Top-K predictions as the final output, where \(K \in \{1, 3, 5\}\).
| Model | Person-level | Image-level | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2-6 (lr)7-11 | Acc@1 | Acc@3 | Acc@5 | F1 | AUC | Acc@1 | Acc@3 | Acc@5 | F1 | AUC |
| IR-18 | 18.98 | 51.39 | 68.52 | 12.84 | 75.09 | 15.31 | 37.83 | 53.90 | 11.84 | 63.95 |
| IR-50 | 20.37 | 50.93 | 70.83 | 12.05 | 70.69 | 15.20 | 38.25 | 55.08 | 10.49 | 60.19 |
| MobileFace | 22.22 | 49.07 | 71.30 | 11.72 | 68.87 | 16.00 | 38.98 | 56.06 | 10.59 | 59.96 |
| ViT-S | 19.91 | 50.93 | 71.30 | 11.53 | 72.44 | 15.87 | 38.86 | 56.50 | 10.40 | 61.78 |
| CosFace | 18.98 | 50.46 | 64.81 | 9.86 | 67.55 | 14.80 | 36.60 | 53.24 | 9.61 | 60.26 |
| ArcFace | 20.83 | 47.69 | 68.52 | 13.11 | 73.06 | 14.36 | 36.35 | 52.95 | 10.56 | 62.26 |
| AdaFace | 21.30 | 52.31 | 67.13 | 14.28 | 75.78 | 15.53 | 37.58 | 54.02 | 11.12 | 64.31 |
| Ours | 26.39 | 56.48 | 75.00 | 17.99 | 78.19 | 18.84 | 43.90 | 62.54 | 12.60 | 68.91 |
5pt
4 Experiments↩︎
4.1 Datasets↩︎
The MBTI personality dataset consists of two parts: a facial image set and a personality annotation set. The facial images are collected from four large-scale face datasets, including MS1MV3 [31], CelebA [32], VGGFace2 [33], and IMDB-Face [34]. The personality annotations are obtained from the Personality Database [35]. The training split contains 3,092 identities with 568,675 images, while the evaluation split contains 216 identities with 39,788 images.
4.2 Experimental Settings↩︎
We use ResNet-18 [36], following the modification in [23], as the backbone network. All face images are cropped and aligned to \(112 \times 112\) using the standard FR preprocessing pipeline [27]. We employ Qwen3-VL-Embedding-2B [28] as the VLM to extract 2048-dimensional semantic embeddings. To enforce identity consistency, we adopt AdaFace [27] as the FR loss. The backbone is initialized with IR-18 weights pretrained on WebFace4M [37]. The model is trained for 10 epochs using SGD with an initial learning rate of 0.1 and a cosine decay schedule. All experiments are conducted on 8 NVIDIA V100 GPUs.
4.3 Evaluation Metrics↩︎
We evaluate performance at both the person level and the image level. For person-level evaluation, predictions are aggregated across all images of the same identity, while for image-level evaluation, each image is treated as an independent sample. For the 16-class MBTI classification task, we report Top-K accuracy (Acc@K) with \(K \in \{1, 3, 5\}\). We further report F1-score and AUC for both person-level and image-level predictions. For the four MBTI dimensions (i.e., I/E, S/N, F/T, and J/P), each formulated as a binary classification task, we report classification accuracy (Acc), F1-score, and AUC.
| Model | I-E | S-N | F-T | J-P | Avg. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 (lr)8-10 (lr)11-13 (lr)14-16 | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC |
| IR-18 | 63.43 | 61.84 | 70.94 | 74.07 | 83.43 | 69.78 | 69.44 | 67.00 | 71.38 | 68.98 | 79.26 | 70.02 | 68.98 | 72.88 | 70.53 |
| IR-50 | 61.11 | 58.42 | 69.11 | 75.46 | 85.24 | 70.23 | 68.06 | 63.87 | 72.61 | 69.44 | 78.43 | 71.74 | 68.52 | 71.49 | 70.92 |
| MobileFace | 64.81 | 60.82 | 67.98 | 76.39 | 85.71 | 70.50 | 66.20 | 58.29 | 73.70 | 69.91 | 80.00 | 70.10 | 69.33 | 71.21 | 70.57 |
| ViT-S | 63.43 | 58.64 | 66.97 | 75.93 | 85.31 | 72.83 | 67.59 | 59.77 | 73.05 | 70.37 | 80.25 | 65.45 | 69.33 | 70.99 | 69.58 |
| CosFace | 61.11 | 56.25 | 66.68 | 73.61 | 83.57 | 63.85 | 65.28 | 57.63 | 71.88 | 72.22 | 82.46 | 65.94 | 68.06 | 69.98 | 67.09 |
| ArcFace | 62.04 | 55.42 | 67.62 | 69.91 | 80.94 | 65.27 | 68.06 | 61.88 | 74.39 | 68.52 | 77.92 | 70.42 | 67.13 | 69.04 | 69.43 |
| AdaFace | 62.50 | 57.14 | 70.21 | 73.61 | 83.85 | 70.26 | 68.06 | 61.88 | 73.54 | 71.76 | 81.11 | 71.93 | 68.98 | 71.00 | 71.48 |
| Ours | 62.96 | 57.89 | 71.29 | 76.85 | 85.80 | 73.63 | 72.69 | 66.29 | 77.77 | 73.15 | 82.74 | 74.09 | 71.41 | 73.18 | 74.19 |
4pt
| Model | I-E | S-N | F-T | J-P | Avg. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 (lr)8-10 (lr)11-13 (lr)14-16 | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC | Acc | F1 | AUC |
| IR-18 | 57.03 | 58.63 | 60.38 | 64.03 | 74.59 | 63.40 | 63.38 | 58.39 | 67.39 | 62.25 | 71.11 | 61.94 | 61.67 | 65.68 | 63.28 |
| IR-50 | 57.01 | 58.34 | 60.07 | 66.88 | 77.96 | 60.19 | 62.62 | 56.72 | 64.92 | 62.25 | 70.66 | 61.87 | 62.19 | 65.92 | 61.76 |
| MobileFace | 56.78 | 56.73 | 58.97 | 67.32 | 78.34 | 60.19 | 64.15 | 55.41 | 66.69 | 62.63 | 71.98 | 60.80 | 62.72 | 65.62 | 61.66 |
| ViT-S | 57.61 | 57.66 | 60.39 | 67.56 | 78.39 | 62.11 | 63.85 | 56.75 | 66.57 | 63.81 | 73.33 | 60.12 | 63.21 | 66.53 | 62.30 |
| CosFace | 55.73 | 54.76 | 58.39 | 63.77 | 75.26 | 58.02 | 62.25 | 52.73 | 64.77 | 63.76 | 74.38 | 59.96 | 61.38 | 64.28 | 60.29 |
| ArcFace | 55.65 | 54.52 | 58.58 | 61.84 | 73.27 | 57.64 | 63.70 | 57.62 | 67.00 | 63.30 | 71.79 | 63.79 | 61.12 | 64.30 | 61.75 |
| AdaFace | 56.42 | 56.50 | 59.46 | 65.16 | 76.38 | 60.36 | 62.96 | 55.67 | 65.71 | 63.78 | 72.73 | 63.46 | 62.08 | 65.32 | 62.25 |
| Ours | 57.44 | 58.15 | 60.47 | 72.97 | 82.68 | 67.02 | 69.18 | 61.29 | 74.05 | 66.91 | 76.89 | 67.03 | 66.63 | 69.75 | 67.14 |
4pt
4.4 Performance↩︎
We compare GlanceFace with representative baselines, including general face backbones (e.g., ResNet [36], MobileFaceNet [38], and ViT [39]) and FR models (e.g., CosFace [25], ArcFace [23], and AdaFace [27]). All FR models use IR-18 as the backbone. As shown in Tab. 1, 2, and 3, GlanceFace outperforms the baselines from both holistic and fine-grained perspectives. For the 16-type MBTI classification task, it achieves the best results under both person-level and image-level protocols, with consistent improvements across Top-K accuracy, F1-score, and AUC. Since these metrics capture complementary aspects of prediction quality, namely ranking quality, class-balanced classification performance, and representation separability, the consistent gains indicate that GlanceFace improves overall personality prediction quality rather than optimizing for a single metric.
The dimension-wise results further reveal the source of the improvement. GlanceFace not only achieves the best average performance over the four MBTI dimensions, but also shows particularly clear advantages on more challenging dimensions such as S/N, F/T, and J/P, while remaining competitive on I/E. This pattern suggests that the proposed method is especially effective at capturing subtle and fine-grained facial cues related to apparent personality inference, rather than only benefiting easier cases. Another noteworthy observation is that the improvement is generally more pronounced under the person-level protocol, where multiple images of the same identity are aggregated. This indicates that GlanceFace learns more stable identity-level personality representations instead of relying on incidental variations in individual images. Moreover, the comparison between IR-18 and deeper backbones (e.g., IR-50) shows that increased backbone capacity does not consistently lead to better performance, suggesting that the gain mainly comes from the proposed personality modeling strategy rather than backbone strength alone.
4.5 Ablation Study↩︎
| Model | 16-MBTI | 4-dim Avg. | ||||
|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 | Acc@1 | F1 | AUC | Acc | F1 | AUC |
| IR-18 | 18.98 | 12.84 | 75.09 | 68.98 | 72.88 | 70.53 |
| w/ SEFR | 19.44 | 14.70 | 76.10 | 69.21 | 73.17 | 71.17 |
| w/ UAPL | 23.61 | 14.88 | 75.53 | 70.49 | 73.01 | 72.81 |
| Ours | 26.39 | 17.99 | 78.19 | 71.41 | 73.18 | 74.19 |
r0.6
4pt
We conduct ablation experiments to evaluate the contribution of each component in a principled manner. As shown in Tab. 4, both SEFR and UAPL consistently improve upon the IR-18 baseline. SEFR strengthens representation learning via semantic-guided feature recalibration, which emphasizes informative channels while suppressing noisy responses. As a result, it produces more discriminative representations, reflected by consistent gains in F1-score and AUC across both 16-MBTI and 4-dimension evaluations. UAPL, in contrast, addresses label uncertainty by adaptively weighting samples according to their reliability. From an optimization perspective, this reduces the influence of noisy supervision and stabilizes training, leading to notable improvements in accuracy and AUC.
When combined, SEFR and UAPL deliver further gains across all metrics. This indicates that the two components contribute from complementary perspectives: SEFR improves representation quality, whereas UAPL refines the supervision signal during optimization. Their combination therefore yields more reliable and discriminative personality prediction, confirming that both high-quality representations and reliable supervision are essential for this task.
4.6 Parameter Sensitivity↩︎
We further analyze the sensitivity of the four hyper-parameters in the UAPL loss, namely the vote-reliability exponent \(\alpha\), the entropy-reliability exponent \(\beta\), the margin exponent \(\gamma\), and the margin weight \(\lambda\). As shown in Fig. 3, performance is generally stable over a reasonable range of values, while moderate settings consistently produce the best results. Specifically, the optimal performance is achieved at \(\alpha=1.0\), \(\beta=1.0\), \(\gamma=1.5\), and \(\lambda=0.5\). This suggests that a balanced weighting strategy is most effective: using vote reliability and target entropy as adaptive sample weights is beneficial, but overly large \(\alpha\) or \(\beta\) over-emphasizes a small subset of samples and degrades performance. Similarly, a moderate margin setting is preferable. When \(\gamma\) or \(\lambda\) becomes too large, the margin term dominates the optimization and weakens the distribution-matching objective, resulting in inferior performance. Overall, these results indicate that the proposed loss is reasonably robust to hyper-parameter choices, highlighting the importance of balancing supervision reliability and distribution alignment.
4.7 Correlation Analysis↩︎
To better understand the cues learned for MBTI prediction, we analyze trait-specific facial response maps using MTCNN [24] and Grad-CAM++ [40]. We aggregate responses to obtain trait-level maps, contrast opposing traits to highlight discriminative regions, and compute pairwise correlations to examine their structure. The results show that most traits share common facial response patterns, while distinctive variations mainly arise between opposing traits within the same dimension, enabling more interpretable comparison of facial cues. This analysis reflects the model’s learned response patterns rather than any deterministic link to real personality traits.
We further examine the correlations among trait-specific facial response maps, as shown in Fig. 4. Overall, most trait pairs exhibit high correlations, typically above 0.90, suggesting that personality prediction is largely grounded in shared facial regions. Notably, opposing traits within the same dimension tend to show relatively lower correlations than cross-dimensional pairs, with the F-T pair exhibiting the lowest correlation at 0.84. In contrast, many cross-dimensional pairs remain highly correlated, such as I-J and E-N. These observations indicate that personality differences are not driven by distinct spatial regions, but rather by subtle shifts in activation patterns within a shared set of facial cues.
As shown in Fig. 5 and Fig. [fig:over], different personality traits exhibit distinct yet largely overlapping facial response patterns, with most activations concentrated on shared regions such as the eyes, nose, and mouth. This indicates that apparent personality inference relies on a common set of facial cues rather than entirely separate spatial regions. The contrastive visualizations further reveal that the differences between opposing traits are mainly reflected as shifts in activation emphasis within these shared areas, rather than distinct region-level separation. In particular, pairs such as F and T show more pronounced variations in activation distribution, suggesting stronger discriminative signals within this dimension. These results demonstrate that personality traits are characterized by structured variations over a shared facial-response basis, where opposing traits are distinguished by subtle but consistent differences in activation patterns.
Overall, these results provide an interpretable view of the model’s learned decision basis: it captures stable, trait-dependent facial response patterns with structured correlations across different traits. Importantly, these findings should be regarded as model-level interpretability evidence only. They reflect how the trained predictor organizes visual cues for the MBTI classification task, rather than implying any real-world causal or one-to-one relationship between personality traits and facial appearance.
5 Conclusion↩︎
We make an innovative attempt to study apparent MBTI personality inference from facial images in a first-impression setting prior to interaction, and present GlanceFace. To capture subtle personality-relevant cues, we design a Semantic-Enhanced Facial Representation module that leverages semantic priors from Qwen3-VL and progressively injects them into hierarchical facial representations through a differential gated fusion mechanism. To address the noisy and subjective nature of personality annotations, we further develop an Uncertainty-Aware Personality Learning strategy that emphasizes reliable samples and exploits soft-label supervision for improved learning effectiveness. Extensive experiments demonstrate the effectiveness of GlanceFace at both the person and image levels. Moreover, our analyses offer an interpretable view of the facial response patterns learned for apparent personality prediction. We hope this work will inspire future research on face-based apparent personality understanding.
6 Broader Impact↩︎
GlanceFace studies apparent personality inference from facial images and may support more adaptive initial interactions for embodied agents and human-computer interfaces. However, the predicted traits reflect perceived impressions rather than intrinsic personality and should not be used for high-stakes decisions. Potential risks include demographic bias, privacy concerns, and misuse in profiling or discrimination. Therefore, the model should be deployed with appropriate consent, fairness evaluation, and human oversight.
6.0.1 Acknowledgments.↩︎
This work was supported in part by Chinese National Natural Science Foundation Projects 92570119, 62276254, U23B2054, 62376265, 62502514, National Key R&D Program of China (No. 2025ZD0123501), the Science and Technology Development Fund of Macau Project 0140/2024/AGJ, and InnoHK program.