1 Introduction↩︎

Facial expression recognition (FER) is a fundamental task in the computer vision community, as it holds significance in various applications, such as human-computer interaction and fatigue driving detection. Given its wide applicability, researchers have developed multiple FER models in recent years [1]–[3]. Despite the progress made so far, FER remains challenging due to noisy labels. Specifically, individuals may interpret one expression differently, resulting in inconsistent annotations. Besides, facial expressions have inherent inter-class similarity, which further exacerbates the problem of label noise. Accordingly, mitigating label noise has become one of the primary tasks in FER.

Mainstream noise-tolerant FER models can be divided into two groups: selecting clean samples and using label distributions as auxiliary training targets. For example, SCN [4] estimates the uncertainty of each sample and corrects the mislabeled samples during training. Moreover, DMUE [5] utilizes multiple learnable branches to estimate the label distribution of each sample. While these methods promote FER performance under label noise, they still encounter some problems. Firstly, sample selection methods assume that the neural network will fit clean samples before overfitting noisy labels [6], [7], which can lead to confusion between hard samples and noisy samples. Secondly, label distributions calculated based solely on expression information can still be noisy, since corrupt labels will disrupt both class and feature spaces [8].

To address the above problems, this paper proposes a new noise-tolerant FER model, called Landmark-Aware Net (LA-Net). The overview of LA-Net is plotted in 1. LA-Net leverages facial landmark information to combat label noise based on the underlying assumption that expressions with analogous landmark patterns probably belong to the same emotion category. The model comprises two key modules: label distribution estimation (LDE) and expression-landmark contrastive loss (EL Loss).

LDE calculates the label distribution of each sample and uses it as an auxiliary supervision signal. Based on the assumption that expressions should have similar emotions to their neighbors in feature space, LDE identifies neighbors in both expression and landmark spaces for each sample. The landmark information is utilized to correct the errors in the expression space. The module then learns pairwise contribution scores and performs neighborhood aggregation to obtain target label distributions. Furthermore, to mitigate the impact of batch division on online aggregation, the target label distributions are summed over previous epochs using exponential moving average (EMA).

EL Loss incorporates landmark information into expression representations to develop a robust backbone that is less susceptible to label noise. The algorithm treats landmarks and expressions as two views of facial images and establishes interactions between them via supervised contrastive learning (SCL) [9]. However, traditional SCL uses one-hot labels to select positive and negative pairs, thus performing poorly in the presence of label noise. Accordingly, our EL Loss designs a new pair selection strategy based on the label distributions to enable noise-tolerant SCL. Specifically, it first assigns pseudo-labels to confident images and takes the rest as unsupervised samples. The module then uses the expression and landmark features of the same images, or images with the identical pseudo-label, as positive pairs and all other combinations as negative pairs.

The proposed modules are solely used during training and thus incur no extra costs in deployment. Overall, our contributions can be summarized as follows:

(1) We present a landmark-aware FER model, named LA-Net, which leverages facial landmarks to alleviate the label noise issue.

(2) The LDE module uses landmark information to correct the errors in expression space and finds a set of neighbors to construct the label distribution of each sample.

(3) EL Loss devises noise-tolerant supervised contrastive learning and strengthens the expression feature extractor via expression-landmark interactions.

(4) LA-Net achieves state-of-the-art performance on both in-the-wild datasets and synthetic noisy datasets.

2 Related Work↩︎

2.1 Facial Expression Recognition↩︎

Researchers have proposed many algorithms to improve FER performance [1]–[3]. At the outset, handcrafted features including HOG [10] and SIFT [11] are applied to analyze expressions, whereas they perform poorly in the presence of strong illumination changes, large pose variations, and occlusions. Subsequently, learning-based methods advance the research mainly in two aspects: combating label noise and locating key areas. To address label ambiguity, SCN [4] evaluates the uncertainty of each sample and corrects mislabeled training samples on-the-fly. DMUE [5], on the other hand, applies multiple branches to calculate the label distribution of each sample. Regarding region-based models, RAN [12] uses self-attention to capture the importance of each facial area. TransFER [13] further introduces a dropout-like strategy to extract diverse key areas.

Concurrent with our work, LDLVA [14] also utilizes auxiliary facial information (valance-arousal) to alleviate label noise. Nonetheless, they use the extra information in a plug-and-play manner, while we take it as an auxiliary task and leverage supervised contrastive learning adapted to the label noise scenario to develop a robust feature extractor. Moreover, we combine the landmark and expression information to construct label distributions for training images and mitigate the impact of batch division by exponential moving average (EMA).

2.2 Learning with Noisy Labels↩︎

Label noise is a common issue in datasets, making learning with noisy labels a crucial topic in the community. Current research can be categorized into two groups: designing robust loss functions and selecting clean samples for training. Regarding the noise-tolerant loss functions, Zhang et al. [15] devise generalized cross-entropy loss to suppress label noise. Moreover, Patrini et al. [16] propose a loss correction approach for robust training. The sample selection strategy is based on the small-loss assumption that neural networks fit clean samples before overfitting noisy labels. Specifically, Han et al. [17] develop a co-teaching method that selects low-loss samples in two models to filter errors caused by noisy labels. Malach et al. [18] propose to train two models and perform updates only in case of disagreement between them, thereby keeping the effective number of noisy labels seen throughout the training process at a constant rate.

This paper aims to address the label noise in FER. The proposed LA-Net leverages facial landmarks to construct label distributions for training samples as well as strengthen the expression feature extractor via noise-tolerant supervised contrastive learning.

2.3 Contrastive Learning↩︎

Recently, Contrastive Learning (CL) [9], [19]–[23], which is based on the Siamese network [24], achieved great progress in unsupervised learning. Two prominent CL frameworks are SimCLR [22] and MoCo [20], [21]. SimCLR takes two random augmented views of the same image as positive pairs and different images as negative pairs, forming an instance discrimination task. Moreover, MoCo maintains a memory bank to increase negative samples and turns one branch of the Siamese network into a momentum encoder to improve the consistency of the memory bank. In addition to the self-supervised scenario, Khosla et al. [9] utilize the contrastive loss in the supervised setting (SCL) and achieve leading performance on multiple benchmarks.

Our EL Loss draws inspiration from supervised contrastive learning and implements interactions between landmarks and expressions to develop a more robust feature extractor. Moreover, we utilize label distribution as an alternative criterion for pair selection, which makes our method more resistant to label noise.

3 Methodology↩︎

We first introduce the notations that will be used in this paper. Let \(\boldsymbol{x}\) be the instance variable and \(\boldsymbol{x_i}\) be the \(i\)-th sample. We denote the class set as \(\mathcal{Y}=\{y_1, ..., y_C\}\), where \(C\) is the number of classes. Let \(\boldsymbol{l_i}=(l_i^{y_1}, ..., l_i^{y_C})\) with \(l_i^{y_j} \in \{0, 1\}\) and \(||\boldsymbol{l_i}||_1=1\) indicate the one-hot label of sample \(\boldsymbol{x_i}\). The label distribution of sample \(\boldsymbol{x_i}\) is denoted as \(\boldsymbol{d_i}=(d_i^{y_1}, ..., d_i^{y_C})\), where \(d_i^{y_j} \in [0, 1]\) and \(||\boldsymbol{d_i}||_1=1\). Let \(\boldsymbol{u_i}\) and \(\boldsymbol{v_i}\) be the expression and landmark features of sample \(\boldsymbol{x_i}\), respectively. The prediction of sample \(\boldsymbol{x_i}\) for the FER task is denoted as \(\boldsymbol{p_i}=(p_i^{y_1}, ..., p_i^{y_C})\), where \(p_i^{y_j} \in [0, 1]\) and \(||\boldsymbol{p_i}||_1=1\).

The overview of LA-Net is shown in 1. The model consists of three main parts: backbone, landmark distribution estimation (LDE), and expression-landmark contrastive loss (EL Loss). Specifically, LA-Net first uses two backbones to extract the expression and landmark features respectively. The landmark localization part employs a fully connected layer as the classifier and minimizes the mean square error, denoted as \(\mathcal{L}_{lm}\), during training. LDE identifies \(2K\) neighbors (\(K\) in expression space and \(K\) in landmark space) for each sample and performs neighborhood aggregation to generate target label distributions, which in turn improve the quality of training supervision. Besides, EL Loss considers the similarity between landmarks and expressions and incorporates landmark information into expression representations using noise-tolerant supervised contrastive learning. In the following sections, we will describe the LDE module and EL Loss in detail.

3.1 Label Distribution Estimation↩︎

Given the presence of noisy labels, label distribution is a better descriptor for facial expressions than the one-hot label. To construct the target label distribution of each sample, previous works hold that expressions should have similar emotions to their neighbors in the feature space or a supporting space [14], [25]. Nonetheless, extreme label noise will disrupt both class and feature spaces. Accordingly, the LDE module uses landmark information to correct the errors in the expression space based on the assumption that images with similar landmark patterns should be assigned to the same emotion class.

Given the mini-batch \(\mathcal{D}_{batch}=\{(\boldsymbol{x_i}, \boldsymbol{l_i})|i=1,2...n\}\), LDE calculates the label distributions in four steps. Firstly, for image \(\boldsymbol{x_i}\), LDE adopts the \(K\)-Nearest Neighbor algorithm to identify its neighbors in the expression space, denoted as \(N^u(i)\), based on the cosine similarity. \[\begin{align} s_{i, j}^u &= \frac{\boldsymbol{u_i}\boldsymbol{u_j}^T}{||\boldsymbol{u_i}||||\boldsymbol{u_j}||}\\ N^u(i) &= KNN(\mathcal{D}_{batch};K;s_i^u) \end{align}\] where \(s_{i, j}^u\) denotes the similarity between \(\boldsymbol{x_i}\) and \(\boldsymbol{x_j}\) in the expression space. The neighbor set in the landmark space, denoted as \(N^v(i)\), can be generated in an identical way. Then, LDE evaluates the pairwise contribution scores in the expression space as follows: \[\begin{align} c_{i, j}^u = \text{Sigmoid}(f([g_1(\boldsymbol{u_i};\theta_1), g_2(\boldsymbol{u_j};\theta_2)];\theta)) \end{align}\] where \(c_{i,j}^u\) denotes the contribution of \(\boldsymbol{x_j}\) to \(\boldsymbol{x_i}\) in the expression space, \([]\) denotes concatenation, \(f\), \(g_1\), and \(g_2\) are three MLPs with learnable parameters \(\theta\), \(\theta_1\), and \(\theta_2\). The contribution scores in the landmark space, denoted as \(c_{i, j}^v\), can be obtained in the same way. Based on the scores, the module performs neighborhood aggregation in each space and produces the target label distributions by mean pooling. \[\begin{align} \boldsymbol{d_i} = \frac{1}{2}(\frac{\sum_{k \in N^u(i)} c^u_{i, k} \;\boldsymbol{p_k}}{\sum_{k \in N^u(i)} c^u_{i, k}} + \frac{\sum_{k \in N^v(i)} c^v_{i, k} \;\boldsymbol{p_k}}{\sum_{k \in N^v(i)} c^v_{i, k}}) \end{align}\] As a result, the label distributions incorporate both expression and landmark information and thus are less susceptible to noisy labels. However, the above online aggregation approach can be affected by batch division, resulting in noisy and erratic label distributions in some cases. In particular, one batch may contain an excessive number of corrupt labels, causing the neighborhood of each sample to be extremely noisy. To address this issue, LDE sums up the targets over previous epochs using exponential moving average (EMA). This allows us to compute the target label distribution of image \(\boldsymbol{x_i}\) in the \(e\)-th epoch as follows: \[\begin{align} d_i^{[e]} = \omega \;d_i^{[e-1]} + (1 - \omega) \;\boldsymbol{d_i} \end{align}\] where \(\omega\) denotes the decay of previous targets. LA-Net leverages both one-hot labels and label distributions as supervision and minimizes the following loss functions: \[\begin{align} \mathcal{L}_{ce} &= -\frac{1}{n} \sum_{i=1}^n \sum_{j=1}^C l_i^{y_j} \text{log}(p_i^{y_j})\\ \mathcal{L}_{kl} &= -\frac{1}{n} \sum_{i=1}^n \sum_{j=1}^C d_i^{[e], y_j} \text{log} \frac{d_i^{[e],y_j}}{p_i^{y_j}} \end{align}\]

3.2 Expression-Landmark Contrastive Loss↩︎

The LDE module estimates label distributions for training samples to enhance resistance to corrupt labels. Besides, a knowledgeable feature extractor can also help mitigate the label noise. As such, LA-Net leverages facial landmarks to strengthen the expression feature extractor. Specifically, we create expression-landmark pairs and utilize supervised contrastive learning (SCL) [9], denoted as expression-landmark contrastive loss (EL Loss), to implement interactions between these two facial modalities. However, traditional SCL constructs positive and negative pairs based on the one-hot labels in datasets, thus performing poorly in the presence of label noise. To address this problem, EL Loss uses label distributions as an alternative criterion for pair selection, enabling noise-tolerant supervised contrastive learning. We describe the pair selection process for an anchor expression feature vector in 1, and provide further details below.

Given sample \(\boldsymbol{x_i}\), EL Loss first derives its pseudo-label \(\hat{l_i}\) from the target label distribution in the current epoch: \[\begin{align} \hat{l_i} &= \begin{cases} argmax(d_i^{[e]}) & \text{if } max(d_i^{[e]} > \delta) \\ -1 & \text{otherwise} \end{cases} \end{align}\] where \(\delta\) denotes the confidence threshold for separating confident and ambiguous samples. We then treat the expression and landmark features of confident samples with the same pseudo-label as positive pairs. Moreover, to suppress the uncertainty of ambiguous samples, only expression and landmark features of the same ambiguous image are utilized as positive pairs. In other words, the indication set \(\mathcal{I}(i)\) for the positive pairs of sample \(\boldsymbol{x_i}\) is computed by: \[\begin{align} \mathcal{I}(i) &= \begin{cases} \{j \;|\;\hat{l_j}=\hat{l_i}\} & \text{if } \hat{l_i}\neq -1 \\ \{i\} & \text{otherwise} \end{cases} \end{align}\] In addition, we take the remaining sample combinations as negative pairs and denote them as \(\Bar{\mathcal{I}}\). Following traditional contrastive learning [20]–[22], the module then performs non-linear projection to obtain query expression features. \[\begin{align} q_i^u = g(\boldsymbol{u_i};\theta) \end{align}\] where \(g\) is a two-layer perceptron with learnable parameters \(\theta\). Query landmark features \(q_i^v\), key expression features \(k_i^u\), and key landmark features \(k_i^v\) are calculated similarly. Finally, EL Loss implements interactions between expressions and landmarks by pulling positive pairs closer and pushing negative pairs farther apart. \[\begin{align} \mathcal{L}_1 &= \frac{1}{n} \sum_{i=1}^n \frac{-1}{|\mathcal{I}(i)|} \sum_{j \in \mathcal{I}(i)} \text{log} \frac{\text{exp}( q_i^u \boldsymbol{\cdot} k_j^v / \tau)}{\sum\limits_{m \in \Bar{\mathcal{I}}(i) \cup \{j\}} \text{exp}(q_i^u \boldsymbol{\cdot} k_m^v / \tau)} \\ \mathcal{L}_2 &= \frac{1}{n} \sum_{i=1}^n \frac{-1}{|\mathcal{I}(i)|} \sum_{j \in \mathcal{I}(i)} \text{log} \frac{\text{exp}(q_i^v \boldsymbol{\cdot} k_j^u / \tau)}{\sum\limits_{m \in \Bar{\mathcal{I}}(i) \cup \{j\}} \text{exp}(q_i^v \boldsymbol{\cdot} k_m^u / \tau)} \\ \mathcal{L}_{el} &= \mathcal{L}_1 + \mathcal{L}_2 \end{align}\] where \(a \boldsymbol{\cdot} b\) denotes the cosine similarity between \(a\) and \(b\), \(\tau\) indicates a temperature parameter. Moreover, we follow the MoCo framework [20], [21] and use memory banks to increase negative pairs.

3.3 Overall Loss↩︎

LA-Net minimizes the following loss function during training: \[\begin{align} \mathcal{L} = \mathcal{L}_{ce} + \mathcal{L}_{lm} + \alpha \mathcal{L}_{kl} + \beta \mathcal{L}_{el} \end{align}\] where \(\alpha\) indicates the importance of label distributions and \(\beta\) denotes the contribution of EL Loss.

4 Experiments↩︎

4.1 Datasets↩︎

RAF-DB [26] consists of 30,000 images with basic or compound labels. Following previous works, we only use the images annotated with six basic expressions (anger, disgust, fear, happiness, sadness, surprise) and neutral, where 12,271 are used for training and 3,068 for testing.

FERPlus [27] is an improved version of FER2013 [28]. It contains 28,709/3,589/3,589 training/validation/testing images. Each image is labeled by ten experts and assigned to one of eight classes (six basic expressions, neutral, and contempt). We use the class with the most votes as the label.

AffectNet [29] is the largest and most challenging FER dataset. It contains more than 280K training images and 4,000 testing images, which are classified into the same eight classes as FERPlus. Previous works vary in the use of AffectNet (with or without contempt), and we utilize 8-class AffectNet by default in the experiments.

Following previous works, we report the overall accuracy of the testing set for all datasets.

4.2 Implementation Details↩︎

We perform experiments on 4 NVIDIA RTX 3080Ti GPUs and use ResNet-18 [30] pretrained on MS-Celeb-1M [31] as the backbone. Regarding data processing, we resize all images to \(224 \times 224\) pixels and use HRNet [32], [33] to generate ground truth landmarks. To alleviate class imbalance, we adopt progressively balanced sampling [34]. Besides, on-the-fly data augmentation techniques, including random cropping, random horizontal flipping, random erasing, and random color jitter, are employed to enhance the generalization performance of LA-Net. During training, we set the batch size to 128 and use the Adam optimizer [35] with an initial learning rate of 1e-3. For all datasets, we train the model for 80 epochs and linearly decrease the learning rate to 0. Moreover, we conduct grid searches to determine all parameters. Specifically, we set the number of neighbors \(K\) to 8, the decay of targets \(\omega\) to 0.9, the temperature \(\tau\) to 0.1, and the confidence threshold \(\delta\) to 0.7. Besides, weights \(\alpha\) and \(\beta\) in the overall objective are set to 1.0 and 0.1.

4.3 Experiments on Synthetic Noisy Datasets↩︎

Noisy labels, caused by ambiguous facial expressions, significantly harm FER performance in real-world scenarios. To address this issue, LA-Net leverages landmark information to estimate label distributions for training samples and strengthen the expression feature extractor. We compare LA-Net with previous noise-tolerant FER models on three synthetic noisy datasets with varying levels of noise (10%, 20%, and 30%). To inject synthetic noise, we follow prior studies [4], [36], [37] and randomly flip the one-hot label to other categories. Additionally, we repeat the experiments five times due to the randomness of label noise injection and report the mean and standard deviation of overall accuracy.

¿tbl:tab:synthetic-noise? shows that LA-Net consistently outperforms other noise-tolerant FER algorithms in all settings. Compared to the current state-of-the-art model EAC [37], LA-Net achieves an average improvement of 0.91%, 0.82%, and 1.70% on RAF-DB, FERPlus, and AffectNet, respectively. Additionally, LA-Net becomes more preferred as the noise ratio increases. Specifically, compared to the baseline, the model promotes the performance by 5.61% and 8.66% on AffectNet with 10% and 30% noise, respectively. In conclusion, the experimental results in ¿tbl:tab:synthetic-noise? demonstrate that the proposed approach is effective in mitigating label noise.

In addition to the aforementioned symmetric noise, we further test the more challenging asymmetric noise, where the label is flipped to its most similar class based on the confusion matrix. As shown in Appendix A, LA-Net consistently outperforms previous models when dealing with asymmetric noise, indicating its effectiveness.

4.4 Ablation Study↩︎

Contribution of each component. To quantify the contribution of the proposed modules, we conduct an ablation study on both original and noisy RAF-DB. For simplicity, we refer to the performance on these two datasets as \((a, b)\) in the following. As shown in ¿tbl:tab:ablation?, complete LDE (line 3) promotes performance by (1.98%, 6.87%) compared to the baseline in line 1. EL Loss (line 5), on the other hand, yields gains of (1.77%, 2.96%) in contrast to the model in line 3. These two modules leverage landmark information from different perspectives and mitigate real-world and synthetic noise effectively. Besides, comparing line 2 and line 3, landmark information in LDE brings the improvement of (0.61%, 3.20%), indicating its importance, especially in the presence of severe noise. Additionally, the pseudo-labels (line 5) enable noise-tolerant supervised contrastive learning and achieve an improvement of 2.12% on corrupt RAF-DB compared to the EL Loss using one-hot labels (line 4). Overall, the results in ¿tbl:tab:ablation? demonstrate the effectiveness of the proposed modules and the advantages of their combination in LA-Net.

Number of nearest neighbors in each space \(K\). We explore the effect of \(K\) on the performance in 2. For original RAF-DB, LA-Net achieves optimal performance with \(K\)=12. Smaller or larger \(K\) can lead to slight degradation, as the former may result in inaccurate target label distributions, while the latter can cause excessive noisy neighbors. The choice of \(K\) has a similar effect on both original and noisy RAF-DB. The only exception is that LA-Net achieves the best performance on the noisy RAF-DB with \(K\)=8 since it contains more corrupt labels. Therefore, we set \(K\) to 8 throughout experiments to alleviate label noise.

4.5 Experiments on Original Datasets↩︎

While the original datasets are considered "clean", they inevitably contain noisy labels due to the ambiguity of expressions. Hence, we compare LA-Net with state-of-the-art methods on original datasets to verify its resistance to real-world uncertainty. Moreover, we consider both 7-class and 8-class AffectNet for a fair comparison.

We compare the performance of LA-Net with several state-of-the-art FER models in 1. Among the existing models, TransFER [13] utilizes the powerful Vision Transformer [42] structure to improve performance, while the others enhance FER performance by locating key areas or alleviating label noise. Compared to existing models, LA-Net gains improvements of 0.65%, 1.37%, 1.70%, and 0.95% on RAF-DB, 7-class AffectNet, 8-class AffectNet, and FERPlus, respectively. Moreover, the proposed model also exhibits substantial advantages in challenging cases, such as 7-class and 8-class AffectNet. Our findings indicate that LA-Net is a promising approach for handling ambiguous expressions in real-world scenarios.

4.6 Experiments with Inconsistent Labels↩︎

The performance of deep neural networks generally improves as more training samples are provided, but FER presents a unique challenge. Due to the subjective nature of interpreting facial expressions, individuals may assign different labels to the same expression, leading to inconsistent labels within a dataset and across different datasets. To evaluate the effectiveness of LA-Net in addressing this problem, we conduct a cross-dataset experiment following previous works [14], [25], [43]. Specifically, we train our model on a mixed dataset composed of the training samples from RAF-DB and AffectNet and evaluate it on the testing set of both datasets. Note that we use the 7-class AffectNet to ensure the identical class set between two datasets.

We will analyze the results in 2 from two perspectives. Firstly, compared to LDLVA [14], which is the best available model for addressing label inconsistency, LA-Net achieves the improvement of 0.84% and 2.54% on RAF-DB and AffectNet, respectively. Secondly, LA-Net helps alleviate the performance drop caused by cross-dataset inconsistency. Specifically, comparing 2 and 1, LA-Net encounters the degradation of 2.71% on RAF-DB and 1.66% on AffectNet, whereas LDLVA suffers from the drop of 3.25% and 3.34% on these two datasets, respectively. In summary, the improvement in two aspects demonstrates the effectiveness of LA-Net in dealing with inconsistent labels. Nonetheless, cross-dataset inconsistency remains a challenge for FER models and warrants further research.

4.7 Visualization Analysis↩︎

High-dimensional features. For an intuitive understanding of LA-Net, we use t-SNE [46] to plot the trained features of different models, including a baseline model, SCN [4], and our LA-Net, on RAF-DB with 30% noise. As shown in 3 (a), the baseline model memorizes the noisy labels, resulting in adjacent feature clusters. SCN learns the uncertainty of each sample and proposes a relabeling mechanism to address label noise. Nonetheless, as depicted in 3 (b), it still overfits many noisy samples, such as the disgust expressions in the cluster of anger. In comparison, LA-Net, plotted in 3 (c), pushes noisy samples to the decision boundary and generates clean clusters. Moreover, our model achieves improved separation across categories and better compactness within each class. Overall, LA-Net prevents overfitting noisy samples and forms easy-to-distinguish category clusters in the presence of severe noise, indicating its effectiveness in dealing with noisy labels.

Cross-entropy between predictions and one-hot labels. We evaluate the ability of LA-Net to handle noisy labels by calculating the cross-entropy between predictions and one-hot labels of the training samples after training. As shown in 4 (a), the baseline model memorizes almost all noisy labels, resulting in poor generalization ability. SCN [4] alleviates the noise to some extent by relabeling, whereas it still overfits some corrupt labels since these noisy samples are not identified or mistakenly relabeled. In contrast, LA-Net, which leverages landmark information to construct label distributions and enhance expression feature extractor, could easily distinguish between clean and noisy samples after training. Overall, the results in 4 indicate that our model can suppress label noise to a large extent.

Target label distributions. We conduct a user study on several images randomly selected from RAD_DB and AffectNet to diagnose the LDE module. We present part of the results in 5, and more details are provided in Appendix C. The left two expressions in 5 are found to be ambiguous in the user study, while the one-hot labels only provide one possible category. In contrast, LDE reveals all possible classes and generates label distributions that are consistent with the user study results. Additionally, the model correctly identifies the latent truth for the third image, which is mistakenly assigned to happiness in the dataset (highlighted in red). In conclusion, the results indicate that LA-Net can achieve agreement with human perception to some extent and effectively mitigate label noise.

Attention visualization. To further analyze how landmark information benefits LA-Net, we randomly select some images from AffectNet and generate their attention maps using GradCAM [47]. Note that we perform visualization in the same setting as in inference mode. That is, no landmark information is provided when extracting attention maps.

6 plots the attention maps of the selected images, where the categories are anger, disgust, fear, happiness, sadness, surprise, and neutral from top to bottom. Column (I) provides the images and their landmarks, and the rest presents the attention maps of three strategies. Comparing different columns, the baseline model typically focuses on a few areas, while the LDE module reveals more crucial areas by providing better supervision in the presence of label noise. Moreover, EL Loss contributes to locating more key regions, such as the eyes in (c, IV) and (d, IV), and boosts the attention of key areas, including the eyes and mouth in (e, IV). Overall, LA-Net generates regions of interest that align well with facial landmarks, indicating that it develops a knowledgeable feature extractor by incorporating landmark information into expression representations.

5 Conclusion↩︎

This paper introduces a new FER model named LA-Net, which aims to mitigate the impact of label noise by incorporating landmark information. LA-Net consists of two main modules, namely label distribution estimation and expression-landmark contrastive loss. The former uses landmark information to correct errors in expression space and estimates the label distribution of each sample, which in turn provides better supervision. The latter enables noise-tolerant supervised contrastive learning and develops a more robust feature extractor by performing interactions between expressions and landmarks. We conduct extensive experiments on multiple scenarios, whose results demonstrate the effectiveness of LA-Net in handling label noise and inconsistent labels. Additionally, we present various visualization studies to illustrate how landmark information enhances the performance of LA-Net.

6 Performance with Asymmetric Noise↩︎

¿tbl:tab:synthetic-noise? follows prior studies and generates noisy labels by randomly flipping the label to other classes uniformly, referred to as symmetric noise. Given that symmetric noise may not reflect real-world ambiguity, we test asymmetric noise, where the label is flipped to its most similar class based on the confusion matrix. As shown in ¿tbl:tab:asymmetric?, LA-Net consistently outperforms previous models when dealing with asymmetric noise. Specifically, compared to EAC, the approach achieves an average promotion of 1.24%, 1.15%, and 1.61% on the three datasets. Overall, the impressive robustness against asymmetric noise demonstrates the potential of LA-Net to be deployed in real-world applications.

7 More Grid Search Results.↩︎

We test various temperature values \(\tau\) (0.05, 0.07, 0.1, 0.15, 0.2) and find 0.1 is optimal. We test various momentum decay \(\omega\) (0.8, 0.85, 0.9, 0.95, 0.99) and find 0.9 is optimal. We present the grid search for threshold \(\delta\) and weights \(\alpha\), \(\beta\) in 7.

8 More User Study Results↩︎

We conduct a user study using 50 images selected from AffectNet [29] and compare the one-hot labels, human perception, and generated label distributions to evaluate the effectiveness of our LA-Net. To present our findings more succinctly, we plot all of the mistakenly annotated and ambiguous images, as well as a part of the correctly labeled images (24 in total) in 8.

The first two lines present several images that are mistakenly annotated in AffectNet (highlighted in red). LA-Net identifies these errors and reveals the latent truth. Moreover, images plotted in rows 3-6 are found to be ambiguous according to user study results (highlighted in blue). Fortunately, the proposed model generates label distributions consistent with human perception, reducing the uncertainty of these ambiguous samples. Additionally, as shown in the last two lines of 8, LA-Net produces targets that align well with the one-hot labels for correctly annotated samples (highlighted in green). Overall, the model demonstrates some level of agreement with human perception and effectively mitigates label noise.

In addition to visualization, we quantitatively evaluate the consistency between the label distributions and the user study results of the selected images using Jensen-Shannon divergence (JS divergence). Mathematically, given probability distributions \(\boldsymbol{p_1}\) and \(\boldsymbol{p_2}\), their JS divergence can be calculated by:

\[\begin{align} JS(\boldsymbol{p_1}||\boldsymbol{p_2}) &= \frac{1}{2}KL(\boldsymbol{p_1}||\frac{\boldsymbol{p_1}+\boldsymbol{p_2}}{2}) + \frac{1}{2}KL(\boldsymbol{p_2}||\frac{\boldsymbol{p_1}+\boldsymbol{p_2}}{2}) \end{align}\] 3 reveals a significant difference between one-hot labels and user study results, indicating that FER datasets [26], [27], [29] suffer from serious label noise. In contrast, LA-Net generates label distributions that are in better agreement with human perception. This improvement leads to better FER performance, especially when dealing with label noise.

References↩︎