Abstract
Because of the development of metaverse, robotics, emotional computing has become a very active research topic. And the most important part of affective computing is facial expression recognition, The existing deep learning-based face expression recognition methods are trained on small face expression datasets, and the current datasets have the problems of low quantity, low quality and category imbalance, and the models are prone to overfitting when trained on these small datasets. Compared with other methods using small datasets, this paper proposes a method to build a large face expression dataset by collecting data from the Internet. It also uses both downsampling and data augmentation to solve the category imbalance problem. Meanwhile, this paper experiments the effect of different network structures on face expression recognition based on this brand new dataset. Compared to approaches using small datasets such as raf-db, fer2013, and relatively large but category-unbalanced datasets such as ExpW and AffectNet, the densenet-based model proposed in this paper, trained on this dataset, achieved 97.5% accuracy on the CK+ benchmark, achieve the state-of-the-art performance, which confirms the effectiveness of our proposed approach.
Article Highlights
-
1.
In this paper, we describe a way to make a new face expression dataset by putting together open source and self-collected datasets.
-
2.
This allows us to create the largest face expression dataset in the industry, which is better in quality and has more data.
-
3.
When we use this new dataset to train a face expression recognition model, it performs the best in the current industry.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
With the development of metaverse, intelligent interaction, and intelligent robotics, affective computing has become the most active research topics. And recognizing expressions is a key part of affective computing. Facial expressions are the most important way for humans to transmit emotions when communicating, and facial expressions greatly enhance the accuracy of expressive communication. Ekman et al. [1] defined six basic expressions through a large number of cross-cultural studies: Angry, Sad, Happy, Fear, Disgust and Surprise as show in Fig. 1. As a result, the academic community has generally started the exploration of Facial expression recognition (FER) by setting these 6 basic expressions. Facial expression recognition is a key technology for understanding human emotions, and is widely used in e-commerce, service industry, security industry, education industry, etc. It has been an active topic in the field of pattern recognition and artificial intelligence for many years [2].
Deep learning has benefited from Moore’s law and big data, and has gained rapid development in the last decade, especially the development of big data, which makes it possible to build large datasets. Training complex models using small datasets can easily lead to overfitting, while large datasets can be used for training more complex models and improving the generalization ability of the models [3].
Deep learning-based face expression recognition also requires the support of large datasets, and training a deep enough network to capture the subtle deformations associated with expressions requires a large amount of relevant data. Therefore, a relatively scarce database in terms of both quantity and quality is the main challenge for today’s deep face expression recognition systems. In recent years, Convolution Neural Network (CNN) models [4], such as VGGface [5], Googlenet [6], and Resnet [7], which have been prominent in the field of image processing and analysis [8], often have a dozen to tens of convolutional layers and hundreds of trillion Bytes of network parameters, such complex models need more large datasets to support.
Some of the most popular facial expression datasets in use in the industry are the Advanced Telecommunications Research Institute International’s (ATR) JAFFE [9] and Cohn-kanade expression database (CK), which are basic expression databases dedicated to expression recognition research. and the improved dataset CK+ by Lucey et al. [10]. These datasets are mainly taken from laborers employed in laboratories, these datasets are of good quality, but they are expensive and very small in size. Although more and more expression datasets have been made publicly available in recent years, face expression datasets generally suffer from small data size, insufficient data volume, relatively homogeneous dataset information, and unbalanced data variety, which are far from sufficient for use in expression recognition for deep learning. Zhang et al. [11] proposed the ExpW dataset and attempted to use a web crawling dataset approach to solve the existing problems, and the Mollahosseini et al. [12] proposed AffectNet dataset, which has a total of 1 million, but these two datasets are relatively poor on quality, and there are a lot of mislabeling which they are using machine learning methods for labeling, and the category imbalance is very serious because the data comes from the Internet.
Inspired by ExpW and AffectNet, we obtain a large number of face expression images from image search engines using a large number of near synonyms of expression names as search keywords and classify them using manual annotation to obtain a total of 272,844 face expression images. By fusing with the publicly available dataset and removing noise as well as duplicate images, a total face expression dataset of 586,712 is obtained. This dataset is larger than ExpW, Kdef, and Raf in size, and after manual annotation, the data quality is much better than affectNet which using machine learning annotation, also has the best inter-category balance. Based on this dataset, this paper also explores the performance of the recently widely used resnet and densenet on face expression recognition tasks. It is demonstrated that the state-of-the-art performance is achieved on benchmarks such as CK+, MMI, etc. using convolutional neural networks with more layers and more parameters based on high quality large-scale datasets. It solves the problems of overfitting, poor generalization ability, and low accuracy that are easily caused by small datasets in the past.
This article has 5 sections. In the next section, we will detail the process of crawling, blending, and creating the dataset. In Sect. 3, we introduce the network structure used and the hyper-parameters of the experiments. Section 4 presents the experimental results and visualizes the features of the model. In the final section, we summarize our work and discuss the limitations and future plans.
2 Process of building large expression datasets
In this paper, we follow three steps to construct a large face expression dataset.
-
a.
Collect existing publicly available lab-acquired face datasets, including Raf, KDEF, CK+, MMI, etc. Collect existing datasets acquired by Internet crawlers, including ExpW, Fer2013, AffectNet, etc., and remove noise from these datasets, including removing duplicate images, removing low-resolution images, etc.
-
b.
Create association words about 6 types of basic expressions, use the association words to crawl a large amount of face images in Internet search engines, and use face detection algorithm to detect whether there is a human face on the crawled images and perform basic de-noise processing.
-
c.
The above two datasets were combined and obtained the initial expression image dataset, the class imbalance of the dataset was solved by using downsampling and data enhancement methods, and the total 586,712 face expression dataset was obtained after filtering the noise of the dataset using the manual review method. These three steps are briefly described below.
2.1 Collect publicly available datasets and remove noise
A total of seven public datasets containing the following sources were collected, and the analysis found that there were many duplicate images and low-resolution images in the dataset from the Internet, and after eliminating these images, a dataset of 180,255 facial expressions was obtained:
-
a.
ExpW, 91,793 Zhang, source network, contains 6 basic expressions and neutral expressions.
-
b.
Fer2013, 35,887, source network, contains 6 basic expressions and neutral expressions.
-
c.
Raf, collected by the source laboratory, removed 13 types of compound expressions, leaving 15,152, including 6 basic expressions and neutral expression data [13].
-
d.
KDEF, 4900 sheets, collected from the source laboratory, including 6 basic expressions and neutral expressions [14].
-
e.
AffectNet total 450,000 sheets, from the network, including 6 basic expressions and neutral expressions, are 287,401 sheets after removing 7 categories (contempt, none, uncertain, nonface).
-
f.
CK+, 327, source laboratory, contains 6 basic expressions and neutral expressions.
-
g.
MMI, 740 sheets, source laboratory, contains 6 basic expressions and neutral expressions [15].
2.2 Create keywords and crawl the web for images
A series of associated words for the six basic expressions were created using methods such as proxemics, and then these keywords were used to crawl images of faces in image search engines, creating the following keywords.
Neutral indifferent, nonaligned, diaphanous.
Surprise amazed, astonishment, astounded, surprised, astonished, besurprised, beamazed, stunned, stupefied, wonder, marvelousness, astonishing, portentous.
Sadness unhappiness, sorrow, grief, tragedy, depression, the blues, misery, melancholy, poignancy, despondency, bleakness, heavy-heart, dejection, wretchedness, gloominess, mournfulness, dolour, dolefulness, cheerlessness, sorrowfulness.
Happy pleased, delighted, content, contented, thrilled, glad, blessed, blest, sunny, cheerful, jolly, merry, ecstatic, gratified, jubilant, joyous, joyful, elated.
Fear dread, horror, panic, terror, dismay, awe, fright, tremors, qualms, consternation, alarm, timidity, fearfulness.
Disgust sicken, outrage, offend, revolt, put off, repel, nauseate.
Anger rage, passion, outrage, temper, fury, resentment, irritation, wrath, indignation, annoyance, agitation, antagonism, displeasure.
The face detection model is used to detect whether there is a face in the crawled face picture, and the face is aligned with the detected key points, and the face is saved to the corresponding classification folder. Human inspection removes the face images that do not match the folder classification, and finally creates a 272,842 facial expression dataset.
2.3 Analyze and solve the problems of the initial expression image dataset
2.3.1 Category imbalance
Almost all datasets have a serious imbalance of categories as show in Fig. 2, which the ratio of the size of big categorie to the size of small categorie reaching 20:1.
Solution:
Down-sampling is used for serveral categories, and repeated sampling is used for a small number of categories according to the proportion between categories, and image enhancement is used [16].
Different coefficients are assigned to the class objective function, and a small number of classes are given a higher loss, so that they learn more.
2.3.2 Dataset noise problem
Several public datasets have the problem of noise, for example, affectnet has non-face data, and there are many mislabeled expression category data.
Solution:
-
a.
A face detection model is used to detect whether a face exist in that picture, and the picture without the face is discarded.
-
b.
Use the reference facial expression recognition model to judge whether the expression classification is correct, and manually confirm if any error is found.
We obtain an initial dataset by merging the data sets crawled from the internet and the publicly available datasets, with a total of 672,090, as shown in Table 1. We use downsampling and data augmentation to balance the categories, so that the total amount of each category is in the same order of magnitude. We also remove non-face and low-resolution data from the dataset, and manually correct the wrong labels to obtain a facial expression dataset with a total of 586,712, as shown in Table 2.
3 Adjust the network structure to optimize the facial expression recognition method
3.1 MTCNN face detection algorithm
In this paper, we use mtcnn as a face extraction model. mtcnn is a lightweight and efficient face detection model that can quickly obtain all face coordinates in images [17], which mainly uses three cascaded convolutional neural network architectures: P-Net, R-Net and O-Net as shown in Fig. 3. The P-Net network for fast generation of candidate windows is the front-end detection network, which is mainly used for fast and high-precision selection of candidate windows. The intermediate level R-Net discriminates the presence of misidentified candidate boxes,The detected candidate boxes and the screened candidate boxes are transmitted to the final network structure O-Net to generate the final bounding box and output the final facial feature key points.
3.2 Basic network of facial expression classification model
In this paper, experiments are conducted on two basic networks: ResNet and DenseNet. In order to solve the problem of degeneration in the deep network, we can artificially make some layers of the neural network skip the connection of the neurons in the next layer, connect the layers and weaken the strong connection between each layer as show in Fig. 4. Such neural networks are called residual networks (ResNets), and ResNet introduces a residual structure (residual structure) to alleviate the degradation problem.The residual structure uses a shortcut connection, which can also be understood as a shortcut. Let the feature matrix be added layer by layer. Note that the shape of F (X) and X should be the same. The so-called addition is to add the numbers in the same position of the feature matrix.
DenseNet breaks away from the fixed thinking of deepening the number of network layers (ResNet) and widening the network structure (Inception) to improve the network performance. From the perspective of features, DenseNet not only greatly reduces the number of network parameters, It also alleviates the gradient vanishing problem to a certain extent [18].
DenseNet show in Fig. 5 breaks away from the fixed thinking of deepening the number of network layers (ResNet) and widening the network structure (Inception) to improve the network performance. From the perspective of features, DenseNet not only greatly reduces the number of network parameters, It also alleviates the gradient vanishing problem to a certain extent.
DenseNet, as another convolutional neural network with a deeper number of layers, has the following advantages. Firstly, it has fewer parameters than ResNet. Secondly, bypassing enhances the reuse of features. Thirdly, the network is easier to train and has a certain regularization effect. Lastly, the problems of gradient vanishing and model degradation are alleviated.
3.3 Network structure adjustment method
3.3.1 Training to adjust hyper-parameters
In this paper, the SGD optimization algorithm is used to train the two basic models, and the learning rates of 0.001, 0.0003 and 0.0005 are tried respectively [19].
3.3.2 Training of tuning data enhancement method
In this paper, random horizontal flip, random angle rotation (within 5 degrees), random color jitter and random fine-tuning brightness are used to enhance the face data. The control variable method is used to test the impact of different enhancement methods on the experimental results [20]. The following is the implementation code.
# random horizontally flip
Flag = int(random.random() > 0.5)
if flag:
output_image = cv2.flip(image,1)
elif:
output_image = image.
flag = int(random.random() > 0.5)
if flag:
output_image = cv2.flip(image,1)
elif:
output_image = image.
# Rotation with random angle
MAX_ANGLE = 10
angle = int (MAX_ANGLE * (random.random()-0.5) * 2)
output_image = image.rotate(angle).
MAX_ANGLE = 10
angle = int (MAX_ANGLE * (random.random()-0.5) * 2)
output_image = image.rotate(angle)
# Adjust brightness
Factor = random.random() * 2
image = ImageEnhance.Brightsness(input_image)
image.enhance(factor)
factor = random.random() * 2
image = ImageEnhance.Brightsness(input_image)
image.enhance(factor)
# Adjust color
Factor = random.random() * 2
image = ImageEnhance.Color(input_image)
image.enhance(factor)
factor = random.random() * 2
image = ImageEnhance.Color(input_image)
image.enhance(factor)
4 Experimental results
All experiments were carried out on a workstation, configured with intel I9-10900 k, 64 GB memory, 2 * RTX3090, 1T pcie3.0 SSD, and the working platform was tensorflow.
4.1 Test dataset
Test datasets Fer2013, CK+, MMI, all three datasets deal with static face images and label them with corresponding labels. After the training model is reasoned on the three datasets and labeled, it is compared with the ground truth of the dataset to obtain the accuracy. The higher the accuracy, the better the model.
4.2 SOTA (state of the art)
At present, the following two models perform best in CK+ on static picture expressions.
-
a.
Facial Expression Recognition by De-Expression Residue Learning (thecvf.com) [21], the highest accuracy is 97.3%.
-
b.
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition [22], Maximum accuracy 97.25.
4.3 Hyper-parameter experiment
In this paper, the hyperparameters of the model are explored, and different combinations of hyperparameters are used to train the model. The control variables include learning rate, Regularization coefficient, as show in Tables 3 and 4.
From the test results, the most stable model training effect is obtained by using the optimization method of SGD, with a learning rate of one in ten thousand and a regularization parameter of one in a thousand.
4.4 Enhancement method experiment
In this paper, four data enhancement methods are explored on two types of models (Resnet, Densenet). In each experiment, only one enhancement method is started, and the others are all turned off, in order to test which enhancement method performs more influential, as shown in Tables 5 and 6.
The experimental results show that the horizontal flip and random angle rotation can improve the stability of the model, while the other two methods have little effect.
4.5 Comparison results
The Resnet50 model we trained achieved 97.3% accuracy on CK+ dataset 7 classification, and the DenseNet (40) model achieved 97.5% accuracy, exceeding SOTA, show in Table 7.
Approachs like [22,23,24] use small models to finetune on CK+, approachs that require careful design of the model and careful monitoring of the training process to avoid overfitting of the model. The dataset we constructed is two orders of magnitude larger in total than, for example, the CK+ dataset, and the larger dataset is consistent with the assumption that deep learning is built on big data. We are able to use a convolutional neural network model with a larger number of parameters without easily causing overfitting, and the model generalization ability is excellent.
4.6 Feature visualization
After getting good experimental results, we also want to know what the “black box” convolutional neural network model learns from training, what features in the input activate the convolutional neural network, and how the convolutional neural network works to help us continue to improve the task in the future. Inspired by Olah et al. [25], we initialize a random value image and select a specific feature layer as the output, freeze the model parameters, iteratively optimize the random value image, and finally obtain a visualization of the feature layer. From the visualization results as show in Fig. 6, we see that the convolutional neural network model focuses on the eyes, mouth, and the surrounding structural relationships when performing the face expression recognition task, which shows that it is quite important to ensure the clarity of the eyes and mouth when the model performs expression recognition.
5 Conclusion
The goal of affective computing is to understand the expression recognition content of natural interaction processes. Facial expression recognition is the basis of human emotion recognition, which is a hot issue in the field of pattern recognition and artificial intelligence in recent years. This paper presents the problems of current datasets used in face expression methods, including low quantity, low quality, high noise, and severe category imbalance. And a method is proposed to crawl data from the Internet, fuse open source datasets, and use data augmentation to solve the category imbalance problem. A total face expression dataset of 586,712 is obtained, and the number of each category is relatively balanced. Compared to other datasets like ExpW and affectNet, this new dataset is larger in number and better in image quality, the annotation is manually corrected and the number of categories is adjusted. Based on this dataset, we explores the effect of different deep network structures on the performance of face expression recognition. Experiments demonstrate that based on a high-quality large-scale dataset, using a deeper, residual-connected densenet can achieve better performance, and the model achieves 97.5% accuracy on CK+ benchmark, sota on validation sets such as ck+, achieve the state-of-the-art performance. But the dataset created by this method mostly comes from the internet, which also introduces biases from the internet, such as even after processing, there are still more data in specific categories. At the same time, a lot of data is collected under deliberately performed conditions, which is quite different from the natural form of expression. Our next phase of work is to obtain more natural expression of emotion from different channels, such as recording from the audience at large sports events, or collecting different emotional expressions from the audience watching different types of films in the laboratory. A broader source of emotional dataset can solve the current limitations of internet crawling.
Data availability
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Change history
16 May 2024
This article has been retracted. Please see the Retraction Notice for more detail: https://doi.org/10.1007/s42452-024-05948-5
References
Ekman P, Grieson WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17:124–129. https://doi.org/10.1037/h0030377
Sun W, Wang B (2012) A survey of facial expression recognition. Comput Knowl Technol
Li H et al (2019) Research on overfitting of deep learning. In: 2019 15th International Conference on Computational Intelligence and Security (CIS). IEEE. https://doi.org/10.1109/CIS.2019.00025
Gu J, Wang Z, Kuen J et al (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377. https://doi.org/10.1016/j.patcog.2017.10.013
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. https://doi.org/10.48550/arXiv.1409.1556
Szegedy C, Liu W, Jia Y et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9. https://doi.org/10.48550/arXiv.1409.4842
He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Li Z, Liu F, Yang W et al (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3084827
Lyons M, Akamatsu S, Kamachi M et al (1998) Coding facial expressions with gabor wavelets. In: Proceedings of third IEEE international conference on automatic face and gesture recognition.[S.1.], pp 200–205. IEEE. https://doi.org/10.5281/zenodo.4029679
Lucey P, Cohn JF, Kanade T et.al (2010) The extended cohn-kanade dataset(CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of 2010 IEEE computer society conference on computer vision and pattern recognition-workshops.[S.1}, pp 94–101. IEEE. https://doi.org/10.1109/CVPRW.2010.5543262
Zhang Z, Luo P, Loy CC et al (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126(5):550–569. https://doi.org/10.1007/s11263-017-1055-1
Mollahosseini A, Hasani B, Mahoor MH (2017) Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31. https://doi.org/10.48550/arXiv.1708.03985
Li S, Deng W, Du JP (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2017.277
Lundqvist D, Flykt A, Öhman A (1998) Karolinska directed emotional faces. Cognit Emot. https://doi.org/10.1037/t27732-000
Pantic M, Valstar M, Rademaker R et al (2005) Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, p 5. IEEE. https://doi.org/10.1109/ICME.2005.1521424
Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6(1):1–54. https://doi.org/10.1186/s40537-019-0192-5
Xiang J, Zhu G (2017) Joint face detection and facial expression recognition with MTCNN. In: 2017 4th international conference on information science and control engineering (ICISCE), pp 424–427. IEEE. https://doi.org/10.48550/arXiv.1604.02878
Zhang Y, Tian Y, Kong Y et al (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481. https://doi.org/10.48550/arXiv.1802.08797
Keskar NS, Socher R (2017) Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv:1712.07628. https://doi.org/10.48550/arXiv.1712.07628
Bloice MD, Stocker C, Holzinger A (2017) Augmentor: an image augmentation library for machine learning. arXiv preprint arXiv:1708.04680
Yang H, Ciftci U, Yin L (2018) Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2168–2177
Jung H, Lee S, Yim J et al (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision, pp 2983–2991
Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. arXiv preprint arXiv:1408.3750. https://doi.org/10.48550/arXiv.1408.3750
Liu M, Li S, Shan S et al (2013) Au-aware deep networks for facial expression recognition. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6. IEEE. https://doi.org/10.1109/FG.2013.6553734
Olah C, Mordvintsev A, Schubert L (2017) Feature visualization. Distill 2.11: e7. (https://distill.pub/2017/feature-visualization/. Accessed 12 Dec 2022
Funding
This work is partially supported by Shenzhen Education Science “14TH FIVE-YEAR PLAN”2021 Subject: Research on online learning emotion analysis and intelligent tutoring based on collaborative perception of multi-modal education data(ybzz21015), Key technology research and innovative application demonstration of intelligent education(2019KZDZX1048), Guangdong Key Laboratory of Big Data Intelligence for Vocational Education(2019GKSYS001), Shenzhen Vocational Education Research Center Jointly Established by the Ministry and the Province(6022240004Q).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declares that there are no conflicts of interest regarding the publication of this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article has been retracted. Please see the retraction notice for more detail: https://doi.org/10.1007/s42452-024-05948-5"
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lu, J., Wan, X. RETRACTED ARTICLE: Affective computing model for natural interaction based on large-scale self-built dataset. SN Appl. Sci. 5, 53 (2023). https://doi.org/10.1007/s42452-023-05277-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42452-023-05277-z