Abstract
Emotion categorization can be the process of identifying different emotions in humans based on their facial expressions. It requires time and sometimes it is hard for human classifiers to agree with each other about an emotion category of a facial expression. However, machine learning classifiers have done well in classifying different emotions and have widely been used in recent years to facilitate the task of emotion categorization. Much research on emotion video databases uses a few frames from when emotion is expressed at peak to classify emotion, which might not give a good classification accuracy when predicting frames where the emotion is less intense. In this paper, using the CK+ emotion dataset as an example, we use more frames to analyze emotion from mid and peak frame images and compared our results to a method using fewer peak frames. Furthermore, we propose an approach based on sequential voting and apply it to more frames of the CK+ database. Our approach resulted in up to 85.9% accuracy for the mid frames and overall accuracy of 96.5% for the CK+ database compared with the accuracy of 73.4% and 93.8% from existing techniques.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
People that are trained in emotion categorization. These people labeled these databases based on the assumption that people smile when happy, frown their faces when sad, and scowl when anger irrespective of their age, race, and ethnicity.
References
Tian, Y.-L., Kanade, T., Cohn, J.F.: Facial expression analysis. In: Handbook of Face Recognition, pp. 247–275. Springer, New York (2005). https://doi.org/10.1007/0-387-27257-7_12
Martinez, B., Valstar, M.F.: Advances, challenges, and opportunities in automatic facial expression recognition. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 63–100. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-25958-1_4
Matsumoto, D., Hwang, H.S.: Evidence for training the ability to read microexpressions of emotion. Motiv. Emot. 35(2), 181–191 (2011)
Krumhuber, E.G., Küster, D., Namba, S., Shah, D., Calvo, M.G.: Emotion recognition from posed and spontaneous dynamic expressions: human observers versus machine analysis. Emotion (2019)
Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)
Chakravarti, A.: Perspectives on human variation through the lens of diversity and race. Cold Spring Harb. Perspect. Biol. 7(9), a023358 (2015)
Islam, B., Mahmud, F., Hossain, A.: Facial region segmentation based emotion recognition using extreme learning machine. In: 2018 International Conference on Advancement in Electrical and Electronic Engineering, ICAEEE, pp. 1–4 (2019)
Mahmud, F., Islam, B., Hossain, A., Goala, P.B.: Facial region segmentation based emotion recognition using K-nearest neighbors. In: 2018 International Conference on Innovation in Engineering and Technology, ICIET 2018, pp. 1–5 (2019)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, pp. 94–101 (2010)
Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: DISFA: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151–160 (2013)
Longmore, C.A., Tree, J.J.: Motion as a cue to face recognition: evidence from congenital prosopagnosia. Neuropsychologia 51, 864–875 (2013)
Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)
Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, pp. 1–10 (2016)
Minaee, S., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019 (2019)
Elaiwat, S., Bennamoun, M., Boussaid, F.: A spatio-temporal RBM-based model for facial expression recognition. Pattern Recognit. 49, 152–161 (2016)
Kim, J.H., Kim, B.G., Roy, P.P., Jeong, D.M.: Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7, 41273–41285 (2019)
Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)
Xiao, R., Li, X., Li, L., Wang, Y.: Can we distinguish emotions from faces? Investigation of implicit and explicit processes of peak facial expressions. Front. Psychol. 7, 1330 (2016). (1664–1078)
Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings - 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000, March, pp. 46–53 (2000)
Ting, G., Moydin, K., Hamdulla, A.: An overview of feature extraction methods for handwritten image retrieval. In: Proceedings - 2018 3rd International Conference on Smart City and Systems Engineering, ICSCSE 2018, pp. 840–843 (2018)
Pisal, A., Sor, R., Kinage, K.S.: Facial feature extraction using hierarchical max(HMAX) method. In: 2017 International Conference on Computing, Communication, Control and Automation, ICCUBEA 2017, (figure 2), pp. 1–5 (2018)
Loussaief, S., Abdelkrim, A.: Machine learning framework for image classification. In: 2016 7th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications, SETIT 2016, pp. 58–61 (2017)
Li, Y., Wang, S., Zhao, Y., Ji, Q.: Simultaneous facial feature tracking and facial expression recognition. IEEE Trans. Image Process. 22(7), 2559–2573 (2013)
Cruz, A.C., Bhanu, B., Thakoor, N.S.: One shot emotion scores for facial emotion recognition. In: 2014 IEEE International Conference on Image Processing, ICIP 2014, (C), pp. 1376–1380 (2014)
Safavian, S.R., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Trans. Syst. Man Cybernet. 21(3), 660–674 (1991)
Shehu, H.A., Browne, W., Eisenbarth, H.: An adversarial attacks resistance-based approach to emotion recognition from images using facial landmarks. In: 2020 IEEE International Conference on Robot and Human Interactive Communication (2020)
Sohail, A.S.M., Bhattacharya, P.: Classification of facial expressions using K-nearest neighbor classifier. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 555–566. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-71457-6_51
Kamiński, B., Jakubczyk, M., Szufel, P.: A framework for sensitivity analysis of decision trees. Central Eur. J. Oper. Res. 26(1), 135–159 (2017). https://doi.org/10.1007/s10100-017-0479-6
Shehu, H.A., Tokat, S., Sharif, M.H., Uyaver, S.: Sentiment analysis of Turkish Twitter data. In: AIP Conference Proceedings, vol. 2183, no. 1, p. 080004. AIP Publishing LLC, December 2019
Shehu, H.A., Tokat, S.: A hybrid approach for the sentiment analysis of Turkish Twitter data. In: Hemanth, D.J., Kose, U. (eds.) ICAIAME 2019. LNDECT, vol. 43, pp. 182–190. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-36178-5_15
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Altman, N.S.: An introduction to kernel and nearest-neighbor non-parametric regression. Am. Stat. 46(3), 175–185 (1992)
Fan, Y., Lam, J.C.K., Li, V.O.K.: Multi-region Ensemble Convolutional Neural Network for Facial Expression Recognition. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 84–94. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01418-6_9
Chengeta, K., Viriri, S.: A review of local, holistic and deep learning approaches in facial expressions Recognition. In 2019 Conference on Information Communications Technology and Society (ICTAS), pp. 1–7. IEEE, March 2019
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Shehu, H.A., Browne, W., Eisenbarth, H. (2020). Emotion Categorization from Video-Frame Images Using a Novel Sequential Voting Technique. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12510. Springer, Cham. https://doi.org/10.1007/978-3-030-64559-5_49
Download citation
DOI: https://doi.org/10.1007/978-3-030-64559-5_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64558-8
Online ISBN: 978-3-030-64559-5
eBook Packages: Computer ScienceComputer Science (R0)