Abstract
The posterity challenging task in the Machine Intelligence field is to design a smarter system to identify the human emotions. Facial Emotion Recognition (FER) is a significant visual based tool to construct a smarter system that can recognize human emotions. The existing methods in FER are based on Action Units (AU), appearance and geometrical parameters. Nearly 7000 different combinations of AUs are used in AU to discriminate the emotions, which can be very expensive and increase processing time. Generalize appearance features across the universe is another challenging task. In this paper, a novel geometrical fuzzy based approach is presented to accurately recognize the emotions. The four corner features from eyes and mouth regions are extracted without considering reference face. The extracted features are used to define the quadrilateral shape that failed to match with the shapes in geometry. The degree of impreciseness exists in the quadrilateral is measured by the proposed Mixed Quadratic Shape Model (MQSM) using fuzzy membership functions. Finally, twelve fuzzy features are extracted from the membership functions and used by the classifier for validation. The CK, JAFFE and ISED datasets are used in the experiment to evaluate the performance of the MSQM. It is observed the proposed method performed better than the contemporary methods using twelve fuzzy features without reference image.
Similar content being viewed by others
References
Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Int J of Multimed Tools Appl 76:1073–1099
Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–118
Anderson K, McOwan PW (2004) Robust real-time face tracker for use in cluttered environments. Comput Vis Image Underst 95(2):184–200
Anderson K, McOwan PW (2006) A real-time automated system for the recognition of human facial expressions. IEEE transactions on systems, man. And Cybernetics-Part B: Cybernetics 36(1):96–105
Bargal SA, Barsoum E, Ferrer CC, Zhang C (2016) Emotion recognition in the wild from videos using images. In: ICM: Proceedings of the 18th ACM International Conference on Multimodal Interaction, ACM
Barros P et al (2017) Emotion-modulated attention improves expression recognition: a deep learning model. Neurocomputing 253:104–114
Barthomeuf L, Droit-Volet S, Rousset S (2012) How emotions expressed by adults’ faces affect the desire to eat liked and disliked foods in children compared to adults. Br J Dev Psychol 30:253–266
Bartlett M, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression :machine learning and application to sponta- neous behavior. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2:568–573
Belhumeur P, Hespanha J, Kriegman D (1997) Eigenfaces vs. fisher face: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720
Boughrara H et al (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75:709–731
Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial motion recognition. IEEE Trans Affect Comput 5:418–431
Deng HB, Jin LW, Zhen LX, Huang JC (2005) A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int J Inf Technol 11(11):86–96
Ding W, Xu M, Huang D, Lin W, Dong M, Yu X, Li H (2016) Audio and face video emotion recognition in the wild using deep neural networks and small datasets. In: ICMI 2016: proceedings of the 18th ACM international conference on multimodal interaction, ACM
Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989
Ekman P, Friesen W (1978) Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto: Consulting Psychologists Press
Elaiwat S, Bennamoun M, Boussaid F (2016) A spatio-temporal RBM-based model for facial expression recognition. Int J of Pattern Recognition 49:152–161
Feng XY, Hadid A, Pietikainen M (2004) A coarse-to-fine classification scheme for facial expression recognition. In: The First International Conference on Image Analysis and Recognition. 668–675
Feng XY, Hadid A, Pietikainen M (2005) Facial expression recognition with local binary patterns and linear programming. Patt Recog Image Anal 15(2):546–548
Gaidhane VH, Yogesh V, Singh V (2016) Emotion recognition using eigenvalues and Levenberg–Marquardt algorithm-based classifier. Ind Acad Sci 41(4):415–423
Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(77):14–7734
Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10(3):443–458
Ghimire D, Lee J, Li Z-N, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Int J Multimed Tools Appl 76:7921–7946
Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76:7803–7821
Gu W, Venkatesh Y, Xiang C (2010) A novel application of self-organizing network for facial expression recognition from radial encoded contours. Soft Comput Fusion Found Methodol Appl 14(2):113–122
Gu W et al. (2012) Facial expression recognition using gradial encoding of local Gabor features and classifier synthesis Pattern Recogn 45:80–91
Gupta S.K, Agrwal S, Meena Y. K, Nain N (2011) A hybrid method of feature extraction for facial expression recognition. In: 7th international conference on signal image technology & internet-based systems. 422–425
Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12
Happy SL, Patnaik P, Routray A, Guha R (2016) The Indian spontaneous expression database for emotion recognition. in IEEE Transactions on Affective Computing https://doi.org/10.1109/TAFFC.2015.2498174
He LH, Zou C.R, Zhao L, Hu D (2005) An enhanced LBP feature based on facial expression recognition. In: IEEE Engineering in Medicine and Biology 27th Annual Conference. 3300–3303
Heisele B, Ho P, Wu J, Poggio T (2003) Face recognition: component-based versus global approaches. Comput Vis Image Underst 91:6–21
Hernandez-Matamoros A (2016) Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach. Knowl-Based Syst 110:1–14
Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2015) Effective semantic features for facial expressions recognition using svm. Int J Multimed Tools Appl:1–20
Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Int J Multimed Tools Appl 73(1):309–326
Huang Y, Lu H (2016) Deep learning driven hypergraph representation for image-based emotion recognition. In: proceedings of the 18th ACM international conference on multimodal interaction, ACM
Ilbeygi M, Hosseini HS (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146
Jain S, Hu C, Aggarwal C K (2011) Facial expression recognition with temporal modeling of shapes. In: IEEE International Conference on Computer Vision Workshops (ICCVWorkshops), Barcelona. 1642–1649
Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. 21st Korea-Japan joint workshop on Frontiers of Comput. Vision (FCV). 1–4
Kazmi SB, Qurat-ul-Ain JMA (2012) Wavelet-based facial expression recognition using a bank of support vector machines. Soft Comput 16(3):369–379
Kharat GU, Dudul SV (2009) Emotion recognition from facial expression using neural networks. In: Human–computer systems interaction, AISC, 60, 207–219
Kim D et al (2017) Multi-view face recognition from single RGBD models of the faces. Comput Vis Image Underst 160:114–132
Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187
Kotsia I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1052–1067
Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19:743–756
Li J, Lam EY (2015) Facial expression recognition using deep neural networks. In Proc. IEEE Int. Conf. Imaging Syst. Techn. (IST). 1–6
Li Z et al (2015) Unsupervised feature selection via nonnegative spectral analysis and redundancy control. IEEE Trans Image Process 24(12):5343–5355
Li Z et al. (2018) Robust Structured Nonnegative Matrix Factorization for Image Representation IEEE transactions on neural networks and learning systems, vol. 29(5)
Liu C, Wechsler H (2003) Independent component analysis of Gabor features for face recognition. IEEE Trans Neural Netw 14(4):919–928
Liu WF, Yi SJ, Wang YJ (2009) Automatic facial expression recognition based on local binary patterns of local areas. In: WASE international conference on information. engineering:197–200
Liu M, Shan S, Wang R, and Chen X (2014) Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Columbus, OH, USA, 2014. 1749–1756
Ye Liu et al. (2016a) Action2Activity: Recognizing Complex Activities from Sensor Data
Liu Y et al., (2016a) Fortune teller: predicting your career path, Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI-16)
Liu Y et al (2016b) From action to activity: sensor- based activity recognition. Neurocomputing 18:1108–1115
Liu Y et al., (2016b) Urban water quality prediction based on multi-task multi-view learning. Proceedings of the twenty-fifth international joint conference on artificial intelligence (IJCAI-16)
Luo R, Huang C, Lin P (2011) Alignment and tracking of facial features with component-based active appearance models and optical flow. In: International Conference on Advanced Intelligent Mechatronics(AIM), IEEE, pp. 1058–1063
Luo Y, Wu C-M, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Int J Light Electron Opt 124(17):2767–2770
Lyons M, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21:1357–1362
Major Torres JM, Stepanov EA (2017) Enhanced face/audio emotion recognition: video and instance level classification using ConvNets and restricted Boltzmann Machines. In: Proceedings of the International Conference on Web Intelligence, ACM
Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recogn 47:1282–1293
Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56
Mollahosseini A, Chan D and Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Lake Placid, NY, USA. 1–10
Moore S, Bowden R (2009) The effects of pose on facial expression recognition. In A. Cavallaro, S. Prince, D. Alexander (Eds.), In: Proceedings of the British machine conference, BMVA Press. 79.1–79.11
Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput VisImage Underst 115:541–558
Nielsen JA, Zielinski BA, Ferguson MA, Lainhart JE, Anderson JS (2013) An evaluation of the left-brain vs. right brain hypothesis with resting state functional connectivity magnetic resonance imaging. PLOS One. Cognitive Neuroscience-Connectomics. DOI: https://doi.org/10.1371/journal.pone.0071275 Featured in PLOS Collections
Padgett C, Cottrell G (1996) Representing face image for emotion classification. Adv Neural Inf Proces Syst 9:894–900
Perez-Gaspar et al (2016) Multimodal emotion recognition with evolutionary computation for human-robot interaction. Expert Syst Appl 66:42–61
Poursaberi A, Noubari HA, Gavrilova M, Yanushkevich SN (2012) Gauss–Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J Image Video Proc. https://doi.org/10.1186/1687-5281-2012-17
Presti LL et al (2017) Boosting Hankel matrices for face emotion recognition and pain detection. Comput Vis Image Underst 156:19–33
Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92
Rosenblum M, Yacoob Y, Davis L (1996) Human expression recognition from motion using a radial basis function network architecture. IEEE Trans Neural Netw 7(5):1121–1138
Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369
Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact:1–13
Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing. 370–373
Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816
Shih FY, Chuang CF, Wang PSP (2008) Performance comparisons of facial expression recognition in JAFFE database. Int J Patt Recog 22(3):445–459
Siddiqi MH, Ali R, Khan AM, Park Y-T, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398
Sohail ASM, Bhattacharya P (2011) Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int J Pattern Recognit Artif Intell 25(06):835–862
Song M, Tao D, Liu Z, Li X, Zhou M (2010) Image ratio features for facial expression recognition application. IEEE Trans Syst, Man, Cybernet-Part B: Cybernet 40(3):779–788
Tarnowski P et al. (2017) Emotion recognition using facial expressions. In: International conference on computational science - ICCS 2017, 12–14 June 2017, Zurich, Switzerland
Tsai H-H, Chang Y-C (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Int J Soft Comput. https://doi.org/10.1007/s00500-017-2634-3
Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86
Uddin MZ, Lee JJ, Kim T-S (2009a) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224
Vadivel A, Shanthi P, Shaila SG (2015) Estimating Emotions Using Geometric Features from Facial Expressions. Encyclopedia of Information Science and Technology, Third Edition. 8
Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Trans Syst, Man, Cybernet-PART B: Cybernet 42(1):28–43
Valstar M, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR Workshops, pp. 76
Vo DM, Le TH (2016) Deep generic features and SVM for facial expression recognition. 3rd National Foundation for Science and Technology Development Conf. Inf. And Comput. Sci (NICS):80–84
Wang Z, Ruan Q (2010) Facial expression based orthogonal local fisher discriminant analysis. In: Proc ICSP 2010:1358–1361
Wang H, Huang H, Makedon F (2014) Emotion detection via discriminant Laplacian embedding. Univ Access Inf Soc 13:23–31
Whitehill J, Bartlett M, Movellan J (2008) Automatic facial expression recognition for intelligent tutoring systems. Proc of Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA, In
Bing-Fei Wu, and Chun-Hsien Lin (2017) Adaptive Feature Mapping for Customizing Deep Learning Based Facial Expression Recognition Model. IEEE access 2017
Wu T, Bartlett M, Movellan J.R (2010) Facial expression recognition using Gabor motion energy filters. In: proceedings of IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW’10), pp. 42–47
Xiang C, Fan XA, Lee TH (2006) Face recognition using recursive fisher linear discriminant. IEEE Trans Image Process 15(8):2097–2105
Xiaorong P, Fan K, Chen X, Ji L, Zhou Z (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180
Xie X, Lam K-M (2009) Facial expression recognition based on shape and texture. Pattern Recogn 42:1003–1011
Yang Q, Wooldridge M (2015) IJCAI'15 Proceedings of the 24th International Conference on Artificial Intelligence. AAAI Press, Buenos Aires, pp 1617–1623
Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508
Zafeiriou S, Pita I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10(8):1528–1540
Thiago H.H. Zavaschi, Alceu S. Britto Jr., Luiz E.S. Oliveira, Alessandro L. Koerich (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl, 40, pp. 646–655
Zhang Z (1999) Feature-based facial expression recognition: sensitivity analysis and experiments with a multi layer perceptron. Int J Pattern Recognit Artif Intell 13:893–911
Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229
Zhang S, Zhao BL (2012) Facial expression based on local binary patterns and local fisher discriminant analysis. WSEAS Trans Sign Proc 8(1):21–31
Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry- based and Gabor-wavelets – based facial expression recognition using multi-layer perceptron. In: proceedings of 3rd international conference on automatic face and gesture recognition, pp. 454–459
Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761
Zhang L, Jiang M, Farid D, Hossain AM (2013) Intelligent facial emotion recognition and semantic- based topic detection for a humanoid robot. Expert Syst Appl 40(13):5160–5168
Zhang L, Mistry K, Jiang M, Neoh SC, Hossain MA (2015) Adaptive facial point detection and emotion recognition for a humanoid robot. Comput Vis Image Underst 140:93–114
Zhang Y-D et al., (2016) Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation, SPECIAL SECTION ON EMOTION-AWARE MOBILE COMPUTING, IEEE Access
Zhao GY, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928
Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors:9573–9588
Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) learning active facial patches for expression analysis. In IEEE conference on computer vision and pattern recognition (CVPR), Providence, RI, USA
Zou J, Ji Q, Nagy G (2007) A comparative study of local matching approach for face recognition. IEEE Trans Image Process 16(10):2617–2628
Further Reading
Khana RA, Meyer A, Konik H, Bouaka S (2013) Framework for reliable, real time facial expression recognition for low resolution images. Pattern Recogn Lett 34:1159–1168
Kobayashi H, Hara F (1997) Facial interaction between animated 3dface robot and human beings. In: IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, IEEE, 4, 3732–3737
Acknowledgements
I would like to thank to the Dr. A. Vadivel, Associate Professor, Depertment of Computer Science and Engineering, SRM University, Amaravathi, Andrapradesh, for his moral support for the manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Vishnu Priya, R. Emotion recognition from geometric fuzzy membership functions. Multimed Tools Appl 78, 17847–17878 (2019). https://doi.org/10.1007/s11042-018-6954-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-6954-9