Emotion recognition from geometric fuzzy membership functions

  • R. Vishnu PriyaEmail author


The posterity challenging task in the Machine Intelligence field is to design a smarter system to identify the human emotions. Facial Emotion Recognition (FER) is a significant visual based tool to construct a smarter system that can recognize human emotions. The existing methods in FER are based on Action Units (AU), appearance and geometrical parameters. Nearly 7000 different combinations of AUs are used in AU to discriminate the emotions, which can be very expensive and increase processing time. Generalize appearance features across the universe is another challenging task. In this paper, a novel geometrical fuzzy based approach is presented to accurately recognize the emotions. The four corner features from eyes and mouth regions are extracted without considering reference face. The extracted features are used to define the quadrilateral shape that failed to match with the shapes in geometry. The degree of impreciseness exists in the quadrilateral is measured by the proposed Mixed Quadratic Shape Model (MQSM) using fuzzy membership functions. Finally, twelve fuzzy features are extracted from the membership functions and used by the classifier for validation. The CK, JAFFE and ISED datasets are used in the experiment to evaluate the performance of the MSQM. It is observed the proposed method performed better than the contemporary methods using twelve fuzzy features without reference image.


Geometric features Primitive shape Emotion recognition Fuzzy membership function Fuzzy-shape 



I would like to thank to the Dr. A. Vadivel, Associate Professor, Depertment of Computer Science and Engineering, SRM University, Amaravathi, Andrapradesh, for his moral support for the manuscript.


  1. 1.
    Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Int J of Multimed Tools Appl 76:1073–1099Google Scholar
  2. 2.
    Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–118Google Scholar
  3. 3.
    Anderson K, McOwan PW (2004) Robust real-time face tracker for use in cluttered environments. Comput Vis Image Underst 95(2):184–200Google Scholar
  4. 4.
    Anderson K, McOwan PW (2006) A real-time automated system for the recognition of human facial expressions. IEEE transactions on systems, man. And Cybernetics-Part B: Cybernetics 36(1):96–105Google Scholar
  5. 5.
    Bargal SA, Barsoum E, Ferrer CC, Zhang C (2016) Emotion recognition in the wild from videos using images. In: ICM: Proceedings of the 18th ACM International Conference on Multimodal Interaction, ACMGoogle Scholar
  6. 6.
    Barros P et al (2017) Emotion-modulated attention improves expression recognition: a deep learning model. Neurocomputing 253:104–114Google Scholar
  7. 7.
    Barthomeuf L, Droit-Volet S, Rousset S (2012) How emotions expressed by adults’ faces affect the desire to eat liked and disliked foods in children compared to adults. Br J Dev Psychol 30:253–266Google Scholar
  8. 8.
    Bartlett M, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression :machine learning and application to sponta- neous behavior. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2:568–573Google Scholar
  9. 9.
    Belhumeur P, Hespanha J, Kriegman D (1997) Eigenfaces vs. fisher face: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720Google Scholar
  10. 10.
    Boughrara H et al (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75:709–731Google Scholar
  11. 11.
    Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial motion recognition. IEEE Trans Affect Comput 5:418–431Google Scholar
  12. 12.
    Deng HB, Jin LW, Zhen LX, Huang JC (2005) A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int J Inf Technol 11(11):86–96Google Scholar
  13. 13.
    Ding W, Xu M, Huang D, Lin W, Dong M, Yu X, Li H (2016) Audio and face video emotion recognition in the wild using deep neural networks and small datasets. In: ICMI 2016: proceedings of the 18th ACM international conference on multimodal interaction, ACMGoogle Scholar
  14. 14.
    Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989Google Scholar
  15. 15.
    Ekman P, Friesen W (1978) Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto: Consulting Psychologists PressGoogle Scholar
  16. 16.
    Elaiwat S, Bennamoun M, Boussaid F (2016) A spatio-temporal RBM-based model for facial expression recognition. Int J of Pattern Recognition 49:152–161Google Scholar
  17. 17.
    Feng XY, Hadid A, Pietikainen M (2004) A coarse-to-fine classification scheme for facial expression recognition. In: The First International Conference on Image Analysis and Recognition. 668–675Google Scholar
  18. 18.
    Feng XY, Hadid A, Pietikainen M (2005) Facial expression recognition with local binary patterns and linear programming. Patt Recog Image Anal 15(2):546–548Google Scholar
  19. 19.
    Gaidhane VH, Yogesh V, Singh V (2016) Emotion recognition using eigenvalues and Levenberg–Marquardt algorithm-based classifier. Ind Acad Sci 41(4):415–423MathSciNetzbMATHGoogle Scholar
  20. 20.
    Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(77):14–7734Google Scholar
  21. 21.
    Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10(3):443–458Google Scholar
  22. 22.
    Ghimire D, Lee J, Li Z-N, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Int J Multimed Tools Appl 76:7921–7946Google Scholar
  23. 23.
    Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76:7803–7821Google Scholar
  24. 24.
    Gu W, Venkatesh Y, Xiang C (2010) A novel application of self-organizing network for facial expression recognition from radial encoded contours. Soft Comput Fusion Found Methodol Appl 14(2):113–122Google Scholar
  25. 25.
    Gu W et al. (2012) Facial expression recognition using gradial encoding of local Gabor features and classifier synthesis Pattern Recogn 45:80–91Google Scholar
  26. 26.
    Gupta S.K, Agrwal S, Meena Y. K, Nain N (2011) A hybrid method of feature extraction for facial expression recognition. In: 7th international conference on signal image technology & internet-based systems. 422–425Google Scholar
  27. 27.
    Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12Google Scholar
  28. 28.
    Happy SL, Patnaik P, Routray A, Guha R (2016) The Indian spontaneous expression database for emotion recognition. in IEEE Transactions on Affective Computing
  29. 29.
    He LH, Zou C.R, Zhao L, Hu D (2005) An enhanced LBP feature based on facial expression recognition. In: IEEE Engineering in Medicine and Biology 27th Annual Conference. 3300–3303Google Scholar
  30. 30.
    Heisele B, Ho P, Wu J, Poggio T (2003) Face recognition: component-based versus global approaches. Comput Vis Image Underst 91:6–21Google Scholar
  31. 31.
    Hernandez-Matamoros A (2016) Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach. Knowl-Based Syst 110:1–14Google Scholar
  32. 32.
    Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2015) Effective semantic features for facial expressions recognition using svm. Int J Multimed Tools Appl:1–20Google Scholar
  33. 33.
    Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Int J Multimed Tools Appl 73(1):309–326Google Scholar
  34. 34.
  35. 35.
  36. 36.
    Huang Y, Lu H (2016) Deep learning driven hypergraph representation for image-based emotion recognition. In: proceedings of the 18th ACM international conference on multimodal interaction, ACMGoogle Scholar
  37. 37.
    Ilbeygi M, Hosseini HS (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146Google Scholar
  38. 38.
    Jain S, Hu C, Aggarwal C K (2011) Facial expression recognition with temporal modeling of shapes. In: IEEE International Conference on Computer Vision Workshops (ICCVWorkshops), Barcelona. 1642–1649Google Scholar
  39. 39.
    Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. 21st Korea-Japan joint workshop on Frontiers of Comput. Vision (FCV). 1–4Google Scholar
  40. 40.
    Kazmi SB, Qurat-ul-Ain JMA (2012) Wavelet-based facial expression recognition using a bank of support vector machines. Soft Comput 16(3):369–379Google Scholar
  41. 41.
    Kharat GU, Dudul SV (2009) Emotion recognition from facial expression using neural networks. In: Human–computer systems interaction, AISC, 60, 207–219Google Scholar
  42. 42.
    Kim D et al (2017) Multi-view face recognition from single RGBD models of the faces. Comput Vis Image Underst 160:114–132Google Scholar
  43. 43.
    Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187MathSciNetGoogle Scholar
  44. 44.
    Kotsia I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1052–1067Google Scholar
  45. 45.
    Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19:743–756Google Scholar
  46. 46.
    Li J, Lam EY (2015) Facial expression recognition using deep neural networks. In Proc. IEEE Int. Conf. Imaging Syst. Techn. (IST). 1–6Google Scholar
  47. 47.
    Li Z et al (2015) Unsupervised feature selection via nonnegative spectral analysis and redundancy control. IEEE Trans Image Process 24(12):5343–5355Google Scholar
  48. 48.
    Li Z et al. (2018) Robust Structured Nonnegative Matrix Factorization for Image Representation IEEE transactions on neural networks and learning systems, vol. 29(5)Google Scholar
  49. 49.
    Liu C, Wechsler H (2003) Independent component analysis of Gabor features for face recognition. IEEE Trans Neural Netw 14(4):919–928Google Scholar
  50. 50.
    Liu WF, Yi SJ, Wang YJ (2009) Automatic facial expression recognition based on local binary patterns of local areas. In: WASE international conference on information. engineering:197–200Google Scholar
  51. 51.
    Liu M, Shan S, Wang R, and Chen X (2014) Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Columbus, OH, USA, 2014. 1749–1756Google Scholar
  52. 52.
    Ye Liu et al. (2016a) Action2Activity: Recognizing Complex Activities from Sensor DataGoogle Scholar
  53. 53.
    Liu Y et al., (2016a) Fortune teller: predicting your career path, Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI-16)Google Scholar
  54. 54.
    Liu Y et al (2016b) From action to activity: sensor- based activity recognition. Neurocomputing 18:1108–1115Google Scholar
  55. 55.
    Liu Y et al., (2016b) Urban water quality prediction based on multi-task multi-view learning. Proceedings of the twenty-fifth international joint conference on artificial intelligence (IJCAI-16)Google Scholar
  56. 56.
    Luo R, Huang C, Lin P (2011) Alignment and tracking of facial features with component-based active appearance models and optical flow. In: International Conference on Advanced Intelligent Mechatronics(AIM), IEEE, pp. 1058–1063Google Scholar
  57. 57.
    Luo Y, Wu C-M, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Int J Light Electron Opt 124(17):2767–2770Google Scholar
  58. 58.
    Lyons M, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21:1357–1362Google Scholar
  59. 59.
    Major Torres JM, Stepanov EA (2017) Enhanced face/audio emotion recognition: video and instance level classification using ConvNets and restricted Boltzmann Machines. In: Proceedings of the International Conference on Web Intelligence, ACM
  60. 60.
    Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recogn 47:1282–1293Google Scholar
  61. 61.
    Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56Google Scholar
  62. 62.
    Mollahosseini A, Chan D and Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Lake Placid, NY, USA. 1–10Google Scholar
  63. 63.
    Moore S, Bowden R (2009) The effects of pose on facial expression recognition. In A. Cavallaro, S. Prince, D. Alexander (Eds.), In: Proceedings of the British machine conference, BMVA Press. 79.1–79.11Google Scholar
  64. 64.
    Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput VisImage Underst 115:541–558Google Scholar
  65. 65.
    Nielsen JA, Zielinski BA, Ferguson MA, Lainhart JE, Anderson JS (2013) An evaluation of the left-brain vs. right brain hypothesis with resting state functional connectivity magnetic resonance imaging. PLOS One. Cognitive Neuroscience-Connectomics. DOI: Featured in PLOS Collections
  66. 66.
    Padgett C, Cottrell G (1996) Representing face image for emotion classification. Adv Neural Inf Proces Syst 9:894–900Google Scholar
  67. 67.
    Perez-Gaspar et al (2016) Multimodal emotion recognition with evolutionary computation for human-robot interaction. Expert Syst Appl 66:42–61Google Scholar
  68. 68.
    Poursaberi A, Noubari HA, Gavrilova M, Yanushkevich SN (2012) Gauss–Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J Image Video Proc.
  69. 69.
    Presti LL et al (2017) Boosting Hankel matrices for face emotion recognition and pain detection. Comput Vis Image Underst 156:19–33Google Scholar
  70. 70.
    Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92Google Scholar
  71. 71.
    Rosenblum M, Yacoob Y, Davis L (1996) Human expression recognition from motion using a radial basis function network architecture. IEEE Trans Neural Netw 7(5):1121–1138Google Scholar
  72. 72.
    Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369Google Scholar
  73. 73.
    Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact:1–13Google Scholar
  74. 74.
    Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing. 370–373Google Scholar
  75. 75.
    Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816Google Scholar
  76. 76.
    Shih FY, Chuang CF, Wang PSP (2008) Performance comparisons of facial expression recognition in JAFFE database. Int J Patt Recog 22(3):445–459Google Scholar
  77. 77.
    Siddiqi MH, Ali R, Khan AM, Park Y-T, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398MathSciNetGoogle Scholar
  78. 78.
    Sohail ASM, Bhattacharya P (2011) Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int J Pattern Recognit Artif Intell 25(06):835–862MathSciNetGoogle Scholar
  79. 79.
    Song M, Tao D, Liu Z, Li X, Zhou M (2010) Image ratio features for facial expression recognition application. IEEE Trans Syst, Man, Cybernet-Part B: Cybernet 40(3):779–788Google Scholar
  80. 80.
    Tarnowski P et al. (2017) Emotion recognition using facial expressions. In: International conference on computational science - ICCS 2017, 12–14 June 2017, Zurich, SwitzerlandGoogle Scholar
  81. 81.
    Tsai H-H, Chang Y-C (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Int J Soft Comput.
  82. 82.
    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86Google Scholar
  83. 83.
    Uddin MZ, Lee JJ, Kim T-S (2009a) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224Google Scholar
  84. 84.
    Vadivel A, Shanthi P, Shaila SG (2015) Estimating Emotions Using Geometric Features from Facial Expressions. Encyclopedia of Information Science and Technology, Third Edition. 8Google Scholar
  85. 85.
    Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Trans Syst, Man, Cybernet-PART B: Cybernet 42(1):28–43Google Scholar
  86. 86.
    Valstar M, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR Workshops, pp. 76Google Scholar
  87. 87.
    Vo DM, Le TH (2016) Deep generic features and SVM for facial expression recognition. 3rd National Foundation for Science and Technology Development Conf. Inf. And Comput. Sci (NICS):80–84Google Scholar
  88. 88.
    Wang Z, Ruan Q (2010) Facial expression based orthogonal local fisher discriminant analysis. In: Proc ICSP 2010:1358–1361Google Scholar
  89. 89.
    Wang H, Huang H, Makedon F (2014) Emotion detection via discriminant Laplacian embedding. Univ Access Inf Soc 13:23–31Google Scholar
  90. 90.
    Whitehill J, Bartlett M, Movellan J (2008) Automatic facial expression recognition for intelligent tutoring systems. Proc of Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA, InGoogle Scholar
  91. 91.
    Bing-Fei Wu, and Chun-Hsien Lin (2017) Adaptive Feature Mapping for Customizing Deep Learning Based Facial Expression Recognition Model. IEEE access 2017Google Scholar
  92. 92.
    Wu T, Bartlett M, Movellan J.R (2010) Facial expression recognition using Gabor motion energy filters. In: proceedings of IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW’10), pp. 42–47Google Scholar
  93. 93.
  94. 94.
    Xiang C, Fan XA, Lee TH (2006) Face recognition using recursive fisher linear discriminant. IEEE Trans Image Process 15(8):2097–2105Google Scholar
  95. 95.
    Xiaorong P, Fan K, Chen X, Ji L, Zhou Z (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180Google Scholar
  96. 96.
    Xie X, Lam K-M (2009) Facial expression recognition based on shape and texture. Pattern Recogn 42:1003–1011Google Scholar
  97. 97.
    Yang Q, Wooldridge M (2015) IJCAI'15 Proceedings of the 24th International Conference on Artificial Intelligence. AAAI Press, Buenos Aires, pp 1617–1623Google Scholar
  98. 98.
    Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508Google Scholar
  99. 99.
    Zafeiriou S, Pita I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10(8):1528–1540Google Scholar
  100. 100.
    Thiago H.H. Zavaschi, Alceu S. Britto Jr., Luiz E.S. Oliveira, Alessandro L. Koerich (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl, 40, pp. 646–655Google Scholar
  101. 101.
    Zhang Z (1999) Feature-based facial expression recognition: sensitivity analysis and experiments with a multi layer perceptron. Int J Pattern Recognit Artif Intell 13:893–911Google Scholar
  102. 102.
    Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229Google Scholar
  103. 103.
    Zhang S, Zhao BL (2012) Facial expression based on local binary patterns and local fisher discriminant analysis. WSEAS Trans Sign Proc 8(1):21–31Google Scholar
  104. 104.
    Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry- based and Gabor-wavelets – based facial expression recognition using multi-layer perceptron. In: proceedings of 3rd international conference on automatic face and gesture recognition, pp. 454–459Google Scholar
  105. 105.
    Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761Google Scholar
  106. 106.
    Zhang L, Jiang M, Farid D, Hossain AM (2013) Intelligent facial emotion recognition and semantic- based topic detection for a humanoid robot. Expert Syst Appl 40(13):5160–5168Google Scholar
  107. 107.
    Zhang L, Mistry K, Jiang M, Neoh SC, Hossain MA (2015) Adaptive facial point detection and emotion recognition for a humanoid robot. Comput Vis Image Underst 140:93–114Google Scholar
  108. 108.
    Zhang Y-D et al., (2016) Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation, SPECIAL SECTION ON EMOTION-AWARE MOBILE COMPUTING, IEEE AccessGoogle Scholar
  109. 109.
    Zhao GY, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928Google Scholar
  110. 110.
    Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors:9573–9588Google Scholar
  111. 111.
    Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) learning active facial patches for expression analysis. In IEEE conference on computer vision and pattern recognition (CVPR), Providence, RI, USAGoogle Scholar
  112. 112.
    Zou J, Ji Q, Nagy G (2007) A comparative study of local matching approach for face recognition. IEEE Trans Image Process 16(10):2617–2628MathSciNetGoogle Scholar

Further Reading

  1. 113.
    Khana RA, Meyer A, Konik H, Bouaka S (2013) Framework for reliable, real time facial expression recognition for low resolution images. Pattern Recogn Lett 34:1159–1168Google Scholar
  2. 114.
    Kobayashi H, Hara F (1997) Facial interaction between animated 3dface robot and human beings. In: IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, IEEE, 4, 3732–3737Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ApplicationsMaulana Azad National Institute of TechnologyBhopalIndia

Personalised recommendations