Skip to main content
Log in

Emotion recognition from geometric fuzzy membership functions

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The posterity challenging task in the Machine Intelligence field is to design a smarter system to identify the human emotions. Facial Emotion Recognition (FER) is a significant visual based tool to construct a smarter system that can recognize human emotions. The existing methods in FER are based on Action Units (AU), appearance and geometrical parameters. Nearly 7000 different combinations of AUs are used in AU to discriminate the emotions, which can be very expensive and increase processing time. Generalize appearance features across the universe is another challenging task. In this paper, a novel geometrical fuzzy based approach is presented to accurately recognize the emotions. The four corner features from eyes and mouth regions are extracted without considering reference face. The extracted features are used to define the quadrilateral shape that failed to match with the shapes in geometry. The degree of impreciseness exists in the quadrilateral is measured by the proposed Mixed Quadratic Shape Model (MQSM) using fuzzy membership functions. Finally, twelve fuzzy features are extracted from the membership functions and used by the classifier for validation. The CK, JAFFE and ISED datasets are used in the experiment to evaluate the performance of the MSQM. It is observed the proposed method performed better than the contemporary methods using twelve fuzzy features without reference image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3.
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Int J of Multimed Tools Appl 76:1073–1099

    Article  Google Scholar 

  2. Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–118

    Article  Google Scholar 

  3. Anderson K, McOwan PW (2004) Robust real-time face tracker for use in cluttered environments. Comput Vis Image Underst 95(2):184–200

    Article  Google Scholar 

  4. Anderson K, McOwan PW (2006) A real-time automated system for the recognition of human facial expressions. IEEE transactions on systems, man. And Cybernetics-Part B: Cybernetics 36(1):96–105

    Article  Google Scholar 

  5. Bargal SA, Barsoum E, Ferrer CC, Zhang C (2016) Emotion recognition in the wild from videos using images. In: ICM: Proceedings of the 18th ACM International Conference on Multimodal Interaction, ACM

  6. Barros P et al (2017) Emotion-modulated attention improves expression recognition: a deep learning model. Neurocomputing 253:104–114

    Article  Google Scholar 

  7. Barthomeuf L, Droit-Volet S, Rousset S (2012) How emotions expressed by adults’ faces affect the desire to eat liked and disliked foods in children compared to adults. Br J Dev Psychol 30:253–266

    Article  Google Scholar 

  8. Bartlett M, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression :machine learning and application to sponta- neous behavior. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2:568–573

    Google Scholar 

  9. Belhumeur P, Hespanha J, Kriegman D (1997) Eigenfaces vs. fisher face: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  10. Boughrara H et al (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75:709–731

    Article  Google Scholar 

  11. Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial motion recognition. IEEE Trans Affect Comput 5:418–431

    Article  Google Scholar 

  12. Deng HB, Jin LW, Zhen LX, Huang JC (2005) A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int J Inf Technol 11(11):86–96

    Google Scholar 

  13. Ding W, Xu M, Huang D, Lin W, Dong M, Yu X, Li H (2016) Audio and face video emotion recognition in the wild using deep neural networks and small datasets. In: ICMI 2016: proceedings of the 18th ACM international conference on multimodal interaction, ACM

  14. Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989

    Article  Google Scholar 

  15. Ekman P, Friesen W (1978) Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto: Consulting Psychologists Press

  16. Elaiwat S, Bennamoun M, Boussaid F (2016) A spatio-temporal RBM-based model for facial expression recognition. Int J of Pattern Recognition 49:152–161

    Article  Google Scholar 

  17. Feng XY, Hadid A, Pietikainen M (2004) A coarse-to-fine classification scheme for facial expression recognition. In: The First International Conference on Image Analysis and Recognition. 668–675

  18. Feng XY, Hadid A, Pietikainen M (2005) Facial expression recognition with local binary patterns and linear programming. Patt Recog Image Anal 15(2):546–548

    Google Scholar 

  19. Gaidhane VH, Yogesh V, Singh V (2016) Emotion recognition using eigenvalues and Levenberg–Marquardt algorithm-based classifier. Ind Acad Sci 41(4):415–423

    MathSciNet  MATH  Google Scholar 

  20. Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(77):14–7734

    Google Scholar 

  21. Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10(3):443–458

    Article  Google Scholar 

  22. Ghimire D, Lee J, Li Z-N, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Int J Multimed Tools Appl 76:7921–7946

    Article  Google Scholar 

  23. Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76:7803–7821

    Article  Google Scholar 

  24. Gu W, Venkatesh Y, Xiang C (2010) A novel application of self-organizing network for facial expression recognition from radial encoded contours. Soft Comput Fusion Found Methodol Appl 14(2):113–122

    Google Scholar 

  25. Gu W et al. (2012) Facial expression recognition using gradial encoding of local Gabor features and classifier synthesis Pattern Recogn 45:80–91

  26. Gupta S.K, Agrwal S, Meena Y. K, Nain N (2011) A hybrid method of feature extraction for facial expression recognition. In: 7th international conference on signal image technology & internet-based systems. 422–425

  27. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  28. Happy SL, Patnaik P, Routray A, Guha R (2016) The Indian spontaneous expression database for emotion recognition. in IEEE Transactions on Affective Computing https://doi.org/10.1109/TAFFC.2015.2498174

  29. He LH, Zou C.R, Zhao L, Hu D (2005) An enhanced LBP feature based on facial expression recognition. In: IEEE Engineering in Medicine and Biology 27th Annual Conference. 3300–3303

  30. Heisele B, Ho P, Wu J, Poggio T (2003) Face recognition: component-based versus global approaches. Comput Vis Image Underst 91:6–21

    Article  Google Scholar 

  31. Hernandez-Matamoros A (2016) Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach. Knowl-Based Syst 110:1–14

    Article  Google Scholar 

  32. Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2015) Effective semantic features for facial expressions recognition using svm. Int J Multimed Tools Appl:1–20

  33. Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Int J Multimed Tools Appl 73(1):309–326

    Article  Google Scholar 

  34. http://www.kasrl.org/jaffe.html

  35. https://sites.google.com/site/iseddatabase/

  36. Huang Y, Lu H (2016) Deep learning driven hypergraph representation for image-based emotion recognition. In: proceedings of the 18th ACM international conference on multimodal interaction, ACM

  37. Ilbeygi M, Hosseini HS (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146

    Article  Google Scholar 

  38. Jain S, Hu C, Aggarwal C K (2011) Facial expression recognition with temporal modeling of shapes. In: IEEE International Conference on Computer Vision Workshops (ICCVWorkshops), Barcelona. 1642–1649

  39. Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. 21st Korea-Japan joint workshop on Frontiers of Comput. Vision (FCV). 1–4

  40. Kazmi SB, Qurat-ul-Ain JMA (2012) Wavelet-based facial expression recognition using a bank of support vector machines. Soft Comput 16(3):369–379

    Article  Google Scholar 

  41. Kharat GU, Dudul SV (2009) Emotion recognition from facial expression using neural networks. In: Human–computer systems interaction, AISC, 60, 207–219

  42. Kim D et al (2017) Multi-view face recognition from single RGBD models of the faces. Comput Vis Image Underst 160:114–132

    Article  Google Scholar 

  43. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  44. Kotsia I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1052–1067

    Article  Google Scholar 

  45. Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19:743–756

    Article  Google Scholar 

  46. Li J, Lam EY (2015) Facial expression recognition using deep neural networks. In Proc. IEEE Int. Conf. Imaging Syst. Techn. (IST). 1–6

  47. Li Z et al (2015) Unsupervised feature selection via nonnegative spectral analysis and redundancy control. IEEE Trans Image Process 24(12):5343–5355

  48. Li Z et al. (2018) Robust Structured Nonnegative Matrix Factorization for Image Representation IEEE transactions on neural networks and learning systems, vol. 29(5)

  49. Liu C, Wechsler H (2003) Independent component analysis of Gabor features for face recognition. IEEE Trans Neural Netw 14(4):919–928

    Article  Google Scholar 

  50. Liu WF, Yi SJ, Wang YJ (2009) Automatic facial expression recognition based on local binary patterns of local areas. In: WASE international conference on information. engineering:197–200

  51. Liu M, Shan S, Wang R, and Chen X (2014) Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Columbus, OH, USA, 2014. 1749–1756

  52. Ye Liu et al. (2016a) Action2Activity: Recognizing Complex Activities from Sensor Data

  53. Liu Y et al., (2016a) Fortune teller: predicting your career path, Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI-16)

  54. Liu Y et al (2016b) From action to activity: sensor- based activity recognition. Neurocomputing 18:1108–1115

    Google Scholar 

  55. Liu Y et al., (2016b) Urban water quality prediction based on multi-task multi-view learning. Proceedings of the twenty-fifth international joint conference on artificial intelligence (IJCAI-16)

  56. Luo R, Huang C, Lin P (2011) Alignment and tracking of facial features with component-based active appearance models and optical flow. In: International Conference on Advanced Intelligent Mechatronics(AIM), IEEE, pp. 1058–1063

  57. Luo Y, Wu C-M, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Int J Light Electron Opt 124(17):2767–2770

    Article  Google Scholar 

  58. Lyons M, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21:1357–1362

    Article  Google Scholar 

  59. Major Torres JM, Stepanov EA (2017) Enhanced face/audio emotion recognition: video and instance level classification using ConvNets and restricted Boltzmann Machines. In: Proceedings of the International Conference on Web Intelligence, ACM

  60. Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recogn 47:1282–1293

  61. Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56

    Google Scholar 

  62. Mollahosseini A, Chan D and Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Lake Placid, NY, USA. 1–10

  63. Moore S, Bowden R (2009) The effects of pose on facial expression recognition. In A. Cavallaro, S. Prince, D. Alexander (Eds.), In: Proceedings of the British machine conference, BMVA Press. 79.1–79.11

  64. Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput VisImage Underst 115:541–558

    Article  Google Scholar 

  65. Nielsen JA, Zielinski BA, Ferguson MA, Lainhart JE, Anderson JS (2013) An evaluation of the left-brain vs. right brain hypothesis with resting state functional connectivity magnetic resonance imaging. PLOS One. Cognitive Neuroscience-Connectomics. DOI: https://doi.org/10.1371/journal.pone.0071275 Featured in PLOS Collections

  66. Padgett C, Cottrell G (1996) Representing face image for emotion classification. Adv Neural Inf Proces Syst 9:894–900

    Google Scholar 

  67. Perez-Gaspar et al (2016) Multimodal emotion recognition with evolutionary computation for human-robot interaction. Expert Syst Appl 66:42–61

    Article  Google Scholar 

  68. Poursaberi A, Noubari HA, Gavrilova M, Yanushkevich SN (2012) Gauss–Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J Image Video Proc. https://doi.org/10.1186/1687-5281-2012-17

  69. Presti LL et al (2017) Boosting Hankel matrices for face emotion recognition and pain detection. Comput Vis Image Underst 156:19–33

    Article  Google Scholar 

  70. Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92

    Article  Google Scholar 

  71. Rosenblum M, Yacoob Y, Davis L (1996) Human expression recognition from motion using a radial basis function network architecture. IEEE Trans Neural Netw 7(5):1121–1138

    Article  Google Scholar 

  72. Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369

    Google Scholar 

  73. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact:1–13

  74. Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing. 370–373

  75. Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  76. Shih FY, Chuang CF, Wang PSP (2008) Performance comparisons of facial expression recognition in JAFFE database. Int J Patt Recog 22(3):445–459

    Article  Google Scholar 

  77. Siddiqi MH, Ali R, Khan AM, Park Y-T, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398

    Article  MathSciNet  MATH  Google Scholar 

  78. Sohail ASM, Bhattacharya P (2011) Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int J Pattern Recognit Artif Intell 25(06):835–862

    Article  MathSciNet  Google Scholar 

  79. Song M, Tao D, Liu Z, Li X, Zhou M (2010) Image ratio features for facial expression recognition application. IEEE Trans Syst, Man, Cybernet-Part B: Cybernet 40(3):779–788

    Article  Google Scholar 

  80. Tarnowski P et al. (2017) Emotion recognition using facial expressions. In: International conference on computational science - ICCS 2017, 12–14 June 2017, Zurich, Switzerland

  81. Tsai H-H, Chang Y-C (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Int J Soft Comput. https://doi.org/10.1007/s00500-017-2634-3

  82. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  83. Uddin MZ, Lee JJ, Kim T-S (2009a) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224

    Article  Google Scholar 

  84. Vadivel A, Shanthi P, Shaila SG (2015) Estimating Emotions Using Geometric Features from Facial Expressions. Encyclopedia of Information Science and Technology, Third Edition. 8

  85. Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Trans Syst, Man, Cybernet-PART B: Cybernet 42(1):28–43

    Article  Google Scholar 

  86. Valstar M, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR Workshops, pp. 76

  87. Vo DM, Le TH (2016) Deep generic features and SVM for facial expression recognition. 3rd National Foundation for Science and Technology Development Conf. Inf. And Comput. Sci (NICS):80–84

  88. Wang Z, Ruan Q (2010) Facial expression based orthogonal local fisher discriminant analysis. In: Proc ICSP 2010:1358–1361

  89. Wang H, Huang H, Makedon F (2014) Emotion detection via discriminant Laplacian embedding. Univ Access Inf Soc 13:23–31

    Article  Google Scholar 

  90. Whitehill J, Bartlett M, Movellan J (2008) Automatic facial expression recognition for intelligent tutoring systems. Proc of Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA, In

    Book  Google Scholar 

  91. Bing-Fei Wu, and Chun-Hsien Lin (2017) Adaptive Feature Mapping for Customizing Deep Learning Based Facial Expression Recognition Model. IEEE access 2017

  92. Wu T, Bartlett M, Movellan J.R (2010) Facial expression recognition using Gabor motion energy filters. In: proceedings of IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW’10), pp. 42–47

  93. www.consortium.ri.cmu.edu/ckagree/

  94. Xiang C, Fan XA, Lee TH (2006) Face recognition using recursive fisher linear discriminant. IEEE Trans Image Process 15(8):2097–2105

    Article  Google Scholar 

  95. Xiaorong P, Fan K, Chen X, Ji L, Zhou Z (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180

    Article  Google Scholar 

  96. Xie X, Lam K-M (2009) Facial expression recognition based on shape and texture. Pattern Recogn 42:1003–1011

    Article  Google Scholar 

  97. Yang Q, Wooldridge M (2015) IJCAI'15 Proceedings of the 24th International Conference on Artificial Intelligence. AAAI Press, Buenos Aires, pp 1617–1623

  98. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508

    Article  Google Scholar 

  99. Zafeiriou S, Pita I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10(8):1528–1540

    Article  Google Scholar 

  100. Thiago H.H. Zavaschi, Alceu S. Britto Jr., Luiz E.S. Oliveira, Alessandro L. Koerich (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl, 40, pp. 646–655

  101. Zhang Z (1999) Feature-based facial expression recognition: sensitivity analysis and experiments with a multi layer perceptron. Int J Pattern Recognit Artif Intell 13:893–911

    Article  Google Scholar 

  102. Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229

    Article  Google Scholar 

  103. Zhang S, Zhao BL (2012) Facial expression based on local binary patterns and local fisher discriminant analysis. WSEAS Trans Sign Proc 8(1):21–31

  104. Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry- based and Gabor-wavelets – based facial expression recognition using multi-layer perceptron. In: proceedings of 3rd international conference on automatic face and gesture recognition, pp. 454–459

  105. Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761

    Article  Google Scholar 

  106. Zhang L, Jiang M, Farid D, Hossain AM (2013) Intelligent facial emotion recognition and semantic- based topic detection for a humanoid robot. Expert Syst Appl 40(13):5160–5168

    Article  Google Scholar 

  107. Zhang L, Mistry K, Jiang M, Neoh SC, Hossain MA (2015) Adaptive facial point detection and emotion recognition for a humanoid robot. Comput Vis Image Underst 140:93–114

    Article  Google Scholar 

  108. Zhang Y-D et al., (2016) Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation, SPECIAL SECTION ON EMOTION-AWARE MOBILE COMPUTING, IEEE Access

  109. Zhao GY, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  110. Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors:9573–9588

  111. Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) learning active facial patches for expression analysis. In IEEE conference on computer vision and pattern recognition (CVPR), Providence, RI, USA

  112. Zou J, Ji Q, Nagy G (2007) A comparative study of local matching approach for face recognition. IEEE Trans Image Process 16(10):2617–2628

    Article  MathSciNet  Google Scholar 

Further Reading

  1. Khana RA, Meyer A, Konik H, Bouaka S (2013) Framework for reliable, real time facial expression recognition for low resolution images. Pattern Recogn Lett 34:1159–1168

    Article  Google Scholar 

  2. Kobayashi H, Hara F (1997) Facial interaction between animated 3dface robot and human beings. In: IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, IEEE, 4, 3732–3737

Download references

Acknowledgements

I would like to thank to the Dr. A. Vadivel, Associate Professor, Depertment of Computer Science and Engineering, SRM University, Amaravathi, Andrapradesh, for his moral support for the manuscript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to R. Vishnu Priya.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vishnu Priya, R. Emotion recognition from geometric fuzzy membership functions. Multimed Tools Appl 78, 17847–17878 (2019). https://doi.org/10.1007/s11042-018-6954-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6954-9

Keywords

Navigation