Skip to main content
SpringerLink
Log in

Human vision inspired feature extraction for facial expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression is a powerful way for human emotional communications. According to various applications, automatic facial expression recognition becomes an interesting problem for researchers in different areas. An automatic facial expression recognition system recognizes the expressed emotion in input facial image using several processing stages. Feature extraction is a vital step in facial expression recognition. One of the most widely used techniques for feature extraction in machine vision is utilizing Gabor filter which is sensitive to lines at various orientations. The disadvantage of Gabor filters is their computational cost and large feature vector length. Inspiring the human vision system and stimulations of complex cells, in this paper, firstly, facial image is convolved with Gabor filters. Then, the achieved convolution matrices are properly coded based on the maximum and minimum responses. Finally, the feature vector is obtained by calculating the histogram of these codes. The length of achieved histogram for 16 and 8 Gabor filters are 240 and 56, respectively, which is considerably less than keeping all Gabor responses. The proposed method is (person-independently) evaluated on four facial expression recognition datasets including CK+, SFEW, MMI, and RAF-DB. The experimental results show that the proposed method outperforms existing image texture descriptors in facial expression recognition in both controlled and uncontrolled images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Acevedo D, Negri P, Buemi ME, Fernández FG, Mejail M (2017) A simple geometric-based descriptor for facial expression recognition. In: Automatic face & gesture recognition (FG 2017), 2017 12th IEEE international conference on. IEEE, pp 802–808

  2. Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expression using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064

    Google Scholar 

  3. Ben X, Jia X, Yan R, Zhang X, Meng W (2018) Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn Lett 107:50–58

    Google Scholar 

  4. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):1–27. Software available at https://www.csie.ntu.edu.tw/~cjlin/libsvm/

    Google Scholar 

  5. Cruz EAS, Jung CR, Franco CHE (2017) Facial expression recognition using temporal POEM features. Pattern Recogn Lett, In Press

  6. Daugmen J (1988) Complete discrete 2d Gabor transforms by neutral networks for image analysis and compression. IEEE Trans Acoust Speech Signal Process 36(7):1169–1179

    Google Scholar 

  7. Dhall A, Ramana Murthy OV, Goecke R, Joshi J, Gedeon T (2015) Video and image based emotion recognition challenges in the wild: Emotiw 2015. In: Proceedings of the 2015 ACM on int. conf. multimodal interaction. ACM, pp 423–426

  8. Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531

    Google Scholar 

  9. Duan Y, Lu J, Feng J, Zhou J (2018) Context-aware local binary feature learning for face recognition. IEEE Trans Pattern Anal Mach Intell 40(5):1139–1153

    Google Scholar 

  10. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129

    Google Scholar 

  11. Eleftheriadis S, Rudovic O, Pantic M (2015) Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans Image Process 24(1):189–204

    MathSciNet  MATH  Google Scholar 

  12. Frijda NH, Tcherkassof A (1997) Facial expressions as modes of action readiness. In: the psychology of facial expression, 2nd edn, pp 78–102

    Google Scholar 

  13. Gabbiani F, Cox SJ (2017) Mathematics for neuroscientists. Chapter 23: models of simple and complex cells. Academic, New York

    MATH  Google Scholar 

  14. Ghimire D, Lee J, Li ZN, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76(6):7921–7946

    Google Scholar 

  15. Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821

    Google Scholar 

  16. Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP (2014) Nonverbal social withdrawal in depression: evidence from manual and automatic analysis. Image Vis Comput 32(10):641–647

    Google Scholar 

  17. Gu W, Xiang C, Venkatesh YV, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45:80–91

    Google Scholar 

  18. Guo M, Hou X, Ma Y, Wu X (2017) Facial expression recognition using ELBP based on covariance matrix transform in KLT. Multimed Tools Appl 76(2):2995–3010

    Google Scholar 

  19. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Google Scholar 

  20. Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2016) Effective semantic features for facial expressions recognition using SVM. Multimed Tools Appl 75(11):6663–6682

    Google Scholar 

  21. Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Multimed Tools Appl 73(1):309–326

    Google Scholar 

  22. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):765–781

    Google Scholar 

  23. Irani R, Nasrollahi K, Simon MO, Corneanu CA, Escalera S, Bahnsen C, Lundtoft DH, Moeslund TB, Pedersen TL, Klitgaard M-L et al (2015) Spatiotemporal analysis of rgb-dt facial images for multimodal pain level recognition. In: CVPR Workshops

  24. Jia X, Ben X, Yuan H, Kpalma K, Meng W (2018) Macro-to-micro transformation model for micro-expression recognition. J Comput Sci-Neth 25:289–297

    Google Scholar 

  25. Joshi J, Dhall A, Goecke R, Breakspear M, Parker G (2012) Neural-net classification for spatio-temporal descriptor based depression analysis. In: Pattern recognition (ICPR), 2012 21st international conference on. IEEE, pp 2634–2638

  26. Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. ISVC, pp 368–377

  27. Kandel ER, Schwartz JH, Jessell TM, Siegelbaum SA, Hudspeth AJ (2013) Principles of neural science, Fifth edn. McGraw-hill, New York

    Google Scholar 

  28. Kazemi V, Josephine S (2014) One millisecond face alignment with an ensemble of regression trees. In: Computer vision and pattern recognition (CVPR), 2014 IEEE conference on. IEEE

  29. Kobayashi H, Hara F (1992) Recognition of six basic facial expression and their strength by neural network. In: Proc. int’l workshop robot and human comm, pp 381–386

  30. Kobayashi H, Hara F (1992) Recognition of mixed facial expression by neural network. In: Proc. int’l workshop robot and human comm, pp 387–391

  31. Lei Y, Zhao X, Wang G, Yu K, Guo W (2017) A novel approach for cirrhosis recognition via improved LBP algorithm and dictionary learning. Biomed Signal Proces 38:281–292

    Google Scholar 

  32. Levenson RW, Ekman P, Friesen WV (1990) Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27(4):363–384

    Google Scholar 

  33. Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861

  34. Li R, Tian J, Chua MCH (2018) Facial expression classification using salient pattern driven integrated geometric and textual features. Multimed Tools Appl:1–13

  35. Linton O (2017) Probability, statistics and econometrics. Chapter 12: hypothesis testing. Academic, London

    Google Scholar 

  36. Liu M, Shan S, Wang R, Chen X (2014) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: IEEE Conf. Comp. Vis. Patt. Rec., pp 1749–1756

  37. Liu Z, Shan L, Weihong D (2017) Boosting-POOF: boosting part based one vs one feature for facial expression recognition in the wild. In: Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE international conference on. IEEE, pp 967–972

  38. Lu J, Liong VE, Zhou X, Zhou J (2015) Learning compact binary face descriptor for face recognition. IEEE Trans Pattern Anal Mach Intell 37(10):2041–2056

    Google Scholar 

  39. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the third international workshop on CVPR for human communicative behavior analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101

  40. Lucey P, Cohn JF, Matthews I, Lucey S, Sridharan S, Howlett J, Prkachin KM (2011) Automatically detecting pain in video through facial action units. IEEE Trans Syst Man Cybern B Cybern 41(3):664–674

    Google Scholar 

  41. Majumdern A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recogn 47(3):1282–1293

    Google Scholar 

  42. Moreno P, Bernardino A, Santos-Victor J Gabor parameter selection for local feature detection. In: Iberian conference on pattern recognition and image analysis 2005 Jun 7. Springer, Berlin, pp 11–19

    Google Scholar 

  43. Niedenthal PM (2007) Embodying emotion. Science 116:1002–1005

    Google Scholar 

  44. Nigam S, Singh R, Misra AK (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain. Multimed Tools Appl:1–23

  45. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distribution. Pattern Recogn 29(1):51–59

    Google Scholar 

  46. Pai N-S, Chang S-P (2011) An embedded system for real-time facial expression recognition based on the extension theory. Comput Math Appl 61:2101–2106

    MATH  Google Scholar 

  47. Palestra G, Pettinicchio A, Coco MD, Carcagnì P, Leo M, Distante C (2015) Improved performance in facial expression recognition using 32 geometric features. In: Image Analysis and Processing, vol 9280. Springer, pp 518–528

  48. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo (ICME)

  49. Rivera AR, Castillo JR, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    MathSciNet  MATH  Google Scholar 

  50. Roger Highfield RW, Jenkins R (2009) How your looks betray your personality. New Scientist

  51. Sadeghi H, Raie AA (2013) Semi-dynamic facial expression recognition based on masked displacement image. In: International symposium on artificial intelligence and signal processing. Springer, Cham, pp 100–108

    Google Scholar 

  52. Scherer S, Stratou G, Mahmoud M, Boberg J, Gratch J, Rizzo A, Morency L-P (2013) Automatic behavior descriptors for psychological disorder analysis. In: Automatic face and gesture recognition (FG), 2013 10th IEEE international conference and workshops on. IEEE, pp 1–8

  53. Shams L, Von Der Malsburg C (2002) The role of complex cells in object recognition. Vis Res 42(22):2547–2554

    Google Scholar 

  54. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Google Scholar 

  55. Shu C, Ding X, Fang C (2011) Histogram of the oriented gradient for face recognition. Tsinghua Sci Technol 16(2):216–224

    Google Scholar 

  56. Tian Y, Kanade T, Cohn J (2005) Handbook of face recognition. Springer (Chapter 11. Facial Expression Analysis).

  57. Valstar MF, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd intern. Workshop on EMOTION (satellite of LREC): corpora for research on emotion and affect, Valletta, Malta. ELRA

  58. Vu NS, Caplier A (2012) Enhanced patterns of oriented edge magnitudes for face recognition and image matching. IEEE Trans Image Process 21(3):1352–1365

    MathSciNet  MATH  Google Scholar 

  59. Xiao R, Zhao Q, Zhang D, Shi P (2011) Facial expression recognition on multiple manifolds. Pattern Recogn 44:107–116

    MATH  Google Scholar 

  60. Yang B, Cao JM, Jiang DP, Lv JD (2017) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tools Appl:1–23

  61. Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229

    Google Scholar 

  62. Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Google Scholar 

  63. Zhao G, Pietikäinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial. Pattern Recogn Lett 13:1117–1127

    Google Scholar 

  64. Zong Y et al (2016) Emotion recognition in the wild via sparse transductive transfer linear discriminant analysis. J Multimodal User In 10(2):163–172

    Google Scholar 

Download references

Acknowledgments

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Electrical Engineering Department, Amirkabir University of Technology, Tehran, Iran

    Hamid Sadeghi & Abolghasem-A. Raie

Authors
  1. Hamid Sadeghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abolghasem-A. Raie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abolghasem-A. Raie.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sadeghi, H., Raie, AA. Human vision inspired feature extraction for facial expression recognition. Multimed Tools Appl 78, 30335–30353 (2019). https://doi.org/10.1007/s11042-019-07863-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07863-z

Keywords

Search

Navigation