Skip to main content
SpringerLink
Log in

A comparative study on facial expression recognition using local binary patterns, convolutional neural network and frequency neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As the principal processing method for nonverbal intentions, Facial Expression Recognition (FER) is an important and promising topic of computer vision and artificial intelligence, as well as one of the subject areas of symmetry. This research work provides a thorough and well-organized comprehensive comparative empirical study of facial expression recognition based on a deep learning study in frequency domain, convolution neural network, and local binary patterns features. We have attained the FER by incorporating neutral, joy, anger, fear, sadness, disgust, and surprise as seven universal emotional categories. In terms of methodology, we present a broad framework for a traditional FER approach and analyze the possible technologies that can be used in each component to emphasis the contrasts and similarities. Even though there has been a lot of research done with static images, there is still a lot of work being done to develop new ways that are easier to compute and use less memory than prior methods. This research could pave the way for a new approach to facial emotion identification in terms of accuracy and high-performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The data used to support the findings of this study are included within the article.

References

  1. Ahmed N, Natarajan T, Rao KR (1974) Discrete cosine transform. IEEE Trans Comput C-23(1):90–93

    Article  MathSciNet  MATH  Google Scholar 

  2. Ahonen T, Hadid A, Pietikäinen M (2004) Face recognition with local binary patterns. In: European conference on computer vision (ECCV)

    Google Scholar 

  3. Al-akam R, Al-darraji S, Paulus D (2018) Human action recognition from RGBD videos based on retina model and local binary pattern features. In: 26 Conference on Computer Graphics, Visualization and Computer Vision (WSCG), pp 1–7

  4. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: machine learning and application to spotaneous behavior. In: IEEE conference on computer vision and pattern recognition (CVPR)

    Google Scholar 

  5. Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. In: IEEE conference on computer vision and pattern recognition (CVPR)

    Google Scholar 

  6. Cohen I, Sebe N, Garg A, Chen L, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91:160–187

    Article  Google Scholar 

  7. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18:32–80

    Article  Google Scholar 

  8. Darwin C (2015) The expression of the emotions in man and animals. University of Chicago press, Chicago

    Google Scholar 

  9. Ekman P (1999) Facial expressions. In: Dalgleish T, Power M (eds) Handbook of cognition and emotion. Wiley, New York

    Google Scholar 

  10. Ekman P, Friesen WV (1975) Emotion in the human face. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  11. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275

    Article  MATH  Google Scholar 

  12. Guo L, Xu D, Qiang Z (2016) Background subtraction using local SVD binary pattern. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 86–94

    Google Scholar 

  13. Hadid A, Pietikäinen M, Ahonen T (2004) A discriminative feature space for detecting and recognizing faces. In: IEEE conference on computer vision and pattern recognition (CVPR)

    MATH  Google Scholar 

  14. Hsu C-W, Chang C-C, Lin C-J (2003) A practical guide to support vector classification. Tech. Rep., Taipei

    Google Scholar 

  15. Jabon M, Bailenson J, Pontikakis E, Takayama L, Nass C (2011) Facial expression analysis for predicting unsafe driving behavior. IEEE Perv Comput 10:84–95

    Article  Google Scholar 

  16. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp 2983–2991

    Google Scholar 

  17. Kaliouby RE, Robinson P (2004) Real-time inference of complex mental states from facial expressions and head gestures. In: IEEE CVPR workshop on real-time vision for human–computer interaction

    Google Scholar 

  18. Khan RA, Meyer A, Konik H, Bouakaz S (2012) Human vision inspired framework for facial expressions recognition. In: 2012 19th IEEE international conference on image processing. IEEE, pp 2593–2596

    Chapter  Google Scholar 

  19. Kumar N, Punetha D, Ehiagwina F (2016) Preserving the edges of a digital image using various filtering algorithms and tools. Int J Signal Process Image Process Pattern Recognit 9(12):11–18

    Google Scholar 

  20. Lee CS, Elgammal A (2005) Facial expression analysis using nonlinear decomposable generative models. In: IEEE international workshop on analysis and modeling of faces and gestures (AMFG)

    Google Scholar 

  21. Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    Article  MathSciNet  Google Scholar 

  22. Li D, Zhao X, Yuan G, Liu Y, Liu G (2021) Robustness comparison between the capsule network and the convolutional network for facial expression recognition. Appl Intell 51(4):2269–2278

    Article  Google Scholar 

  23. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  24. Mehta V, Punetha D (2015) A fascinating territory approaching edge detection using feasibility of eigen face to identify an individual. In: Second international conference on advances in computing and communication engineering. IEEE, pp 530–534

    Google Scholar 

  25. Mohseni S, Kordy HM, Ahmadi R (2013) Facial expression recognition using DCT features and neural network based decision tree. In: Proc. ELMAR, pp 361–364

    Google Scholar 

  26. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  27. Pantic M, Rothkrantz L (2000) Expert system for automatic analysis of facial expression. Image Vis Comput 18(11):881–905

    Article  Google Scholar 

  28. Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K (2020) Human behavior understanding in big multimedia data using CNN based facial expression recognition. Mob Netw Appl 25(4):1611–1621

    Article  Google Scholar 

  29. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  30. Sun Y, Mao X, Hong S, Xu W, Gui G (2019) Template matching-based method for intelligent invoice information identification. IEEE Access 7:28392–28401

    Article  Google Scholar 

  31. Suwa M, Sugie N, Fujimora K (1978) A preliminary note on pattern recognition of human emotional expression. In: International joint conference on pattern recognition, pp 408–410

    Google Scholar 

  32. Tang Y, Zhang XM, Wang H (2018) Geometric-convolutional feature fusion based on learning propagation for facial expression recognition. IEEE Access 6:42532–42540

    Article  Google Scholar 

  33. Tian Y (2004) Evaluation of face resolution for expression analysis. In: CVPR workshop on face processing in video

    Google Scholar 

  34. Tian Y, Kanade T, Cohn J (2005) Handbook of face recognition. Springer, New York

    Google Scholar 

  35. Valstar M, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, vol 3, pp 76–84

    Google Scholar 

  36. Varga D, Havasi L, Szirányi T (2015) Pedestrian detection in surveillance videos based on CS-LBP feature. In: 2015 International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS). IEEE, pp 413–417

    Chapter  Google Scholar 

  37. Wang S, Eccleston C, Keogh E (2021) The time course of facial expression recognition using spatial frequency information: comparing pain and Core emotions. J Pain 22(2):196–208

    Article  Google Scholar 

  38. Xiao Y, Ma L, Khorasani K (2006) A new facial expression recognition technique using 2-D DCT and neural networks based decision tree. In: Proc. IEEE Int. Joint Conf. Neural Netw., pp 2421–2428

    Google Scholar 

  39. Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 27(5):1–16

    Google Scholar 

  40. Zhang Z, Lyons MJ, Schuster M, Akamatsu S (1998) Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: IEEE international conference on automatic face & gesture recognition (FG)

    Google Scholar 

  41. Zhao G, Pietikainen M (Jun. 2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Sanjeev Kumar and Vikas Sagar contributed equally to this work.

Authors and Affiliations

  1. Department of Master of Computer Applications, G L Bajaj Institute of Technology and Management, Greater Noida, Uttar Pradesh, India

    Sanjeev Kumar

  2. Department of Artificial Intelligence, Noida Institute of Engineering and Technology, Greater Noida, Uttar Pradesh, India

    Vikas Sagar

  3. School of Electronics Engineering, Vellore Institute of Technology (VIT) University, Chennai, Tamilnadu, 600127, India

    Deepak Punetha

Authors
  1. Sanjeev Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vikas Sagar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Deepak Punetha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deepak Punetha.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, S., Sagar, V. & Punetha, D. A comparative study on facial expression recognition using local binary patterns, convolutional neural network and frequency neural network. Multimed Tools Appl 82, 24369–24385 (2023). https://doi.org/10.1007/s11042-023-14753-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14753-y

Keywords

Search

Navigation