Skip to main content

Advertisement

SpringerLink
Log in

A novel approach for facial expression recognition using local binary pattern with adaptive window

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial Expression Recognition (FER) is an important area in human computer interaction. FER has different applications such as analysis of student behaviour in virtual class room, driver mood detection, security systems, and medicine. The analysis of facial expressions is an interesting and exciting problem. Feature extraction plays important role in any FER system. Local Binary Pattern (LBP) and its variants are popular for feature extraction due to simplicity in computation and monotonic illumination invariant property. However, the performance of LBP is poor in the presence of noise. This work proposes a novel approach for feature extraction to improve the performance of the FER. In this approach, the LBP is calculated considering 4-neighbors and diagonal neighbours separately. Further, for affective feature description, the concept of adaptive window and averaging in radial directions is introduced. This approach reduces the length of the feature vector as well as immune to noise. Support Vector Machine (SVM) is considered for classification. Recognition rate and confusion matrix are used to assess the performance of the proposed algorithm. Extensive experimental results on JAFFE, CK, FERG and FEI face databases show significant improvement in recognition rate compared to the available techniques both in noise free and noisy conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  Google Scholar 

  2. An Q, Han Y, Li J, Lu S (2018) Human-computer interaction nursing system and related algorithms for severely paralyzed patients. In: 2018 15th International conference on control, automation, robotics and vision (ICARCV). IEEE, pp 1929–1935

  3. Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. In: Asian conference on computer vision. Springer, pp 136–153

  4. Awad AI, Hassaballah M (2016) Image feature detectors and descriptors. Studies in Computational Intelligence Springer International Publishing, Cham

    Book  Google Scholar 

  5. Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expressions using gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21 (7):1056–1064

    Article  Google Scholar 

  6. Bellamkonda S, Gopalan N (2018) Facial expression recognition using kirsch edge detection, lbp and gabor wavelets. In: 2018 Second international conference on intelligent computing and control systems (ICICCS). IEEE, pp 1457–1461

  7. Bi H, Li N, Guan H, Lu D, Yang L (2019) A multi-scale conditional generative adversarial network for face sketch synthesis. In: 2019 IEEE international conference on image processing (ICIP). IEEE, pp 3876–3880

  8. Chao WL, Ding JJ, Liu JZ (2015) Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process 117:1–10

    Article  Google Scholar 

  9. Chen A, Xing H, Wang F (2020) A facial expression recognition method using deep convolutional neural networks based on edge computing. IEEE Access 8:49741–49751

    Article  Google Scholar 

  10. Chengeta K, Viriri S (2019) A review of local, holistic and deep learning approaches in facial expressions recognition. In: 2019 Conference on information communications technology and society (ICTAS). IEEE, pp 1–7

  11. Cho M, Kim T, Kim IJ, Lee S (2020) Relational deep feature learning for heterogeneous face recognition. arXiv:200300697

  12. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  13. Dan Z, Chen Y, Yang Z, Wu G (2014) An improved local binary pattern for texture classification. Optik 125(20):6320–6324

    Article  Google Scholar 

  14. Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989

    Article  Google Scholar 

  15. Ekweariri AN, Yurtkan K (2017) Facial expression recognition using enhanced local binary patterns. In: 2017 9th International conference on computational intelligence and communication networks (CICN). IEEE, pp 43–47

  16. Eng S, Ali H, Cheah A, Chong Y (2019) Facial expression recognition in jaffe and kdef datasets using histogram of oriented gradients and support vector machine. In: IOP Conference series: materials science and engineering, vol 705. IOP Publishing, p 012031

  17. Fan DP, Cheng MM, Liu JJ, Gao SH, Hou Q, Borji A (2018) Salient objects in clutter: Bringing salient object detection to the foreground. In: Proceedings of the European conference on computer vision (ECCV), pp 186–202

  18. Fan DP, Zhang S, Wu YH, Liu Y, Cheng MM, Ren B, Rosin PL, Ji R (2019) Scoot: A perceptual metric for facial sketches. In: Proceedings of the IEEE international conference on computer vision, pp 5612–5622

  19. Farajzadeh N, Hashemzadeh M (2018) Exemplar-based facial expression recognition. Inf Sci 460:318–330

    Article  Google Scholar 

  20. Hassaballah M, Awad AI (2016) Detection and description of image features: an introduction. In: Image feature detectors and descriptors. Springer, pp 1–8

  21. Holder RP, Tapamo JR (2017) Improved gradient local ternary patterns for facial expression recognition. EURASIP J Image Vide Process 2017(1):42

    Article  Google Scholar 

  22. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C App Rev 41(6):765–781

    Article  Google Scholar 

  23. Huang Z, Song G, Zhao Y, Han J, Zhao X (2018) Smile recognition based on support vector machine and local binary pattern. In: 2018 IEEE 8th Annual international conference on cyber technology in automation, control, and intelligent systems (CYBER). IEEE, pp 938–942

  24. Huang Y, Wang Y, Tai Y, Liu X, Shen P, Li S, Li J, Huang F (2020) Curricularface: adaptive curriculum learning loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5901–5910

  25. Jabid T, Kabir MH, Chae O (2010) Local directional pattern (ldp) for face recognition. In: 2010 Digest of technical papers international conference on consumer electronics (ICCE). IEEE, pp 329–330

  26. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  27. Jaiswal S, Nandi G (2019) Robust real-time emotion detection system using cnn architecture. Neural Comput & Applic, 1–10

  28. Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. In: 2015 21st Korea-Japan joint workshop on frontiers of computer vision (FCV). IEEE, pp 1–4

  29. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580). IEEE, pp 46–53

  30. Kaplan K, Kaya Y, Kuncan M, Minaz MR, Ertunç HM (2020) An improved feature extraction method using texture analysis with lbp for bearing fault diagnosis. Appl Soft Comput 87:106019

    Article  Google Scholar 

  31. Kaushik MS, Kandali AB (2017) Recognition of facial expressions extracting salient features using local binary patterns and histogram of oriented gradients. In: 2017 International conference on energy, communication, data analytics and soft computing (ICECDS). IEEE, pp 1201–1205

  32. Khan RA, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recogn Lett 34(10):1159–1168

    Article  Google Scholar 

  33. Kim JH, Kim BG, Roy PP, Jeong DM (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  34. Lekdioui K, Messoussi R, Ruichek Y, Chaabi Y, Touahni R (2017) Facial decomposition for expression recognition using texture/shape descriptors and svm classifier. Signal Process Image Commun 58:300–312

    Article  Google Scholar 

  35. Li J, Zhang D, Zhang J, Zhang J, Li T, Xia Y, Yan Q, Xun L (2017) Facial expression recognition with faster r-cnn. Procedia Comput Sci 107:135–140

    Article  Google Scholar 

  36. Liliana D (2019) Emotion recognition from facial expression using deep convolutional neural network. In: Journal of physics: conference series, vol 1193. IOP Publishing, p 012004

  37. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition. IEEE, pp 200–205

  38. Mehta R, Egiazarian K (2016) Dominant rotated local binary patterns (drlbp) for texture classification. Pattern Recogn Lett 71:16–22

    Article  Google Scholar 

  39. Minaee S, Abdolrashidi A (2019) Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv:1902.01019

  40. Nigam S, Singh R, Misra A (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain. Multimed Tools Appl 77(21):28725–28747

    Article  Google Scholar 

  41. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  42. Pan Z, Wu X, Li Z, Zhou Z (2017) Local adaptive binary patterns using diamond sampling structure for texture classification. IEEE Signal Process Lett 24(6):828–832

    Article  Google Scholar 

  43. Patil M, Veni S (2019) Driver emotion recognition for enhancement of human machine interface in vehicles. In: 2019 International conference on communication and signal processing (ICCSP). IEEE, pp 0420–0424

  44. Perez-Gaspar LA, Caballero-Morales SO, Trujillo-Romero F (2016) Multimodal emotion recognition with evolutionary computation for human-robot interaction. Expert Syst Appl 66:42–61

    Article  Google Scholar 

  45. Pitaloka DA, Wulandari A, Basaruddin T, Liliana DY (2017) Enhancing cnn with preprocessing stage in automatic emotion recognition. Procedia Comput Sci 116:523–529

    Article  Google Scholar 

  46. Roy SD, Bhowmik MK, Saha P, Ghosh A K (2016) An approach for automatic pain detection through facial expression. Procedia Comput Sci 84:99–106

    Article  Google Scholar 

  47. Salahat E, Qasaimeh M (2017) Recent advances in features extraction and description algorithms: a comprehensive survey. In: 2017 IEEE international conference on industrial technology (ICIT). IEEE, pp 1059–1063

  48. Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International conference on image processing 2005, vol 2. IEEE, pp II–370

  49. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vision Comput 27 (6):803–816

    Article  Google Scholar 

  50. Shan K, Guo J, You W, Lu D, Bie R (2017) Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th international conference on software engineering research, management and applications (SERA). IEEE, pp 123–128

  51. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  Google Scholar 

  52. Thomaz CE, Giraldi GA (2010) A new ranking method for principal components analysis and its application to face image analysis. Image Vision Comput 28(6):902–913

    Article  Google Scholar 

  53. Tong Y, Chen R, Cheng Y (2014) Facial expression recognition algorithm using lgc based on horizontal and diagonal prior principle. Optik 125 (16):4186–4189

    Article  Google Scholar 

  54. Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  55. Yee SY, Rassem TH, Mohammed MF, Makbol NM (2019) Performance evaluation of completed local ternary pattern (cltp) for face image recognition. Perform Eval, 10(4)

  56. Zhang J, Fan DP, Dai Y, Anwar S, Saleh FS, Zhang T, Barnes N (2020) Uc-net: uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8582–8591

  57. Zhang Y, Hua C (2015) Driver fatigue recognition based on facial expression analysis using local binary patterns. Optik 126(23):4501–4505

    Article  Google Scholar 

  58. Zhang J, Yu X, Li A, Song P, Liu B, Dai Y (2020) Weakly-supervised salient object detection via scribble annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12546–12555

  59. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J, Cheng MM (2019) Egnet: Edge guidance network for salient object detection. In: Proceedings of the IEEE international conference on computer vision, pp 8779–8788

Download references

Acknowledgments

The corresponding author acknowledges the research colleagues Dr. G. Sridevi, Professor of ECE, Aditya Engineering College, Surampalem, and Dr. B. Chandra Mohan, Professor of ECE, Bapatla Engineering College, Bapatla, for their valuable advices and suggestions during the execution of this work.

Author information

Authors and Affiliations

  1. Department of ECE, UCEK, JNTUK, Kakinada, 533003, India

    Durga Ganga Rao Kola & Srinivas Kumar Samayamantula

Authors
  1. Durga Ganga Rao Kola
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srinivas Kumar Samayamantula
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Durga Ganga Rao Kola.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kola, D.G.R., Samayamantula, S.K. A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 80, 2243–2262 (2021). https://doi.org/10.1007/s11042-020-09663-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09663-2

Keywords

Search

Navigation