Skip to main content
SpringerLink
Log in

A novel maximum and minimum response-based Gabor (MMRG) feature extraction method for facial expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In facial expression recognition applications, the images are corrupted with random noise, and this affects the classification accuracy. This article proposes a maximum and Minimum Response-based Gabor (MMRG) that can encode the facial texture more discriminatively and eliminate random noise. Two code images are produced from the available Gabor images. Then, after dividing the code images into grids, feature vectors are formed using histograms. A technique based on the bat algorithm is proposed for the optimization of the Gabor filter banks as Bat Algorithm-based Gabor Optimization (BAGO). The MMRG increases the efficiency of Gabor filter-based features by precisely distinguishing the texture frequencies. It also helps in reducing the dimensions of feature vector which is a major problem in Gabor filter-based feature extraction. Radial Basis Function-Extreme Learning Machine (RBF-ELM) classifier is used for a faster and accurate multi-classification. The proposed approach has been evaluated with six datasets namely, Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK+), Multi-media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 97.2, 97.4, 95.4, 35.4, 87.4 and 82.3% for seven class emotion detection, which is high when compared to other state-of –the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Agarwal S, Santra B, Mukherjee D (2016) Anubhav: recognizing emotions through facial expression. Vis Comput:1–5

  2. Ahmed F, Kabir MH (2012) Directional ternary pattern (DTP) for facial expression recognition. In: IEEE international conference on consumer electronics, pp 265–266

  3. Aifanti N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In: Proceedings of 11th international workshop on image analysis for multimedia interactive services (WIAMIS), 2010 Apr 12, pp 1–4

  4. Alam RN, Younesi F, Alam MR (2009) Computer-aided mass detection on digitized mammograms using a novel hybrid segmentation system. International Journal Biology Biomedical Engineering 3(4):51–58

    Google Scholar 

  5. Alphonse AS, Dharma D (2017) A novel monogenic directional pattern (MDP) and pseudo-Voigt kernel for facilitating the identification of facial emotions. J Vis Commun Image Represent 49:459–470

    Article  Google Scholar 

  6. Alphonse AS, Dharma D (2017) Enhanced Gabor (E-Gabor), Hypersphere-based normalization and Pearson General Kernel-based discriminant analysis for dimension reduction and classification of facial emotions. Expert Syst Appl 90:127–145

    Article  Google Scholar 

  7. Alphonse AS, Dharma D (2018) Novel directional patterns and a generalized supervised dimension reduction system (GSDRS) for facial emotion recognition. Multimed Tools Appl 77(8):9455–9488

    Article  Google Scholar 

  8. Anisetti M, Valerio B (2009) Emotional state inference using face related features. In: New directions in intelligent interactive multimedia systems and services-2. Springer, Berlin, pp 401–411

  9. Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1859–1866

  10. Avola D, Bernardi M, Foresti GL (2018) Fusing depth and colour information for human action recognition. Multimed Tools Appl 2018:1–21

    Google Scholar 

  11. Avola D, Bernardi M, Cinque L, Foresti GL, Massaroni C (2019) Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Transactions on Multimedia 21(1):234–245

    Article  Google Scholar 

  12. Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Trans Neural Netw 13(6):1450–1464. https://doi.org/10.1109/TNN.2002.804287

    Article  Google Scholar 

  13. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720. https://doi.org/10.1109/34.598228. 770

    Article  Google Scholar 

  14. Bertasius G, Shi J, Torresani L (2015) Deepedge: a multi-scale bifurcated deep network for top-down contour detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4380–4389. https://doi.org/10.1109/CVPR.2015.7299067.805

  15. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27

    Article  Google Scholar 

  16. Chen L, Lu G, Zhang D (2004) Effects of different Gabor filters parameters on image retrieval by texture. In: Multimedia modelling conference, proceedings of 10th international 2004 Jan 5, pp 273–278

  17. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Computer vision and pattern recognition, CVPR 2005. IEEE computer society conference, pp 1886–893

  18. Daugman JG (1980) Two-dimensional spectral analysis of cortical receptive field profiles. Vis Res 20:847–856

    Article  Google Scholar 

  19. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial expression databases from movies. IEEE Multimedia 19:34–41

    Article  Google Scholar 

  20. Dhall A, Goecke R, Joshi J, Sikka K, Gedeon T (2014) Emotion recognition in the wild challenge 2014: baseline, data and protocol. ACM ICMI

  21. Doerner K, Gutjahr WJ, Hartl RF, Strauss C, Stummer C (2004) Pareto ant colony optimization: a metaheuristic approach to multiobjective portfolio selection. Ann Oper Res 131(1-4):79–99

    Article  MathSciNet  MATH  Google Scholar 

  22. Eddy SR (1996) Hidden markov models. Curr Opin Struct Biol 6(3):361–365

    Article  Google Scholar 

  23. Etemad K, Chellappa R (1997) Discriminant analysis for recognition of human face images. JOSA A 14(8):1724–1733. https://doi.org/10.1364/josaa.14.001724

  24. Ghimire D, Lee J, Li ZN, Jeong S (2016) Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 15:1–26

    Google Scholar 

  25. Haghighat M, Zonouz S, Abdel-Mottaleb M (2015) CloudID: trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916

    Article  Google Scholar 

  26. Hao XL, Tian M (2017) Deep belief network based on double weber local descriptor in micro-expression recognition. In: Advanced multimedia and ubiquitous engineering, pp 419–425

  27. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. Part B: IEEE Transactions on Systems, Man, and Cybernetics 42(2):513–529

    Google Scholar 

  28. Hussain M (2013) False positive reduction using Gabor feature subset selection. In: international conference on information science and applications (ICISA) vol 0, 1–5. http://dx.doi.ieeecomputersociety.org/10.1109/ICISA.2013.6579383

  29. Hwang J-J, Liu T-L (1989) Pixel-wise deep learning for contour detection. arXiv preprint arXiv:1504.0

    Google Scholar 

  30. Iosifidis A, Tefas A, Pitas I (2015) On the kernel extreme learning machine classifier. Pattern Recogn Lett 54:11–17

    Article  Google Scholar 

  31. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  32. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of fourth IEEE international conference in automatic face and gesture recognition, pp 46–53

  33. Keyvanrad MA, Homayounpour MM (2014) A brief survey on deep belief networks and introducing a new object oriented toolbox (DeeBNet). arXiv:1408.3264 [cs]

  34. Khokher R, Singh RC, Kumar R (2015) Footprint recognition with principal component analysis and independent component analysis. Macromol Symp 347(1):16–26. https://doi.org/10.1002/masy.201400045

    Article  Google Scholar 

  35. Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv:1703.07140

    Google Scholar 

  36. Konak A, Coit DW, Smith AE (2006) Multi-objective optimization using genetic algorithms: a tutorial. Reliab Eng Syst Saf 91(9):992–1007

    Article  Google Scholar 

  37. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  38. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  39. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. In: IJCAI 2015 Jul 25 2015, pp 1617–1623

    Google Scholar 

  40. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 12(181):108–115

    Article  Google Scholar 

  41. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  42. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101

  43. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, pp 200–205

  44. Makhmudkhujaev F, Abdullah-Al-Wadud M, Iqbal MT, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun

  45. Manjunath BS, Ma WY (1996) Texture features for browsing and retrieval of image data. IEEE Trans Pattern Anal Mach Intell (PAMI) 18(8):837–842

    Article  Google Scholar 

  46. Marcelja S (1980) Mathematical description of the responses of simple cortical cells. J Opt Soc Am 70(11):1297–1300

    Article  MathSciNet  Google Scholar 

  47. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  48. Ojansivu V, Heikkilä J (2008) Blur insensitive texture classification using local phase quantization. In: International conference on image and signal processing, pp 236–243

  49. Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Face 36 recognition. I-Tech Education and Publishing, pp 377–416

  50. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: Multimedia and expo, ICME 2005. IEEE international conference on Jul 6 IEEE, p 5

  51. Perona P (1995) Deformable kernels for early vision. IEEE Trans Pattern Anal Mach Intell 17(5):488–499

    Article  Google Scholar 

  52. Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  MathSciNet  MATH  Google Scholar 

  53. Rangayyan RM, Ferrari RJ, Desautels JEL, Frère AF (2000) Directional analysis of images with Gabor wavelets. In: Proc of XIII Brazilian Symposium on Computer Graphics and Image SIBGRAPI, pp 170–177

  54. Reyes-Sierra M, Coello CC (2006) Multi-objective particle swarm optimizers: a survey of the state-of-the-art. Int J Comput Intell Res 2(3):287–308

    MathSciNet  Google Scholar 

  55. Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100

    Article  Google Scholar 

  56. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816. https://doi.org/10.1016/j.imavis.2008.08.005

    Article  Google Scholar 

  57. Shen W, Wang X, Wang Y, Bai X, Zhang Z (2015) Deepcontour: a deep convolutional feature learned by positive-sharing loss for contour detection. In: Proceedings of the IEEE conference on computer vision and pattern 800 recognition, pp 3982–3991. https://doi.org/10.1109/CVPR.2015.7299024

  58. Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713

    Article  Google Scholar 

  59. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  60. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  MATH  Google Scholar 

  61. Turk MA, Pentland AP (1991) Face recognition using eigenfaces. In: Computer vision and pattern recognition, 1991. Proceedings CVPR’91, IEEE Computer Society Conference on. IEEE, pp 586–591. https://doi.org/10.1109/cvpr.1991.139758

  62. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proceedings of 3rd international workshop on EMOTION (satellite of LREC). Corpora for Research on Emotion and Affect, p 65

  63. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  64. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput:1–4

  65. Wu P, Manjunathm BS, Newsam SD, Shin HD (1999) A texture descriptor for image retrieval and browsing. Computer vision and pattern recognition workshop, Fort Collins, CO, USA, June 1999

  66. Xie S, Tu Z (2015) Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision, pp 1395–1403. https://doi.org/10.1109/ICCV.2015.164

  67. Yang XS (2011) Bat algorithm for multi-objective optimisation. International Journal of Bio-Inspired Computation 3(5):267–274

    Article  Google Scholar 

  68. Yang XS, He X (2013) Bat algorithm: literature review and applications. International Journal of Bio-Inspired Computation 5(3):141–149. 38

    Article  Google Scholar 

  69. Yang J, Zhang D, Frangi AF, Yang JY (2004) Two-dimensional pca: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–137. https://doi.org/10.1109/tpami.2004.1261097.775

  70. Zhang K, Huang Y, Du Y (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26:4193–4203

    Article  MathSciNet  MATH  Google Scholar 

  71. Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29(9):607–619

    Article  Google Scholar 

  72. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and Gabor multi-orientation fusion histogram. Math Probl Eng, Math Problems of Engineering. https://doi.org/10.1155/2017/7206041

Download references

Author information

Authors and Affiliations

  1. Department of CSE, Regional Campus of Anna University, Tirunelveli, 627007, India

    A. Sherly Alphonse

  2. Department of Mechanical Engineering, University College of Engineering, Nagercoil, 629004, India

    M. S. Starvin

Authors
  1. A. Sherly Alphonse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. S. Starvin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Sherly Alphonse.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alphonse, A.S., Starvin, M.S. A novel maximum and minimum response-based Gabor (MMRG) feature extraction method for facial expression recognition. Multimed Tools Appl 78, 23369–23397 (2019). https://doi.org/10.1007/s11042-019-7646-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7646-9

Keywords

Search

Navigation