Soft Computing

, Volume 22, Issue 3, pp 889–903 | Cite as

Edge-enhanced bi-dimensional empirical mode decomposition-based emotion recognition using fusion of feature set

  • Arghya Bhattacharya
  • Dwaipayan Choudhury
  • Debangshu Dey
Methodologies and Application


Emotion recognition has been of great interest in psychology, machine intelligence, human–machine interaction and biomedical fields. This paper proposes a novel soft computing technique for facial emotion recognition by introducing edge- enhanced bidimensional empirical mode decomposition (EEBEMD) as a feature extraction tool for facial emotion recognition. Facial images are subjected to optimized cost function-based self-guided edge enhancement algorithm. BEMD has been applied on the edge- enhanced facial images, and the first four intrinsic mode functions (IMFs) and the residue have been calculated. On the basis of an empirical analysis, the first IMF is selected for further analysis. A proposed fusion model that consists of selected features from the gray-level co-occurrence matrix, the histogram of oriented gradients and the local ternary pattern of the IMF response is fed to a recursive feature elimination-based algorithm to select the appropriate feature subsets for classification. These feature vectors have been trained in three machine learning algorithms namely multi-class SVM, ELM with RBF kernel and k-NN classifier independently. The IMFs have been subjected to principal component analysis and linear discriminant analysis (LDA) algorithm successively for dimensionality reduction, and the facial images with different emotions have been clustered in different zones in the LDA subspace. The proposed method demonstrates promising accuracy when tested on the JAFFE database, Cohn–Kanade database and the eNTERFACE database.


Facial emotion recognition EEBEMD BEMD GLCM HOG LTP MSVM Intrinsic mode function (IMF) Soft computing Principal component analysis Linear discriminant analysis Recursive feature elimination Guided filter 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Ali H (2015) Facial emotion recognition using empirical mode decomposition. Expert Syst Appl 42(3):1261–1277CrossRefGoogle Scholar
  2. Bhattacharya A, Choudhury D, Dey D (2016) Emotion recognition from facial image analysis Using composite similarity measure aided bidimensional empirical mode decomposition. In: IEEE conference on control, measurement and instrumentation, Jadavpur, KolkataGoogle Scholar
  3. Chakraborti T, Chatterjee A, Konar A, Roy AH (2015) Automated emotion recognition employing a novel modified binary quantum-behaved gravitational search algorithm with differential mutation. Expert Syst Appl 32(4):552–530Google Scholar
  4. Chatterjee S , Shi H (2010) A novel neuro fuzzy approach to human emotion determination. In: Digital image computing: techniques and applications, pp 282–287Google Scholar
  5. Clausi David A (2002) An analysis of co-occurrence texture statistics as a function of grey level quantization. Can J Remote Sens 28(1):45–62CrossRefGoogle Scholar
  6. Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HOG features. In: IEEE gesture recognition, Santa Barbara, CA, USAGoogle Scholar
  7. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Internal computer society conference on computer visions and pattern recognitionGoogle Scholar
  8. Deng HB, Jin LW, Zhen LX, Huang JC (2005) A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int J Inform Technol 11(11):86–96Google Scholar
  9. Ekman P, Frisen W (1978) Facial action coding system (FACS). Consulting Psychologists Press, Palo AltoGoogle Scholar
  10. Eleyan A, Demirel H (2011) Co-occurrence matrix and its statistical features as a new approach of face recognition. Turk J ElecEng Comp Sci 19(1):97–107Google Scholar
  11. Gomathi V, Ramar K, Jeevakumar AS (2009) Human facial expression recognition using MANFIS model. Int J Comput Sci Eng 3(2):93–97Google Scholar
  12. Gu WF, Venkatesh YV, Xiang C (2010) A novel application of self-organizing network for facial expression recognition from radial encoded contours. Soft Comput 14:113–122CrossRefGoogle Scholar
  13. Gupta SK, Agarwal S, Meena YK, Nain N (2011) A hybrid method of feature extraction for facial expression recognition. In: 7th international conference on signal image technology & internet-based systems, pp 422–425Google Scholar
  14. Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3(6):610–621CrossRefGoogle Scholar
  15. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409CrossRefGoogle Scholar
  16. Huang et al (2002) Emotion recognition from cauchy naive based classifier. In: 16th international conference on pattern recognition, vol 1Google Scholar
  17. Huang N et al (1998) The empirical mode decomposition and the Hilbert spectrum for non-linear and non-stationary time series analysis. Proc R Soc Lond A 454:903–995MathSciNetCrossRefzbMATHGoogle Scholar
  18. Huang CL, Wang CJ (2006) A GA-based feature selection and parameters optimization for support vector machines. Expert Syst Appl 31(2):231–240CrossRefGoogle Scholar
  19. Hussain S, Triggs B (2010) Feature sets and dimensionality reduction for visual object detection. In: Proceedings of the British machine vision conference, pp 112.1–112.10Google Scholar
  20. Ilbeygi M, Hosseini HS (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146CrossRefGoogle Scholar
  21. Jin Y, Ruan QQ (2009) Face recognition using Gabor-based improved supervised locality preserving projections. Comput Inform 28:81–95zbMATHGoogle Scholar
  22. Kanade T, Cohn JF, Tian YL (2000) Comprehensive database for facial expression analysis. In: Proceedings of 4th IEEE international conference on automatic face and gesture recognition (FG’00), pp 46–53Google Scholar
  23. Kazmi SB, Qurat-ul-Ain, Jaffar MA (2012) Wavelet-based facial expression recognition using a bank of support vector machines. Soft Comput 16(3):369–379Google Scholar
  24. Kharat GU, Dudul SV (2009) Emotion recognition from facial expression using neural networks. In: Hippe ZS, Kulikowski JL (eds) Human–computer systems interaction, AISC 60, pp 207–219Google Scholar
  25. Kim DJ, Song WK, Han JS, Zenn Bien Z (2003) Soft computing based intention reading techniques as a means of human-robot interaction for human centered system. Soft Comput Fusion Found Methodol Appl 7(3):160–166zbMATHGoogle Scholar
  26. Koley B, Dey D (2012) An ensemble system for automatic sleep stage classification using single channel EEG signal. Comput Biol Med 42(12):1186–1195CrossRefGoogle Scholar
  27. Lee CM, Narayanan S, Pieraccini R (2001) Recognition of negative emotions from the speech signal. In: IEEE workshop on automatic speech recognition and understanding, 2001Google Scholar
  28. Lin HJ, Wang SY, Yen SH, Kao YT (2005) Face detection based on skin color segmentation and neural network. In: International conference on neural network and brain. Beijing, pp 1144 – 1149Google Scholar
  29. Lin X, Yang F, Zhou L, Yin P, Kong H, Xing W, Lu X, Jia L, Wang Q, Xu G (2012) A support vector machine-recursive feature elimination feature selection method based on artificial contrast variables and mutual information. J Chromatogr 910:149–155Google Scholar
  30. Liu Z, Wang S (2011) Emotion recognition using hidden markov models from facial temperature sequence. In: Affective computing and intelligent interaction. vol 6975. Springer, Heidelberg, pp 240–247Google Scholar
  31. Martin O, Kotsia I, Macq B, Pitas I (2005) The eNTERFACE’05 audio-visual emotion database. In: Proceedings of the first IEEE workshop on multimedia database management, AtlantaGoogle Scholar
  32. Matsuno K, Lee CW, Kimura S , Tsuji S (1995) Automatic recognition of human facial expressions. In: Proceeding of the fifth international conference on computer vision (ICCV’ 95)Google Scholar
  33. Murthy GRS, Jadon RS (2009) Effectiveness of Eigenspaces for facial expressions recognition. Int J Comput Theory Eng 1(5):1793–8201Google Scholar
  34. Nunes JC, Bouaoune Y, Delechelle E, Niang O, Bunel P (2003) Image analysis by bidimensional empirical mode decomposition. Image Vis Comput 21(2003):1019–1026CrossRefzbMATHGoogle Scholar
  35. Oliveira LS, Koerich RL, Mansano M (2011) 2D Principal Component Analysis for Face and Facial-Expression Recognition. J Comput Sci Eng 13(3):9–13. doi:  10.1109/MCSE.2010.149
  36. Rahulamathavan Y, Phan RCW, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92CrossRefGoogle Scholar
  37. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816CrossRefGoogle Scholar
  38. Shih FY, Chuang CF, Wang PSP (2008) Performance comparisons of facial expression recognition in JAFFE database. Int J Pattern Recognit 22(3):445–459CrossRefGoogle Scholar
  39. Soh L, Tsatsoulis C (1999) Texture analysis of SAR sea ice imagery using gray level co-occurrence matrices. IEEE Trans Geosci Remote Sens 37(2):780–795CrossRefGoogle Scholar
  40. Sugiyama M (2006) Local fisher discriminant analysis for supervised dimensionality reduction. In: Proceedings of the 23rd international conference on machine learning, PittsburghGoogle Scholar
  41. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19:1635–1650MathSciNetCrossRefzbMATHGoogle Scholar
  42. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86CrossRefGoogle Scholar
  43. Visutsak P (2013) Emotion classification through lower facial expressions using adaptive support vector machines. J Man Mach Technol 2(1):12–20Google Scholar
  44. Wang Z, Ruan Q (2010) Facial expression based orthogonal local fisher discriminant analysis. In: Proceedings of ICSP 2010, pp 1358–1361Google Scholar
  45. Xu X, Liang J, Lv S, Wu Q (2014) Human facial expression analysis based on granule LPP. Int J Mach Learn Cybern 5:907–921CrossRefGoogle Scholar
  46. Zhang S, Zhao X, Lei B (2012) Facial expression recognition based on local binary patterns and LFDA. WSEAS Trans Signal Process 8(1):21–31Google Scholar
  47. Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229CrossRefGoogle Scholar
  48. Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10):9573–9588CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Arghya Bhattacharya
    • 1
  • Dwaipayan Choudhury
    • 1
  • Debangshu Dey
    • 1
  1. 1.Department of Electrical EngineeringJadavpur UniversityKolkataIndia

Personalised recommendations