Multimedia Tools and Applications

, Volume 75, Issue 2, pp 935–959 | Cite as

Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection

  • Muhammad Hameed Siddiqi
  • Rahman Ali
  • Muhammad Idris
  • Adil Mehmood Khan
  • Eun Soo Kim
  • Min Cheol Whang
  • Sungyoung LeeEmail author


To recognize expressions accurately, facial expression systems require robust feature extraction and feature selection methods. In this paper, a normalized mutual information based feature selection technique is proposed for FER systems. The technique is derived from an existing method, that is, the max-relevance and min-redundancy (mRMR) method. We, however, propose to normalize the mutual information used in this method so that the domination of the relevance or of the redundancy can be eliminated. For feature extraction, curvelet transform is used. After the feature extraction and selection the feature space is reduced by employing linear discriminant analysis (LDA). Finally, hidden Markov model (HMM) is used to recognize the expressions. The proposed FER system (CNF-FER) is validated using four publicly available standard datasets. For each dataset, 10-fold cross validation scheme is utilized. CNF-FER outperformed the existing well-known statistical and state-of-the-art methods by achieving a weighted average recognition rate of 99 % across all the datasets.


Facial Expressions Curvelet Transform Mutual Information Minimal Redundancy Maximal Relevance 



This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2013-067321)).

This research was also supported by the MSIP (Ministry of Science, ICT & Future Planning), Korea, under the ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2014-(H0301-14-1003).


  1. 1.
    Abd El Meguid M, Levine M (2014) Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Trans Affect Comput 5(2):141–154CrossRefGoogle Scholar
  2. 2.
    Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–550CrossRefGoogle Scholar
  3. 3.
    Baudat G, Anouar F (2000) Generalized discriminant analysis using a kernel approach. Neural Comput 12(10):2385–2404CrossRefGoogle Scholar
  4. 4.
    Baum LE (1972) An equality and associated maximization technique in statistical estimation for probabilistic functions of markov processes. Inequalities 3:1–8Google Scholar
  5. 5.
    Belhumeur PN, Hespanha JP, Kriegman D (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720CrossRefGoogle Scholar
  6. 6.
    Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv preprint. arXiv:12036722
  7. 7.
    Candes E, Demanet L, Donoho D, Ying L (2006) Fast discrete curvelet transforms. Multiscale Modeling & Simulation 5(3):861–899CrossRefMathSciNetzbMATHGoogle Scholar
  8. 8.
    Candes EJ, Donoho DL (2000) Curvelets, multiresolution representation, and scaling laws. In Proc. SPIE, vol 4119, pp 1–12Google Scholar
  9. 9.
    Dash M, Liu H (1997) Feature selection for classification. Intell Data Anal 1(3):131–156CrossRefGoogle Scholar
  10. 10.
    Estevez PA, Tesmer M, Perez CA, Zurada JM (2009) Normalized mutual information feature selection. IEEE Trans Neural Netw 20(2):189–201CrossRefGoogle Scholar
  11. 11.
    Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275CrossRefzbMATHGoogle Scholar
  12. 12.
    Filko D, Martinović G (2013) Emotion recognition system by a neural network based facial expression analysis. Automatika: Journal Control Meas Electron Comput Commun 54(2):263– 272Google Scholar
  13. 13.
    Fodor IK (2002) A survey of dimension reduction techniquesGoogle Scholar
  14. 14.
    Georghiades AS, Belhumeur PN, Kriegman D (2001) From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6):643–660CrossRefGoogle Scholar
  15. 15.
    Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6):7714–7734CrossRefGoogle Scholar
  16. 16.
    Hall MA (1999) Correlation-based feature selection for machine learning. PhD thesis. The University of WaikatoGoogle Scholar
  17. 17.
    He S, Wang S, Lv Y (2011) Spontaneous facial expression recognition based on feature point tracking. In 2011 Sixth International Conference on Image and Graphics (ICIG). IEEE, pp 760–765Google Scholar
  18. 18.
    Hossain MA, Sanyal G (2012) A new improved tactic to extract facial expressions based on genetic algorithm and wvdf. Int J Adv Inf Technol 2(5):37–44CrossRefGoogle Scholar
  19. 19.
    Kamimura R (2011) Structural enhanced information and its application to improved visualization of self-organizing maps. Appl Intell 34(1):102–115CrossRefMathSciNetGoogle Scholar
  20. 20.
    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In Proceedings. Fourth IEEE International Conference onAutomatic Face and Gesture Recognition, 2000. IEEE, pp 46–53Google Scholar
  21. 21.
    Kashyap KL, Shantaiya S (2012) Study and analysis of statistical features of face expression in noisy environment. Int J Image Process Vis Sci 1(2):29–34Google Scholar
  22. 22.
    Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187CrossRefMathSciNetGoogle Scholar
  23. 23.
    Kwak N, Choi CH (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159CrossRefGoogle Scholar
  24. 24.
    Liu M, Shan S, Wang R, Chen X (2014a) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014. CVPR 2014Google Scholar
  25. 25.
    Liu P, Han S, Meng Z, Tong Y (2014b) Facial expression recognition via a boosted deep belief network. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014, CVPR 2014Google Scholar
  26. 26.
    Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In Proceedings. Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. IEEE, pp 200–205Google Scholar
  27. 27.
    Mika S (2002) Kernel fisher discriminants. PhD thesis, UniversitätsbibliothekGoogle Scholar
  28. 28.
    Mika S, Ratsch G, Weston J, Scholkopf B, Mullers K (1999) Fisher discriminant analysis with kernels. In Proceedings of the 1999 IEEE Signal Processing Society Workshop. Neural Networks for Signal Processing IX, 1999. IEEE, pp 41–48Google Scholar
  29. 29.
    Pagariya RR, Bartere MM (2013) Facial emotion recognition in videos using hmm 3(4):111–118Google Scholar
  30. 30.
    Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238CrossRefGoogle Scholar
  31. 31.
    Piatkowska E, Martyna J (147) Spontaneous facial expression recognition: automatic aggression detection. In Hybrid Artificial Intelligent Systems. SpringerGoogle Scholar
  32. 32.
    Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286CrossRefGoogle Scholar
  33. 33.
    Ramirez-Gutierrez K, Sanchez-Perez D, Perez-Meana H, Fujita H, Sasaki J (2010) Face recognition and verification using histogram equalization Selected Topics in Applied Computer Science. WSEAS, pp 85–89Google Scholar
  34. 34.
    Ramirez Rivera A, Castillo R, Chae O (2013) Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans Image Process 22(5):1740–1752CrossRefMathSciNetGoogle Scholar
  35. 35.
    Russell JA (2003) Core affect and the psychological construction of emotion. Psychol Rev 110(1):145CrossRefGoogle Scholar
  36. 36.
    Samaria FS (1994) Face recognition using hidden markov models. PhD thesis. University of CambridgeGoogle Scholar
  37. 37.
    Schels M , Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequences utilizing gmm supervectors. In 2010 20th International Conference on Pattern Recognition (ICPR). IEEE, pp 4251–4254Google Scholar
  38. 38.
    Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis Comput 27(6):803–816CrossRefGoogle Scholar
  39. 39.
    Siddiqi MH, Lee S, Lee YK, Khan AM, Truc PTH (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13(12):16682–16713CrossRefGoogle Scholar
  40. 40.
    Siddiqi MH, Ali R, Khan AM, Kim ES, Kim GJ, Lee S (2014) Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimedia SystemsGoogle Scholar
  41. 41.
    Smith BM, Brandt J, Lin Z, Zhang L (2014) Nonparametric context modeling of local appearance for pose-and expression-robust facial landmark localizationGoogle Scholar
  42. 42.
    Tang M, Chen F (2013) Facial expression recognition and its application based on curvelet transform and pso-svm. Optik-Int J Light Electron Optics 124(22):5401–5406CrossRefMathSciNetGoogle Scholar
  43. 43.
    Uddin MZ, Lee J, Kim TS (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224CrossRefGoogle Scholar
  44. 44.
    Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, Wang X (2010) A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans Multimedia 12(7):682–691CrossRefGoogle Scholar
  45. 45.
    Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12(3):3747–3761CrossRefGoogle Scholar
  46. 46.
    Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10):9573–9588CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Muhammad Hameed Siddiqi
    • 1
  • Rahman Ali
    • 1
  • Muhammad Idris
    • 1
  • Adil Mehmood Khan
    • 2
  • Eun Soo Kim
    • 3
  • Min Cheol Whang
    • 4
  • Sungyoung Lee
    • 1
    Email author
  1. 1.Department of Computer EngineeringKyung Hee UniversitySuwonRepublic of Korea
  2. 2.Division of Information and Computer EngineeringAjou UniversitySuwonRepublic of Korea
  3. 3.Department of Electronic EngineeringKwangwoon UniversitySeoulRepublic of Korea
  4. 4.Division of Digital Media EngineeringSang-Myung UniversitySuwonRepublic of Korea

Personalised recommendations