Skip to main content
SpringerLink
Log in

Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Facial emotion recognition finds a major role in affective computing. Recognizing emotion by facial expression is an extremely important activity to design control oriented and human computer interactive applications especially in cognitive science and neuroscience. For a precise and robust recognition, feature extraction is one of the major challenges in facial expression recognition system. Wavelet transform is one of the major key methods utilized for feature extraction in facial emotion recognition. In this paper, the statistical parameters from the proposed subband selective multilevel stationary wavelet gradient transform are calculated and are utilized as features for efficacious recognition of emotion. The features of the wavelet transform contain both spatial and spectral domain information which is best suited for identifying human emotions through facial expression. The introduction of gradient transform to find the gradient of subband avails to estimate the edges in images for the quality amelioration of subbands. The dimension reduction in the extracted features is done by using Pearson–kernel–principal component analysis method. The classification of emotion using the selected features is done by the proposed Gaussian membership function fuzzy SVM classifier. Experiments were performed on the well-known database for facial expression such as JAFEE database, CK + database and FG Net database and obtained promising emotion classification results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Gavrilescu, M.: Recognizing emotions from videos by studying facial expressions, body postures and hand gestures. In: 23rd Telecommunications ForumTelfor, Belgrade, SERBIA (2015). https://doi.org/10.1109/telfor.2015.7377568

  2. Li, W., Zhang, Y., Fu, Y.: Speech Emotion recognition in elearning system based on affective computing. In: Proceedings of Natural Computation (ICNC 2007), Aug 2007 (2007). https://doi.org/10.1109/icnc.2007.677

  3. Cambria, E.: Affective computing and sentiment analysis. IEEE Intell. Syst. 31(2), 102–107 (2016). https://doi.org/10.1109/MIS.2016.31

    Article  Google Scholar 

  4. Darwin, C.: The expression of the emotions in man and animals. J. Murray, London (1872)

    Book  Google Scholar 

  5. Ekman, P., Friesen, W.V.: Constant across cultures in face and emotions. J. Personal. Soc. Psychol. 17(2), 124–129 (1971)

    Article  Google Scholar 

  6. Bendjillali, R.I., Beladgham, M., Merit, K., Taleb-Ahmed, A.: Improved facial expression recognition based on DWT feature for deep CNN. Electronics 8(3), 324 (2019). https://doi.org/10.3390/electronics8030324

    Article  Google Scholar 

  7. Zhang, Yu.-Dong., Yang, Zhang.-Jing., Hui-Min, Lu., Zhou, Xing.-Xing., Phillips, Preetha., Liu, Qing.-Ming., Wang, Shui.-Hua.: Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified crossvalidation. IEEE Access (2016). https://doi.org/10.1109/ACCESS.2016.2628407.4

    Article  Google Scholar 

  8. Qayyum, H., Majid, M., Anwar, S.M., Khan, B.: Facial expression recognition using stationary wavelet transform features. Hindawi Math. Probl. Eng. (2017). https://doi.org/10.1155/2017/9854050

    Article  Google Scholar 

  9. Wang, S.H., Phillips, P., Dong, Z.C., Zhang, Y.D.: Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm. Neurocomputing 10, 15–20 (2017). https://doi.org/10.1016/j.neucom.2017.08.015

    Article  Google Scholar 

  10. Qin, Shu., Zhu, Zhengzhou., Zou, Yuhang., Wang, Xiaowei.: Facial expression recognition based on Gabor wavelet transform and 2-channel CNN, Int. J. Wavelets Multiresol. Inf. Process. (2020). https://doi.org/10.1142/S0219691320500034

    Article  MATH  Google Scholar 

  11. Meena, H.K., Joshi, S.D., Sharma, K.K.: Facial expression recognition using graph signal processing on HOG. IETE J. Res. (2019). https://doi.org/10.1080/03772063.2019.1565952

    Article  Google Scholar 

  12. Khan, R.A., Meyer, A., Konik, H., Bouakaz, S.: Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit. Lett. 34, 1159–1168 (2013). https://doi.org/10.1016/j.patrec.2013.03.022

    Article  Google Scholar 

  13. Makhmudkhujaev, F., Abdullah-Al-Wadud, M., Iqbal, M.T.B., Ryu, B., Chae, O.: Facial expression recognition with local prominent directional pattern. Signal Process. Image Commun. (2019). https://doi.org/10.1016/j.image.2019.01.002

    Article  Google Scholar 

  14. Uma Maheswari, V., Varaprasad, G., Viswanadha Raju, S.: Local directional maximum edge patterns for facial expression recognition. J. Ambient Intell. Hum. Comput. (2020). https://doi.org/10.1007/s12652-020-018863

    Article  Google Scholar 

  15. Ali, H., Hariharan, M., Yaacob, S., Adom, A.H.: Facial emotion recognition based on higher-order spectra using support vector machines. J. Med. Imaging Health Inform. 5(6), 1272–1277 (2015). https://doi.org/10.1166/jmihi.2015.1527

    Article  Google Scholar 

  16. Gogić, I., Manhart, M., Pandžić, I.S., et al.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 36, 97–112 (2020). https://doi.org/10.1007/s00371-018-1585-8

    Article  Google Scholar 

  17. Jamshidnezhad, A., Nordin, M.J.: Bee royalty offspring algorithm for improvement of facial expressions classification model. Int. J. Bio-Inspired Comput. (2013). https://doi.org/10.1504/IJBIC.2013.055092

    Article  Google Scholar 

  18. Yu, M., Zheng, H., Peng, Z., Dong, J., Du, H.: Facial expression recognition based on a multi-task global-local network. Pattern Recognit. Lett. (2020). https://doi.org/10.1016/j.patrec.2020.01.016

    Article  Google Scholar 

  19. Zhang, H., Su, W., Wang, Z.: Weakly supervised local-global attention network for facial expression recognition. IEEE Access (2020). https://doi.org/10.1109/ACCESS.2020.2975913

    Article  Google Scholar 

  20. Gan, Yanling., Chen, Jingying., Yang, Zongkai., Luhui, Xu.: Multiple attention network for facial expression recognition. IEEE Access (2019). https://doi.org/10.1109/ACCESS.2020.2963913

    Article  Google Scholar 

  21. Yu, N., Bai, D.: Facial expression recognition by jointly partial image and deep metric learning. IEEE Access 8, 4700–4707 (2019). https://doi.org/10.1109/ACCESS.2019.2963201

    Article  Google Scholar 

  22. Sun, X., Zheng, S., Fu, H.: ROI-attention vectorized CNN model for static facial expression recognition. IEEE Access (2020). https://doi.org/10.1109/ACCESS.2020.2964298

    Article  Google Scholar 

  23. Fan, X., Tjahjadi, T.: Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J. Vis. Commun. Image Represent. (2019). https://doi.org/10.1016/j.jvcir.2019.102659

    Article  Google Scholar 

  24. Pan, X.: Fusing HOG and convolutional neural network spatial–temporal features for video-based facial expression recognition. IET Image Process 14(1), 176–182 (2020). https://doi.org/10.1049/iet-ipr.2019.0293

    Article  Google Scholar 

  25. Li, K., Jin, Y., Akram, M.W., et al.: Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis. Comput. 36, 391–404 (2020). https://doi.org/10.1007/s00371-019-01627-4

    Article  Google Scholar 

  26. Sun, X., Xia, P., Zhang, L., Shao, L.: A ROI-guided deep architecture for robust facial expressions recognition. Inf. Sci. 522, 35–48 (2020). https://doi.org/10.1016/j.ins.2020.02.047

    Article  Google Scholar 

  27. Reddy, G.V., Savarni, C.D., Mukherjee, S.: Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features. Cognit. Syst. Res. 62, 23–34 (2020). https://doi.org/10.1016/j.cogsys.2020.03.002

    Article  Google Scholar 

  28. Iqbal, M.T.B., Abdullah-Al-Wadud, M., Ryu, B., Makhmudkhujaev, F., Chae, O.: Facial expression recognition with neighborhood-aware edge directional pattern (NEDP). IEEE Trans. Affect. Comput 11(1), 125–137 (2020). https://doi.org/10.1109/taffc.2018.2829707

    Article  Google Scholar 

  29. Joseph, A., Geetha, P.: Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow. Vis. Comput. 36, 529–539 (2020). https://doi.org/10.1007/s00371-019-01628-3

    Article  Google Scholar 

  30. Reza, A.M.: Realization of the contrast limited adaptive histogram equalization (CLAHE) for real time image enhancement. J. VLSI Signal Process. Syst. Signal Image Video Technol. 38(1), 35–44 (2004). https://doi.org/10.1023/B:VLSI.0000028532.53893.82

    Article  Google Scholar 

  31. Ma, J., Fan, X., Yang, S.X., Zhang, X., Zhu, X.: Contrast limited adaptive histogram equalization-based fusion in YIQ and HSI color spaces for underwater image enhancement. Int. J. Pattern Recognit Artif Intell. 32(07), 1854018 (2018). https://doi.org/10.1142/S0218001418540186

    Article  Google Scholar 

  32. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004). https://doi.org/10.1023/B:VISI.0000013087.49260

    Article  Google Scholar 

  33. Nezhadarya, E., Ward, R.K., & Wang, Z.J.: Wavelet-based gradient transform and its applications. In: IEEE 14th International Workshop on Multimedia Signal Processing (MMSP) (2012). https://doi.org/10.1109/mmsp.2012.6343424

  34. Datta, A., Ghosh, S., Ghosh, A.: PCA, Kernel PCA and dimensionality reduction in hyperspectral images. In: Advances in Principal Component Analysis, pp. 19–46 (2017) https://doi.org/10.1007/978-981-10-6704-4_2

  35. Sevakula, R.K., Verma, N.K.: Compounding general purpose membership functions for fuzzy support vector machine under noisy environment. IEEE Trans. Fuzzy Syst. (2017). https://doi.org/10.1109/tfuzz.2017.2722421

    Article  Google Scholar 

  36. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: 3rd IEEE International Conference on Automatic Face and Gesture Recognition, pp 200–205 (1998). https://doi.org/10.1109/afgr.1998.670949

  37. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z.: The extended cohn–kanadedataset (CK +): a complete dataset for action unit and emotion-specified expression. In: Proceedings of the Third International Workshop on CVPR for Human Communicative Behaviour Analysis (CVPR4HB 2010) (2010). https://doi.org/10.1109/cvprw.2010.5543262

  38. Wallhoff, F., Schuller, B., Hawellek, M., Rigoll, G.: Efficient Recognition of authentic dynamic facial expressions on the feedtum database. In: IEEE ICME, IEEE Computer Society, pp. 493–496 (2006). https://doi.org/10.1109/icme.2006.262433

  39. Goh, K.M., Ng, C.H., Lim, L.L., Sheikh, U.U.: Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. 36(3), 445–468 (2020). https://doi.org/10.1007/s00371-018-1607-6

    Article  Google Scholar 

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of ECE, Bethlahem Institute of Engineering, Karungal, Tamil Nadu, 629157, India

    R. Jeen Retna Kumar

  2. Department of ECE, VSB Engineering College, Karur, Tamil Nadu, 639111, India

    M. Sundaram

  3. Department of ECE, National Engineering College, Kovilpatti, Tamil Nadu, 628503, India

    N. Arumugam

Authors
  1. R. Jeen Retna Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Sundaram
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. Arumugam
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have made significant contributions to the research reported and have read and approved the submitted manuscript.

Corresponding author

Correspondence to R. Jeen Retna Kumar.

Ethics declarations

Conflict of interest

No potential conflict of interest was reported by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, R.J.R., Sundaram, M. & Arumugam, N. Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine. Vis Comput 37, 2315–2329 (2021). https://doi.org/10.1007/s00371-020-01988-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01988-1

Keywords

Search

Navigation