Skip to main content
SpringerLink
Log in

Multi-features Based Multi-layer Perceptron for Facial Expression Recognition System

  • Conference paper
  • First Online:
Second International Conference on Image Processing and Capsule Networks (ICIPCN 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 300))

Included in the following conference series:

  • 971 Accesses

Abstract

Facial Expression Recognition (FER) is a complex research topic in the computer vision field for the last decades. Several approaches and methods have been used to resolve the recognition problem. However, traditional and deep learning-based methods achieve improved performance but are limited to their features limiting the recognition rate. Therefore, the latest strategy is developed for the FER system based on multi-features extraction using the traditional method and deep learning method to reduce the intra-class variation and raise the inter-class dissimilarity. Here a Multi-Feature based MLP (MF-MLP) Classifier is proposed to concentrate on the facial appearance detection problem. First, the Multi-Feature Extraction method uses the LBP and ResNet-50 to extract the multi-features, texture features and high-level visual concepts from the images. Second, the Multi-Layer Perceptron Neural Network is proposed with the Fusion function, which fuses the output vectors generated by MLP to produce the final output vector representing the probability distribution of the expressions. Finally, the proposed MF-MLP Classifier model is skilled in an end-to-end fashion by using the fusion function to fine-tune the MLP. Extensive experiments have been conducted on both constrained datasets (CK+, JAFFE) and unconstrained datasets (FER2013) to demonstrate the effectiveness of the proposed MF-MLP model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Shan, C., Gong, S., McCowan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  2. Tariq, U., Yang, J., Huang, T.S.: Multi-view facial expression recognition analysis with generic sparse coding feature. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7585, pp. 578–588. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33885-4_58

    Chapter  Google Scholar 

  3. Tariq, U., et al.: Emotion recognition from an ensemble of features. In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition Workshops (FG), Santa Barbara, CA, USA, March 2011, pp. 872–877 (2011)

    Google Scholar 

  4. Liu, M., Shan, S., Wang, R., Chen, X.: Learning expressions via a universal manifold model for dynamic facial expression recognition. IEEE Trans. Image Process. 25(12), 5920–5932 (2016)

    Article  MathSciNet  Google Scholar 

  5. Fan, X., Tjahjadi, T.: A dynamic framework based on local Zernikemoment and motion history image for facial expression recognition. Pattern Recognit. 64, 399–406 (2017)

    Article  Google Scholar 

  6. Szegedyet, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR), Boston, MA, USA, June 2015, pp. 1–9 (2015)

    Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision Pattern Recognition (CVPR), Las Vegas, NV, USA, June 2016, pp. 770–778 (2016)

    Google Scholar 

  8. Yang, B., Cao, J., Ni, R., Zhang, Y.: Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6, 4630–4640 (2018)

    Article  Google Scholar 

  9. Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., Xun, E.: Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn. Comput. 9(5), 597–610 (2017). https://doi.org/10.1007/s12559-017-9472-6

    Article  Google Scholar 

  10. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2177 (2018)

    Google Scholar 

  11. Georgescu, M.I., Ionescu, R.T., Popescu, M.: Local learning with deep and handcraftedfeatures for facial expression recognition. arXiv preprint arXiv:1804.10892 (2018)

  12. Giannopoulos, P., Perikos, I., Hatzilygeroudis, I.: Deep learning approaches for facial emotion recognition: a case study on FER-2013. In: Hatzilygeroudis, I., Palade, V. (eds.) Advances in Hybridization of Intelligent Methods. SIST, vol. 85, pp. 1–16. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-66790-4_1

    Chapter  Google Scholar 

  13. Minaee, S., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. arXiv, vol. 1902.01019 (2019)

    Google Scholar 

  14. Zhao, H., Liu, Q., Yang, Y.: Transfer learning with an ensemble of multiple feature representations. In: 2018 IEEE16th International Conference on Software Engineering Research, Management, and Applications (SERA), IEEE (2018)

    Google Scholar 

  15. Feutry, C., Piantanida, P., Bengio, Y., Duhamel, P.: Learning anonymized representations with adversarial neural networks. arXiv preprint arXiv:1802.09386 (2018)

  16. Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)

    Article  Google Scholar 

  17. Shima, Y., Omori, Y.: Image augmentation for classifying facial expression images by using deep neural network pre-trained with object image database. In: Proceedings of the 3rd International Conference on Robotics, Control and Automation. ACM (2018)

    Google Scholar 

  18. Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition. In: IEEE International Conference on Automatic Face & Gesture Recognition, pp. 558–565. IEEE (2017)

    Google Scholar 

  19. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Li, Y.: Spatial temporal recurrent neural network for emotion recognition. IEEE Trans. Cybern. 99, 1–9 (2018)

    Google Scholar 

  20. Carrier, P.L., Courville, A., Goodfellow, I.J., Mirza, M., Bengio, Y.: FER-2013 face database. University de Montreal (2013)

    Google Scholar 

  21. Aneja, D., Colburn, A., Faigin, G., Shapiro, L., Mones, B.: Modeling stylized character expressions via deep learning. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10112, pp. 136–153. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54184-6_9

    Chapter  Google Scholar 

  22. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J.: The Japanese female facial expression (JAFFE) database. In: Third International Conference on Automatic Face and Gesture Recognition, pp. 14–16 (1998)

    Google Scholar 

  23. Lucey, P., et al.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE (2010)

    Google Scholar 

  24. Huang, D., Shan, C., Ardabilian, M., Wang, Y., Chen, L.: Local binary patterns and its application to facial image analysis: a survey. IEEE Trans. Syst. Man Cybern.—Part C: Appl. Rev. 41(6) (2011)

    Google Scholar 

  25. Huang, D., Wang, Y., Wang, Y.: A robust method for near-infrared face recognition based on extended local binary pattern. In: Proceeding of the International Symposium Vision Computing, pp. 437–446 (2007)

    Google Scholar 

  26. Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998)

    MATH  Google Scholar 

  27. Ekman, P., Erika, L.: What the Face reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press, USA (1997)

    Google Scholar 

  28. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, ppp. 3359–3368 (2018)

    Google Scholar 

  29. Hameed Siddiqi, M., Farooq, F., Lee, S.: A robust feature extraction method for human facial expressions recognition systems. In: Proceedings of the 27th Conference on Image and Vision Computing New Zealand, November 2012, pp. 464–468 (2012). https://doi.org/10.1145/2425836.2425924

  30. Owusu, E., Zhan, Y., Mao, Q.R.: An SVM-AdaBoost facial expression recognition system. Appl. Intell. 40(3), 536–545 (2013). https://doi.org/10.1007/s10489-013-0478-9

    Article  Google Scholar 

  31. Perikos, I., Ziakopoulos, E., Hatzilygeroudis, I.: Recognizing emotions from facial expressions using neural network. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H. (eds.) AIAI 2014. IAICT, vol. 436, pp. 236–245. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44654-6_23

    Chapter  Google Scholar 

  32. DamirFilko, G.M.: Emotion recognition system by a neural network-based facial expression analysis. Automatika 54(2), 263–272 (2013)

    Article  Google Scholar 

  33. Sharifzadeh, F., Akbarizadeh, G., Kavian, Y.S.: Ship classification in SAR ımages using a new hybrid CNN–MLP classifier. J. Indian Soc. Remote Sens. 47(4), 551–562 (2019)

    Article  Google Scholar 

  34. Gaikwad, N.B., Tiwari, V., Keskar, A., Shivaprakash, N.C.: Efficient FPGA ımplementation of multilayer perceptron for real-time human activity classification. IEEE Access 7, 26696–26706 (2019)

    Article  Google Scholar 

  35. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for ımage recognition (2015). https://arxiv.org/pdf/1512.03385.pdf

  36. Sankupellay, M., Konovalov, D.: Bird call recognition using deep convolutional neural network, ResNet-50. In: Proceedings of the Australian Acoustical Society Conference. 134. From: AAS2018: Acoustics 2018, 6–9 November 2018

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Gitam University, Bangalore, India

    Sneha Sureddy & Jeena Jacob

Authors
  1. Sneha Sureddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeena Jacob
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sneha Sureddy .

Editor information

Editors and Affiliations

  1. Department of Electrical Engineering, Da-Yeh University, Changhua, Taiwan

    Joy Iong-Zong Chen

  2. Departamento de Engenharia Mecanica, Universidade do Porto, Porto, Portugal

    João Manuel R. S. Tavares

  3. College of Engineering, Prince Sattan Bin Abdulaziz University, Al-Kharj, Saudi Arabia

    Abdullah M. Iliyasu

  4. Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, Canada

    Ke-Lin Du

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sureddy, S., Jacob, J. (2022). Multi-features Based Multi-layer Perceptron for Facial Expression Recognition System. In: Chen, J.IZ., Tavares, J.M.R.S., Iliyasu, A.M., Du, KL. (eds) Second International Conference on Image Processing and Capsule Networks. ICIPCN 2021. Lecture Notes in Networks and Systems, vol 300. Springer, Cham. https://doi.org/10.1007/978-3-030-84760-9_19

Download citation

Publish with us

Policies and ethics

Search

Navigation