Skip to main content
SpringerLink
Log in

Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In view of the high dimensionality, nonrigidity, multiscale variation and the influence of illumination and angle on facial expressions, it is quite difficult to obtain facial expression images or videos using computers and analyze facial morphology and changes to accurately obtain the emotional changes of the subjects. Existing facial expression recognition algorithms have the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. The facial expression recognition method based on deep learning has problems such as overfitting, gradient explosion and parameter initialization. Therefore, this paper develops a facial expression recognition algorithm based on the deep learning method. An adaptive model parameter initialization based on the multilayer maxout network linear activation function is proposed to initialize the convolutional neural network (CNN) and the long–short-term memory network (LSTM) method. It can effectively overcome the gradient disappearance and gradient explosion problems in the deep learning model training process. At the same time, the convolutional neural network with an LSTM memory unit is used to extract the related information from the image sequence, and the facial expression judgment is based on a single-frame image and historical-related information. However, the top-level structure of the CNN model is a fully connected feedforward neural network, which undertakes the task of expression classification. Therefore, the SVM classification method replaces the top-level classifier to further improve the expression classification accuracy. Experiments show that the facial expression recognition method proposed in this paper not only accurately identifies various expressions but also has good adaptive ability. This is because the method achieves the adaptive initialization of the parameters of the deep learning model construction process and also analyzes the relevance of the expression database expression, thereby improving the accuracy of expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Pransky, J.: The Pransky interview–Martin Haegele, Head of Department Robotics and Assistive Systems. Fraunhofer IPA. Ind. Robot Int. J. 45(3), 307–310 (2018). https://doi.org/10.1108/IR-04-2018-0060

    Article  Google Scholar 

  2. Vouloutsi, V., Verschure, P.F.M.J.: Emotions and self-regulation. Living Mach. Handb. Res. Biomim. Biohybrid Syst. 10, 327 (2018)

    Google Scholar 

  3. Pickett, L.: Don’t fear the cobot: collaborative robots, or cobots, are infiltrating factories on a global scale. But can robots and humans really work together in harmony? We asked the experts. Quality 57(1), 12A (2018)

    Google Scholar 

  4. Wu, Y., Schuster, M., Chen, Z. et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

  5. Mehrabian, A.: Communication without words. Commun. Theory 12, 193–200 (2008)

    Google Scholar 

  6. Deng, H.B., Jin, L.W., Zhen, L.X., et al.: A new facial expression recognition method based on local Gabor filter bank and PCA plus lda. Int. J. Inf. Technol. 11(11), 86–96 (2005)

    Google Scholar 

  7. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  8. Satiyan, M., Nagarajan, R., Hariharan, M.: Recognition of facial expression using Haar wavelet transform. Trans. Int. J. Electr. Electron. Syst. Res. JEESR Univ. Technol. Mara UiTM 3, 91–99 (2010)

    Google Scholar 

  9. Chen, J., Takiguchi, T., Ariki, Y.: Facial expression recognition with multithreaded cascade of rotation-invariant HOG. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, pp. 636–642 (2015)

  10. Soyel, H., Demirel, H.: Improved SIFT matching for pose robust facial expression recognition. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), IEEE, pp. 585–590 (2011)

  11. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ACM, pp. 435–442 (2015)

  12. Jung, H., Lee, S., Yim, J. et al.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

  13. Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24(1), 189–204 (2015)

    Article  MathSciNet  Google Scholar 

  14. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  15. Liu, M., Shan, S., Wang, R. et al.: Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1749–1756 (2014)

  16. Maninchedda, F., Oswald, M.R., Pollefeys, M.: Fast 3d reconstruction of faces with glasses. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 4608–4617 (2017)

  17. Kacem, A., Daoudi, M., Amor, B.B. et al.: A novel space-time representation on the positive semidefinite cone for facial expression recognition. In: ICCV, pp. 3199–3208 (2017)

  18. Liu, P., Han, S., Meng, Z. et al.: Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)

  19. Lopes, A.T., de Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, pp. 273–280 (2015)

  20. Zhang, F., Yu, Y., Mao, Q., et al.: Pose-robust feature learning for facial expression recognition. Front. Comput. Sci. 10(5), 832–844 (2016)

    Article  Google Scholar 

  21. Zhang, T.: Facial expression recognition based on deep learning: a survey. In: International Conference on Intelligent and Interactive Systems and Applications, Springer, Cham, pp. 345–352 (2017)

  22. Zhang, K., Huang, Y., Du, Y., et al.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)

    Article  MathSciNet  Google Scholar 

  23. Zhao, X., Liang, X., Liu, L., et al.: Peak-piloted deep network for facial expression recognition. In: European Conference on Computer Vision, Springer, Cham, pp. 425–442 (2016)

  24. Cao, C., Weng, Y., Zhou, S., et al.: Facewarehouse: a 3d facial expression database for visual computing. IEEE Trans. Vis. Comput. Gr. 20(3), 413–425 (2014)

    Article  Google Scholar 

  25. Yin, L., Wei, X., Sun, Y., et al.: A 3D facial expression database for facial behaviour research. In: 7th International Conference on Automatic Face and Gesture Recognition, FGR 2006, IEEE, pp. 211–216 (2006)

  26. Goodfellow, I.J., Erhan, D., Carrier, P.L., et al.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015)

    Article  Google Scholar 

  27. Zhao, G., Huang, X., Taini, M., et al.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Article  Google Scholar 

  28. Liu, M., Li, S., Shan, S., et al.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, Springer, Cham, pp. 143–157 (2014)

  29. Lopes, A.T., de Aguiar, E., De Souza, A.F., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)

    Article  Google Scholar 

  30. Ding, H., Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), IEEE, pp. 118–126 (2017)

Download references

Acknowledgements

This paper is supported by National Natural Science Foundation of China (No. 61701188).

Author information

Authors and Affiliations

  1. School of Physics and Electronic Electrical Engineering, Huaiyin Normal University, Huai’an, 223300, China

    Fengping An

  2. School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081, China

    Fengping An & Zhiwen Liu

Authors
  1. Fengping An
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiwen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fengping An.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

An, F., Liu, Z. Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Vis Comput 36, 483–498 (2020). https://doi.org/10.1007/s00371-019-01635-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01635-4

Keywords

Search

Navigation