Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM

  • Fengping AnEmail author
  • Zhiwen Liu
Original Article


In view of the high dimensionality, nonrigidity, multiscale variation and the influence of illumination and angle on facial expressions, it is quite difficult to obtain facial expression images or videos using computers and analyze facial morphology and changes to accurately obtain the emotional changes of the subjects. Existing facial expression recognition algorithms have the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. The facial expression recognition method based on deep learning has problems such as overfitting, gradient explosion and parameter initialization. Therefore, this paper develops a facial expression recognition algorithm based on the deep learning method. An adaptive model parameter initialization based on the multilayer maxout network linear activation function is proposed to initialize the convolutional neural network (CNN) and the long–short-term memory network (LSTM) method. It can effectively overcome the gradient disappearance and gradient explosion problems in the deep learning model training process. At the same time, the convolutional neural network with an LSTM memory unit is used to extract the related information from the image sequence, and the facial expression judgment is based on a single-frame image and historical-related information. However, the top-level structure of the CNN model is a fully connected feedforward neural network, which undertakes the task of expression classification. Therefore, the SVM classification method replaces the top-level classifier to further improve the expression classification accuracy. Experiments show that the facial expression recognition method proposed in this paper not only accurately identifies various expressions but also has good adaptive ability. This is because the method achieves the adaptive initialization of the parameters of the deep learning model construction process and also analyzes the relevance of the expression database expression, thereby improving the accuracy of expression recognition.


Model parameter initialization method CNN Facial expression recognition Deep learning LSTM SVM 



This paper is supported by National Natural Science Foundation of China (No. 61701188).


  1. 1.
    Pransky, J.: The Pransky interview–Martin Haegele, Head of Department Robotics and Assistive Systems. Fraunhofer IPA. Ind. Robot Int. J. 45(3), 307–310 (2018). CrossRefGoogle Scholar
  2. 2.
    Vouloutsi, V., Verschure, P.F.M.J.: Emotions and self-regulation. Living Mach. Handb. Res. Biomim. Biohybrid Syst. 10, 327 (2018)Google Scholar
  3. 3.
    Pickett, L.: Don’t fear the cobot: collaborative robots, or cobots, are infiltrating factories on a global scale. But can robots and humans really work together in harmony? We asked the experts. Quality 57(1), 12A (2018)Google Scholar
  4. 4.
    Wu, Y., Schuster, M., Chen, Z. et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
  5. 5.
    Mehrabian, A.: Communication without words. Commun. Theory 12, 193–200 (2008)Google Scholar
  6. 6.
    Deng, H.B., Jin, L.W., Zhen, L.X., et al.: A new facial expression recognition method based on local Gabor filter bank and PCA plus lda. Int. J. Inf. Technol. 11(11), 86–96 (2005)Google Scholar
  7. 7.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  8. 8.
    Satiyan, M., Nagarajan, R., Hariharan, M.: Recognition of facial expression using Haar wavelet transform. Trans. Int. J. Electr. Electron. Syst. Res. JEESR Univ. Technol. Mara UiTM 3, 91–99 (2010)Google Scholar
  9. 9.
    Chen, J., Takiguchi, T., Ariki, Y.: Facial expression recognition with multithreaded cascade of rotation-invariant HOG. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, pp. 636–642 (2015)Google Scholar
  10. 10.
    Soyel, H., Demirel, H.: Improved SIFT matching for pose robust facial expression recognition. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), IEEE, pp. 585–590 (2011)Google Scholar
  11. 11.
    Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ACM, pp. 435–442 (2015)Google Scholar
  12. 12.
    Jung, H., Lee, S., Yim, J. et al.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)Google Scholar
  13. 13.
    Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24(1), 189–204 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Liu, M., Shan, S., Wang, R. et al.: Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1749–1756 (2014)Google Scholar
  16. 16.
    Maninchedda, F., Oswald, M.R., Pollefeys, M.: Fast 3d reconstruction of faces with glasses. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 4608–4617 (2017)Google Scholar
  17. 17.
    Kacem, A., Daoudi, M., Amor, B.B. et al.: A novel space-time representation on the positive semidefinite cone for facial expression recognition. In: ICCV, pp. 3199–3208 (2017)Google Scholar
  18. 18.
    Liu, P., Han, S., Meng, Z. et al.: Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)Google Scholar
  19. 19.
    Lopes, A.T., de Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, pp. 273–280 (2015)Google Scholar
  20. 20.
    Zhang, F., Yu, Y., Mao, Q., et al.: Pose-robust feature learning for facial expression recognition. Front. Comput. Sci. 10(5), 832–844 (2016)CrossRefGoogle Scholar
  21. 21.
    Zhang, T.: Facial expression recognition based on deep learning: a survey. In: International Conference on Intelligent and Interactive Systems and Applications, Springer, Cham, pp. 345–352 (2017)Google Scholar
  22. 22.
    Zhang, K., Huang, Y., Du, Y., et al.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Zhao, X., Liang, X., Liu, L., et al.: Peak-piloted deep network for facial expression recognition. In: European Conference on Computer Vision, Springer, Cham, pp. 425–442 (2016)Google Scholar
  24. 24.
    Cao, C., Weng, Y., Zhou, S., et al.: Facewarehouse: a 3d facial expression database for visual computing. IEEE Trans. Vis. Comput. Gr. 20(3), 413–425 (2014)CrossRefGoogle Scholar
  25. 25.
    Yin, L., Wei, X., Sun, Y., et al.: A 3D facial expression database for facial behaviour research. In: 7th International Conference on Automatic Face and Gesture Recognition, FGR 2006, IEEE, pp. 211–216 (2006)Google Scholar
  26. 26.
    Goodfellow, I.J., Erhan, D., Carrier, P.L., et al.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015)CrossRefGoogle Scholar
  27. 27.
    Zhao, G., Huang, X., Taini, M., et al.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)CrossRefGoogle Scholar
  28. 28.
    Liu, M., Li, S., Shan, S., et al.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, Springer, Cham, pp. 143–157 (2014)Google Scholar
  29. 29.
    Lopes, A.T., de Aguiar, E., De Souza, A.F., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)CrossRefGoogle Scholar
  30. 30.
    Ding, H., Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), IEEE, pp. 118–126 (2017)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Physics and Electronic Electrical EngineeringHuaiyin Normal UniversityHuai’anChina
  2. 2.School of Information and ElectronicsBeijing Institute of TechnologyBeijingChina

Personalised recommendations