In view of the high dimensionality, nonrigidity, multiscale variation and the influence of illumination and angle on facial expressions, it is quite difficult to obtain facial expression images or videos using computers and analyze facial morphology and changes to accurately obtain the emotional changes of the subjects. Existing facial expression recognition algorithms have the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. The facial expression recognition method based on deep learning has problems such as overfitting, gradient explosion and parameter initialization. Therefore, this paper develops a facial expression recognition algorithm based on the deep learning method. An adaptive model parameter initialization based on the multilayer maxout network linear activation function is proposed to initialize the convolutional neural network (CNN) and the long–short-term memory network (LSTM) method. It can effectively overcome the gradient disappearance and gradient explosion problems in the deep learning model training process. At the same time, the convolutional neural network with an LSTM memory unit is used to extract the related information from the image sequence, and the facial expression judgment is based on a single-frame image and historical-related information. However, the top-level structure of the CNN model is a fully connected feedforward neural network, which undertakes the task of expression classification. Therefore, the SVM classification method replaces the top-level classifier to further improve the expression classification accuracy. Experiments show that the facial expression recognition method proposed in this paper not only accurately identifies various expressions but also has good adaptive ability. This is because the method achieves the adaptive initialization of the parameters of the deep learning model construction process and also analyzes the relevance of the expression database expression, thereby improving the accuracy of expression recognition.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Pransky, J.: The Pransky interview–Martin Haegele, Head of Department Robotics and Assistive Systems. Fraunhofer IPA. Ind. Robot Int. J. 45(3), 307–310 (2018). https://doi.org/10.1108/IR-04-2018-0060
Vouloutsi, V., Verschure, P.F.M.J.: Emotions and self-regulation. Living Mach. Handb. Res. Biomim. Biohybrid Syst. 10, 327 (2018)
Pickett, L.: Don’t fear the cobot: collaborative robots, or cobots, are infiltrating factories on a global scale. But can robots and humans really work together in harmony? We asked the experts. Quality 57(1), 12A (2018)
Wu, Y., Schuster, M., Chen, Z. et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Mehrabian, A.: Communication without words. Commun. Theory 12, 193–200 (2008)
Deng, H.B., Jin, L.W., Zhen, L.X., et al.: A new facial expression recognition method based on local Gabor filter bank and PCA plus lda. Int. J. Inf. Technol. 11(11), 86–96 (2005)
Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)
Satiyan, M., Nagarajan, R., Hariharan, M.: Recognition of facial expression using Haar wavelet transform. Trans. Int. J. Electr. Electron. Syst. Res. JEESR Univ. Technol. Mara UiTM 3, 91–99 (2010)
Chen, J., Takiguchi, T., Ariki, Y.: Facial expression recognition with multithreaded cascade of rotation-invariant HOG. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, pp. 636–642 (2015)
Soyel, H., Demirel, H.: Improved SIFT matching for pose robust facial expression recognition. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), IEEE, pp. 585–590 (2011)
Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ACM, pp. 435–442 (2015)
Jung, H., Lee, S., Yim, J. et al.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)
Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24(1), 189–204 (2015)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Liu, M., Shan, S., Wang, R. et al.: Learning expression lets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1749–1756 (2014)
Maninchedda, F., Oswald, M.R., Pollefeys, M.: Fast 3d reconstruction of faces with glasses. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 4608–4617 (2017)
Kacem, A., Daoudi, M., Amor, B.B. et al.: A novel space-time representation on the positive semidefinite cone for facial expression recognition. In: ICCV, pp. 3199–3208 (2017)
Liu, P., Han, S., Meng, Z. et al.: Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)
Lopes, A.T., de Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, pp. 273–280 (2015)
Zhang, F., Yu, Y., Mao, Q., et al.: Pose-robust feature learning for facial expression recognition. Front. Comput. Sci. 10(5), 832–844 (2016)
Zhang, T.: Facial expression recognition based on deep learning: a survey. In: International Conference on Intelligent and Interactive Systems and Applications, Springer, Cham, pp. 345–352 (2017)
Zhang, K., Huang, Y., Du, Y., et al.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)
Zhao, X., Liang, X., Liu, L., et al.: Peak-piloted deep network for facial expression recognition. In: European Conference on Computer Vision, Springer, Cham, pp. 425–442 (2016)
Cao, C., Weng, Y., Zhou, S., et al.: Facewarehouse: a 3d facial expression database for visual computing. IEEE Trans. Vis. Comput. Gr. 20(3), 413–425 (2014)
Yin, L., Wei, X., Sun, Y., et al.: A 3D facial expression database for facial behaviour research. In: 7th International Conference on Automatic Face and Gesture Recognition, FGR 2006, IEEE, pp. 211–216 (2006)
Goodfellow, I.J., Erhan, D., Carrier, P.L., et al.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015)
Zhao, G., Huang, X., Taini, M., et al.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)
Liu, M., Li, S., Shan, S., et al.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, Springer, Cham, pp. 143–157 (2014)
Lopes, A.T., de Aguiar, E., De Souza, A.F., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)
Ding, H., Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), IEEE, pp. 118–126 (2017)
This paper is supported by National Natural Science Foundation of China (No. 61701188).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
An, F., Liu, Z. Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Vis Comput 36, 483–498 (2020). https://doi.org/10.1007/s00371-019-01635-4
- Model parameter initialization method
- Facial expression recognition
- Deep learning