Signal, Image and Video Processing

, Volume 13, Issue 3, pp 609–616 | Cite as

Fusing multi-stream deep neural networks for facial expression recognition

  • Fatima Zahra Salmam
  • Abdellah Madani
  • Mohamed KissiEmail author
Original Paper


Among the factors contributing to conveying emotional state of an individual is facial expression. It represents the most important nonverbal communication and a challenging task in the field of computer vision. In this work, we propose a combined deep architecture model for facial expression recognition that uses appearance and geometric features extracted separately using convolution layers and supervised decent method, respectively. The proposed model is trained on three public databases [the Extended Cohn Kanade (CK+), the OULU-CASIA VIS, and the JAFFE]. The three databases contain a limited amount of data that we enlarge by adding a step of data augmentation to original images. For further comparison, two additional models that use appearance features only and geometric features only are trained on the same subset of data, to show how the combination of the two deep architectures influences results. On the other hand, in order to investigate the generalization of the combined model, a cross-database evaluation is performed. The obtained results achieve the state-of-the-art and improve recent work, especially in case of cross-database evaluation.


Facial expression Supervised descent method, correlation feature selection Best first Convolution neural network, deep neural network 


  1. 1.
    Kamarol, S.K.A., Jaward, M.H., Kälviäinen, H., Parkkinen, J., Parthiban, R.: Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recognit. Lett. 92, 25 (2017)CrossRefGoogle Scholar
  2. 2.
    Mao, Q., Rao, Q., Yu, Y., Dong, M.: Hierarchical Bayesian theme models for multipose facial expression recognition. IEEE Trans. Multimed. 19(4), 861 (2017)CrossRefGoogle Scholar
  3. 3.
    Li, J., Zhang, D., Zhang, J., Zhang, J., Li, T., Xia, Y., Yan, Q., Xun, L.: Facial expression recognition with faster R-CNN. Procedia Comput. Sci 107, 135 (2017)CrossRefGoogle Scholar
  4. 4.
    Mehrabian, A.: Communication without words. Commun. Theory 1, 193–200 (2011)Google Scholar
  5. 5.
    Ekman, P.: An argument for basic emotions. Cognit. Emot. 6(3–4), 169 (1992)CrossRefGoogle Scholar
  6. 6.
    Mohammadian, A., Aghaeinia, H., Towhidkhah, F.: Incorporating prior knowledge from the new person into recognition of facial expression. Signal Image Video Process. 10(2), 235 (2016)CrossRefGoogle Scholar
  7. 7.
    Yurtkan, K., Demirel, H.: Entropy-based feature selection for improved 3D facial expression recognition. Signal Image Video Process. 8(2), 267 (2014)CrossRefGoogle Scholar
  8. 8.
    Ashir, A.M., Eleyan, A.: Facial expression recognition based on image pyramid and single-branch decision tree. Signal Image Video Process. 11(6), 1017 (2017)CrossRefGoogle Scholar
  9. 9.
    Zarbakhsh, P., Demirel, H.: Low-rank sparse coding and region of interest pooling for dynamic 3D facial expression recognition. Signal Image Video Process. 12(8), 1611–1618 (2018)CrossRefGoogle Scholar
  10. 10.
    Liu, P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  11. 11.
    LeCun, Y., Haffner, P., Bottou, L., Bengio, Y.: Object recognition with gradient-based learning. In: Shape, Contour and Grouping in Computer Vision, vol. 1681, pp. 319–345. Springer (1999)Google Scholar
  12. 12.
    Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610 (2017)CrossRefGoogle Scholar
  13. 13.
    Sánchez, A., Ruiz, J.V., Moreno, A.B., Montemayor, A.S., Hernández, J., Pantrigo, J.J.: Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74(8), 1272 (2011)CrossRefGoogle Scholar
  14. 14.
    Fan, X., Tjahjadi, T.: A dynamic framework based on local Zernike moment and motion history image for facial expression recognition. Pattern Recognit. 64, 399 (2017)CrossRefGoogle Scholar
  15. 15.
    Pu, X., Fan, K., Chen, X., Ji, L., Zhou, Z.: Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168, 1173 (2015)CrossRefGoogle Scholar
  16. 16.
    Cruz, E.A.S., Jung, C.R., Franco, C.H.E.: Facial expression recognition using temporal POEM features. Pattern Recognit. Lett. 114, 13–21 (2018)CrossRefGoogle Scholar
  17. 17.
    Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2983–2991. IEEE (2015)Google Scholar
  18. 18.
    Mlakar, U., Potočnik, B.: Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal Image Video Process. 9(1), 245 (2015)CrossRefGoogle Scholar
  19. 19.
    Sun, Z., Hu, Z., Chiong, R., Wang, M., Zhao, S.: An adaptive weighted fusion model with two subspaces for facial expression recognition. Signal Image Video Process. 12(5), 835 (2018)CrossRefGoogle Scholar
  20. 20.
    Barman, A., Dutta, P.: Facial expression recognition using distance and shape signature features. Pattern Recognit. Lett. (2017).
  21. 21.
    Xiong, X., Torre, F.: Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)Google Scholar
  22. 22.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. 1–511. IEEE (2001)Google Scholar
  23. 23.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929 (2014)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Zeiler, M.D.: ADADELTA: an adaptive learning rate method. arXiv:1212.5701 (2012)
  25. 25.
    Salmam, F.Z., Madani, A., Kissi, M.: Facial expression recognition using decision trees. In: 2016 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV), pp. 125–130. IEEE (2016)Google Scholar
  26. 26.
    Hall, M.A.: Correlation-based feature selection for machine learning, University of Waikato Hamilton (1999)Google Scholar
  27. 27.
    Devi, M.I., Rajaram, R., Selvakuberan, K.: Generating best features for web page classification. Webology 5(1), 52 (2008)Google Scholar
  28. 28.
    Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1 (2015)CrossRefGoogle Scholar
  29. 29.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)Google Scholar
  30. 30.
    Zhao, G., Huang, X., Taini, M., Li, S.Z., Pietikäinen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607 (2011)CrossRefGoogle Scholar
  31. 31.
    Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In:Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp 200–205 (1998).
  32. 32.
    Chollet, F., et al.: Keras. (2015)
  33. 33.
    De la Torre, F., Chu, W.S., Xiong, X., Vicente, F., Ding, X., Cohn, J.: IntraFace. In: IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, vol. 1. NIH Public Access (2015)Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  • Fatima Zahra Salmam
    • 1
  • Abdellah Madani
    • 1
  • Mohamed Kissi
    • 2
    Email author
  1. 1.LAROSERI Laboratory, Department of Computer Science, Faculty of SciencesChouaib Doukkali UniversityEl JadidaMorocco
  2. 2.LIM Laboratory, Department of Computer Science, Faculty of Sciences and TechnologiesHassan II University-CasablancaMohammediaMorocco

Personalised recommendations