Abstract
Facial Expression evaluation has become necessary for human machine interaction, behavior analysis and also forensic and clinical evaluation. Deep convolutional neural networks (CNN) have been largely used for facial expression recognition but due to locality of convolution, CNNs results in lower accuracy when trained with facial expression data of varying ethnicity and emotion intensity. Recurrent neural networks (RNN) are used to work with sequential data and used to predict the sequences. We propose CNN-RNN network approach, a hybrid network, wherein the outputs from CNN and RNN have been concatenated to predict the final emotion, similarly a CNN model followed by a RNN layers has been designed that gives promising results. The proposed hybrid models are evaluated on two publically available datasets CK+ and JAFFE which provide us with variation in ethnicity and emotion intensity. Promising results have been obtained with this hybrid approach when compared to various machine learning and deep learning methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)
Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., Budynek, J.: The Japanese female facial expression (JAFFE) database (1998)
Ekman, P.: Pictures of Facial Affect. Consulting Psychologists Press, USA (1976)
Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: International Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, United States, pp. 886–893 (June 2005)
Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)
Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0054760
Ou, J., Bai, X.B., Pei, Y., Ma, L., Liu, W.: Automatic facial expression recognition using Gabor filter and expression analysis. In: IEEE International Conference on Computer Modeling and Simulation, pp. 215–218 (2010)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Szegedy, C., et al.: Going deeper with convolutions. In: CVPR (2015)
Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634 (2015)
Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3367–3375 (2015)
Wang, J., Yi, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2285–2294 (2016)
Zuo, Z., et al.: Convolutional recurrent neural networks: learning spatial dependencies for image representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 18–26 (2015)
Khorrami, P., Paine, T.L., Brady, K., Dagli, C., Huang, T.S.: How deep neural networks can improve emotion recognition on video data. In: IEEE Conference on Image Processing (ICIP) (2016)
Visin, F., Kastner, K., Cho, K., Matteucci, M., et al.: ReNet: a recurrent neural network based alternative to convolutional networks. arXiv:1505.00393 (2015)
Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)
Sanin, A., Sanderson, C., Harandi, M.T., Lovell, B.C.: Spatiotemporal covariance descriptors for action and gesture recognition. In: IEEE Workshop on Applications of Computer Vision (2013)
Jain, S., Hu, C., Aggarwal, J.K.: Facial expression recognition with temporal modeling of shapes. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1642–1649 (2011)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Connie, T., Al-Shabi, M., Cheah, W.P., Goh, M.: Facial expression recognition using a hybrid CNN–SIFT aggregator. In: Phon-Amnuaisuk, S., Ang, S.-P., Lee, S.-Y. (eds.) MIWAI 2017. LNCS (LNAI), vol. 10607, pp. 139–149. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69456-6_12
Kaya, H., Gürpınar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)
Lowe, D.G.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Le, Q.V., Jaitly, N., Hinton, G.E.: A simple way to initialize recurrent networks of rectified linear units. arXiv:1504.00941 (2015)
Minaee, S., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019 (2019)
Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research Management and Applications (SERA), pp. 123–128 (2017)
Kim, J.-H., Kim, B.-G., Roy, P.P., Jeong, D.-M.: Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7, 41273–41285 (2019)
Jain, N., et al.: Hybrid deep neural networks for face emotion recognition. Pattern Recogn. Lett. 115, 101–106 (2018)
Visin, F., Kastner, K., Courville, A., Bengio, Y., et al.: ReSeg: a recurrent neural network for object segmentation arXiv:1511.07053 (2015)
Patil, P., Murala, S.: FgGAN: a cascaded unpaired learning for background estimation and foreground segmentation. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1770–1778. IEEE (2019)
Patil, P., Murala, S., Dhall, A., Chaudhary, S.: MsEDNet: multi-scale deep saliency learning for moving object detection. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1670–1675. IEEE (2018)
Patil, P.W., Murala, S.: MSFgNet: a novel compact end-to-end deep network for moving object detection. IEEE Trans. Intell. Transp. Syst. 20(11), 4066–4077 (2018)
Dudhane, A., Murala, S.: CDNet: single image de-hazing using unpaired adversarial training. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1147–1155. IEEE (2019)
Dudhane, A., Murala, S.: C^2MSNet: a novel approach for single image haze removal. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1397–1404. IEEE (2018)
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8789–8797 (2018)
Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: The IEEE International Conference on Computer Vision (ICCV), pp. 2223–2232 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jadhav, A.B., Burewar, S.L., Waghumbare, A.A., Gonde, A.B. (2020). Deep Hybrid Neural Networks for Facial Expression Classification. In: Nain, N., Vipparthi, S., Raman, B. (eds) Computer Vision and Image Processing. CVIP 2019. Communications in Computer and Information Science, vol 1148. Springer, Singapore. https://doi.org/10.1007/978-981-15-4018-9_26
Download citation
DOI: https://doi.org/10.1007/978-981-15-4018-9_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-4017-2
Online ISBN: 978-981-15-4018-9
eBook Packages: Computer ScienceComputer Science (R0)