Abstract
Expressions play an imperative part in human interactions as it let humans express their intentions and feelings without words. Mental state of a subject can be mined from the expression extracted. In recent times, deep neural network has outperformed traditional handcrafted descriptors including spatiotemporal local binary pattern (LBP), LBP-TOP, HOG, etc., when it comes to feature extraction for facial expression recognition (FER). In this paper, an amalgam of convolution neural network (CNN) and long short-term memory (LSTM) [recurrent neural network (RNN)] is employed to extract essential features for recognizing the expression from the target frame. To increase the performance, transfer learning concept is engaged to get learned parameters (weight/bias). To accomplish transfer learning, leading layers of ResNet-50 (trained on thousands of image frames) are used. Further, a LSTM layer (time distributed) is affixed to the existing model. The model is further trained (CK+ database) with different activation functions, and a relative analysis is performed. Maximum accuracy of 94% is attained with the hybrid model (CNN-LSTM with SELU and ELU).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Abbreviations
- LBP:
-
Local Binary Pattern
- FER:
-
Facial Expression Recognition
- CNN:
-
Convolution Neural-Network
- LSTM:
-
Long Short-Term Memory
- RNN:
-
Recurrent Neural Network
- CK+:
-
Cohn-Kanade+
- ROC:
-
Receiver Operating Characteristic
- AUC:
-
Area Under Curve
- MTCNN:
-
Multitask Cascaded Neural Network
References
Mukherjee S, Vamshi B, Reddy KSVK, Krishna RV, Harish SVS (2016) Recognizing facial expressions using novel motion based features. In: Proceedings of the tenth Indian conference on computer vision, graphics and image processing. ACM, pp 1–8
Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M (2017) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577
Huang X, Wang SJ, Zhao G, Piteikainen M (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–9
Lamba PS, Virmani D (2018) Information retrieval from emotions and eye blinks with help of sensor nodes. Int J Electr Comput Eng (IJECE) 8(4):2433–2441
Lamba PS, Virmani D (2019) Information retrieval from facial expression using voting to assert exigency. J Discrete Math Sci Crypt 22(2):177–190
Wang Y, See J, Phan RCW, Oh YH (2014) LBP with six intersection points: reducing redundant information in LBP-top for micro-expression recognition. In: Asian conference on computer vision, pp 525–537. Springer, Cham
Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Guo J, Zhou S, Wu J, Wan J, Zhu X, Lei Z, Li SZ (2017) Multi-modality network with visual and geometrical information for micro emotion recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017), pp 814–819
Gan YS, Liong ST, Yau WC, Huang YC, Tan LK (2019) Off-apexnet on micro-expression recognition system. Sig Process Image Commun 74:129–139
Satya V (2018) Revealing true emotions through micro-expressions: a machine learning approach. CMU SEI Insights
Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR), pp 2258–2263. IEEE
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Mollahosseini A, Hasani B, Salvador MJ, Abdollahi H, Chan D, Mahoor MH (2016) Facial expression recognition from world wild web. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 58–65
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on machine learning, pp 448–456
Graves A, Mohamed AR, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 6645–6649. IEEE
Graves A, Jaitly N (2014) Towards end-to-end speech recognition with recurrent neural networks. In: International conference on machine learning, pp 1764–1772
Mikolov T, Karafiat M, Burget L, Cernocky J, Khudanpur S (2010) Recurrent neural network based language model. In: Proceedings of INTERSPEECH, pp 1045–1048
Sanin A., Sanderson C, Harandi MT, Lovell BC (2013) Spatio-temporal covariance descriptors for action and gesture recognition. In: 2013 IEEE workshop on applications of computer vision (WACV), pp 103–110. IEEE
Visin F, Kastner K, Cho K, Matteucci M, Courville A, Bengio Y (2015) Renet: a recurrent neural network based alternative to convolutional networks. arXiv preprint arXiv:1505.00393
Visin F, Kastner K, Courville A, Bengio Y, Matteucci M, Cho K (2015) Reseg: a recurrent neural network for object segmentation. arXiv preprint arXiv:1511.07053
Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial–temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49(3):839–847
Khorrami P, Le Paine T, Brady K, Dagli C, Huang TS (2016) How deep neural networks can improve emotion recognition on video data. In: 2016 IEEE international conference on image processing (ICIP), pp 619–623. IEEE
Jain DK, Kumar R, Jain N (2017) Decision-based spectral embedding approach for identifying facial behaviour on RGB-D images. In: Proceedings of international conference on communication and networks, pp 677–687. Springer, Singapore
Jain DK, Zhang Z, Huang K (2016) Hybrid patch based diagonal pattern geometric appearance model for facial expression recognition. In: Chinese conference on intelligent visual surveillance, pp 107–113. Springer, Singapore
Chernykh V, Prikhodko P (2017) Emotion recognition from speech with recurrent neural networks. arXiv preprint arXiv:1701.08071
Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp. 445–450
Xie S, Hu H (2017) Facial expression recognition with FRR-CNN. Electron Lett 53(4):235–237
Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. arXiv preprint arXiv:1408.3750
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lamba, P.S., Virmani, D. (2021). CNN-LSTM-Based Facial Expression Recognition. In: Abraham, A., Castillo, O., Virmani, D. (eds) Proceedings of 3rd International Conference on Computing Informatics and Networks. Lecture Notes in Networks and Systems, vol 167. Springer, Singapore. https://doi.org/10.1007/978-981-15-9712-1_32
Download citation
DOI: https://doi.org/10.1007/978-981-15-9712-1_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9711-4
Online ISBN: 978-981-15-9712-1
eBook Packages: EngineeringEngineering (R0)