Abstract
Recognition of facial micro-expressions (MEs), which indicates conscious or unconscious suppressing of true emotions, is still a challenging task in the affective computing and computer vision. There are two main reasons for that: First, the lack of spontaneous MEs databases, preferably focused on one emotion. So far, posed facial MEs databases were developed, and in the most cases, machines were trained on this posed MEs, which are stronger and more visible than spontaneous ones. Second, in order to achieve high recognition rate, deep learning structures are required that can achieve the best performance with very large number of data. To address these challenges, we make the following contributions: (i) extension of our MEs spontaneous database by adding new subjects; (ii) We analysed spontaneous MEs in long videos only for hidden sadness; (iii) We presented deeper analysis for automatic hidden sadness detection algorithm with deep learning architecture and compared results with standard machine learning techniques for hidden sadness detection. It is shown that with our method 99.08% recognition performance has been achieved observing only the eye region of the face.
Similar content being viewed by others
References
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst 16(3):1097–1105
Anbarjafari G (2013) Face recognition using color local binary pattern from mutually independent color channels. EURASIP J Adv Signal Process 2013(1):6
Baltrušaitis T, Robinson P, Morency L-P, An open source facial behavior analysis toolkit (2016). In: IEEE Winter conference on applications of computer vision
Birchfield S (1997) Derivation of kanade-lucas-tomasi tracking equation, unpublished notes
Bolotnikova A, Demirel H, Anbarjafari G (2017) Real-time ensemble based face recognition system for nao humanoids using local binary pattern. Analog Integr Circ Sig Process 92(3):467–475
Breiman L (2001) Random forests. Mach Learn 45(1):5–32
Chang C-C, Lin C-J (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):27
Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685
Corneanu C, Noroozi F, Kaminska D, Sapinski T, Escalera S, Anbarjafari G (2018) Survey on emotional body gesture recognition. In: IEEE transactions on affective computing
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297
Davison AK, Lansley C, Costen N, Tan K, Yap M H (2017) Samm: a spontaneous micro-facial movement dataset. In: IEEE transactions on affective computing
Demirel H, Anbarjafari G (2009) Data fusion boosted face recognition based on probability distribution functions in different colour channels. EURASIP J Adv Signal Process 2009:25
Eisenbarth GW, Hedwig, Alpers (2011) Happy mouth and sad eyes: scanning emotional facial expressions. Emotion 11 [Online]. Available: http://gen.lib.rus.ec/scimag/index.php?s=10.1037/a0022758
Ekman P Mett and sett online tools. [Online]. Available: http://www.ekmaninternational.com/paul-ekman-international-plc-home/training-courses/online-training.aspx
Ekman P, Friesen WV (1974) Nonverbal behavior and psychopathology. In: The psychology of depression: contemporary theory and research, p 3
Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Ishk
Fischer AH, Manstead AS, Zaalberg R (2003) Social influences on the emotion process. Eur Rev Soc Psychol 14(1):171–201
Fung G, Mangasarian OL (2000) Data selection for support vector machine classifiers. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 64–70
Gorbova J, Avots E, Lüsi I, Fishel M, Escalera S, Anbarjafari G (2018) Integrating vision and language for first-impression personality analysis. IEEE MultiMed 25(2):24–33
Gorbova J, Marjanović M, Njeguš A, Čolović M, Hasan D, Gholammreza A (2017) Automatic hidden sadness detection using micro-expressions. In: Proceedings from the 12th IEEE international conference on automatic face and gesture recognition (FG 2017). IEEE
Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior J C S J, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26391–26403
Guo Z, Zhang L, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663
Haamer RE, Kulkarni K, Imanpour N, Haque MA, Avots E, Breisch M, Nasrollahi K, Escalera S, Ozcinar C, Baro X et al (2018) Changes in facial expression as biometric: a database and benchmarks of identification. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 621–628
Haggard EA, Isaacs KS (1966) Micro-momentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy, pp 154–165
Heisele TPB, Ho P (2001) Face recognition with support vector machines: global versus component-based approach. In: IEEE international conference on computer vision
Howard A, Zhang C, Horvitz E (2017) Addressing bias in machine learning algorithms: a pilot study on emotion recognition for intelligent systems. In: 2017 IEEE workshop on advanced robotics and its social impacts (ARSO). IEEE, pp 1–7
Hsu S-C, Huang H-H, Huang C-L (2017) Facial expression recognition for human-robot interaction. In: IEEE international conference on robotic computing (IRC). IEEE, pp 1–7
Jia J, Xu Y, Zhang S, Xue X (2016) The facial expression recognition method of random forest based on improved pca extracting feature. In: 2016 IEEE international conference on signal processing, communications and computing (ICSPCC). IEEE, pp 1–5
Junior J, Jacques C, Ozcinar C, Marjanovic M, Baró X, Anbarjafari G, Escalera S (2019) On the effect of age perception biases for real age regression. arXiv preprint arXiv:1902.07653
Kulkarni K, Corneanu C, Ofodile I, Escalera S, Baró X, Hyniewska S, Allik J, Anbarjafari G (2018) Automatic recognition of facial displays of unfelt emotions. In: IEEE transactions on affective computing
Lévesque J, Eugène F, Joanette Y, Paquette V, Mensour B, Beaudoin G, Leroux J-M, Bourgouin P, Beauregard M (2003) Neural circuitry underlying voluntary suppression of sadness. Biol Psychiatry 53(6):502–510
Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG),. IEEE, pp 1–6
Li X, Yu J, Zhan S (2016) Spontaneous facial micro-expression detection based on deep learning. In: 2016 IEEE 13th international conference on signal processing (ICSP). IEEE, pp 1130–1134
Li X, Xiaopeng H, Moilanen A, Huang X, Pfister T, Zhao G, Pietikainen M (2017) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. In: IEEE transactions on affective computing
Liong S-T, See J, Wong K, Phan R C-W (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92
Liu Y-J, Zhang J-K, Yan W-J, Wang S-J, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101
Mansoorizadeh M, Charkari NM (2010) Multimodal information fusion application to human emotion recognition from face and speech. Multimed Tools Appl 49(2):277–297
Martinez B, Binefa X, Pantic M (2010) Facial component detection in thermal imagery. In: 2010 IEEE Computer Society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 48–54
Mstafa RJ, Elleithy KM (2016) A video steganography algorithm based on kanade-lucas-tomasi tracking algorithm and error correcting codes. Multimed Tools Appl 75(17):10311–10333
Parkhi OM, Vedaldi A, Zisserman A et al (2015) Deep face recognition. BMVC 1(3):6
Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE, pp 2258–2263
Pavlović M, Zlatanović L (2011) Facijalni znaci eksperimentalnoizazvane tuge kod ispitanika muškog i ženskog pola. Godisnjak za psihologiju 8(10):11–11
Pavlović M, Zlatanović L (2011) Facijalni znaci maskiranja, blokiranja i suzbijanja eksperimentalno izazvane emocije tuge. Godišnjak za psihologiju 8 (10):152–168
Pfister T, Li X, Zhao G, Pietikäinen M (2011) Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: IEEE international conference on computer vision workshops (ICCV Workshops)
Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 IEEE international conference on computer vision (ICCV). IEEE, pp 1449–1456
Qi Z, Tian Y, Shi Y (2013) Robust twin support vector machine for pattern classification. Pattern Recogn 46(1):305–316
Rahman A, Anam AI, Yeasin M (2017) Emoassist: emotion enabled assistive tool to enhance dyadic conversation for the blind. Multimed Tools Appl 76(6):7699–7730
Schmid PC, Schmid-Mast M (2010) Mood effects on emotion recognition. Motiv Emot 34:288–292
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Smith ML, Cottrell GW, Gosselin F, Schyns PG (2005) Transmitting and decoding facial expressions. Psychol Sci 16(3):184–189
Tertychnyi P, Ozcinar C, Anbarjafari G (2018) Low-quality fingerprint classification using deep neural network. IET Biom 7(6):550–556
Wang S-J, Yan W-J, Li X, Zhao G, Fu X (2014) Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition (ICPR). IEEE, pp 4678–4683
Wang S-J, Yan W-J, Sun T, Zhao G, Fu X (2016) Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214:218–232
Wang Y, See J, Oh Y-H, Phan R C-W, Rahulamathavan Y, Ling H-C, Tan S-W, Li X (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690
Warren G, Schertler E, Bull P (2009) Detecting deception from emotional and unemotional cues. J Nonverbal Behav 33(1):59–69
Wu Q, Shen X, Fu X (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. In: International conference on affective computing and intelligent interaction. Springer, pp 152–162
Yan W-J, Wu Q, Liang J, Chen Y-H, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230
Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X (2013) Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–7
Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1):e86041
Yang B, Cao J-M, Jiang D-P, Lv J-D (2017) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tools Appl 77:20477–20499
Acknowledgments
This work is supported Estonian Research Council Grant (PUT638), the Scientific and Technological Research Council of Turkey (TÜBITAK) (Project 1001 - 116E097), and the Estonian Centre of Excellence in IT (EXCITE) funded by the European Regional Development Fund. The authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X and Titan V Pascal GPUs. The author would like to thank Miss Dariia Temirova for helping with deep neural network codes.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Gorbova, J., Colovic, M., Marjanovic, M. et al. Going deeper in hidden sadness recognition using spontaneous micro expressions database. Multimed Tools Appl 78, 23161–23178 (2019). https://doi.org/10.1007/s11042-019-7658-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-7658-5