Abstract
The sample category distribution of spontaneous facial micro-expression datasets is unbalanced, due to the experimental environment, collection equipment, and individualization of subjects, which brings great challenges to micro-expression recognition. Therefore, this paper introduces a micro-expression recognition model based on the Hierarchical Support Vector Machine (H-SVM) to reduce the interference of sample category distribution imbalance. First, we calculated the position of the apex frame in the micro-expression image sequence. To keep micro-expression frames balanced, we sparsely sample the images sequence according to the apex frame. Then, the Low-level Descriptors of the region of interest of the micro-expression image sequence and the High-level Descriptors of apex frame are extracted. Finally, the H-SVM model is used to classify the fusion features of different levels. The experimental results on SMIC, CAMSE2, SAMM, and their composite datasets show that our method can achieve superior performance in micro-expression recognition.
Similar content being viewed by others
References
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297
Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions Affective Computing 9(1):116–129
Davison AK, Yap MH, Costen N, Tan K, Lansley C, Leightley D (2014). Micro-facial movements: an investigation on spatio-temporal descriptors. In: European conference on computer vision. Springer, pp 111–123
Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106
Fan DP, Zhang YH, Wu YH, Liu Y, Cheng MM, Ren B, Rosin P, Ji R (2019). Scoot: a perceptual metric for facial sketches. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE, pp 5611–5621
Fernandez A, Garcia S, Herrera F, Chawla NV (2018) Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. J Artif Intell Res 61:863–905
Gan YS, Liong ST, Yau WC, Huang YC, Tan LK (2019) OFF-ApexNet on micro-expression recognition system. Signal Process Image Commun 74:129–139
Grinsven MJ, Ginneken B, Hoyng CB, Theelen T, Sánchez CI (2016) Fast convolutional neural network training using selective data sampling: application to hemorrhage detection in color fundus images. IEEE Trans Med Imaging 35(5):1273–1284
Haggard EA, Isaacs KS (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, pp 154–165
He H, Garcia EA (2008) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284
Huang X, Wang SJ, Zhao G, Piteikainen M (2015). Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops (ICCV workshops). IEEE, pp. 1–9
Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578
Khan SA, Naveed R, Sheeraz A, Shahzad L (2015) Facial expression recognition using computationally intelligent techniques. J Intell Fuzzy Syst 28(6):2881–2887
Kim DH, Wissam JB, Yong MR (2016). Micro-expression recognition with expression-state constrained Spatio-temporal feature representations. In: Proceedings of the 24th ACM international conference on multimedia. ACM, pp 382–386
Le Ngo AC, Phan RCW, See J (2014). Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision (ACCV). Springer, pp 33–48
Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikainen M (2015). Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv:http://arXiv.org/abs/1511.00423
Li X, Pfister T, Huang X, Zhao G, Pietikainen M (2013). A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–6
Li Q, Zhan S, Xu L, Wu C (2018) Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimed Tools Appl 78(20):29307–29322
Liong ST, See J, Wong K, Le Ngo AC, Oh YH, Phan R (2015). Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian conference on pattern recognition (ACPR). IEEE, pp 665–669
Liong ST, See J, Wong K, Phan RCW (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92
Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310
Ma Y, Chen W, Ma X, Xu J, Huang X, Ross M, Anthony KH (2017) EasySVM: a visual analysis approach for open-box support vector machines. Computational Visual Media 3(2):161–175
Matsumoto D, Hwang HS (2011) Evidence for training the ability to read micro expressions of emotion. Motiv Emot 35(2):181–191
Merghani W, Davison AK, Yap M (2018). Facial micro-expressions grand challenge 2018: evaluating spatio-temporal features for classification of objective classes. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 662–666
Merghani W, Davison AK, Yap MH (2019) The implication of spatial temporal changes on facial micro-expression analysis. Multimed Tools Appl 78(15):21613–21628
MunirA AH, Khan SA, Muhammad N, Sadia A (2018) Illumination invariant facial expression recognition using selected merged binary patterns for real world images. Optik 158:1016–1025
Nie, GY, Cheng MM, Liu Y, Liang Z, Fan DP, Liu Y, Wang Y (2019) Multi-level context ultra-aggregation for stereo matching. In: 2019 IEEE/CVF conference on computer vision and pattern recognition. IEEE, pp 3283–3291
Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59
Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence 7:971–987
OSullivan M, Frank MG, Hurley CM, Tiwana J (2009) Police lie detection accuracy: the effect of lie scenario. Law Hum Behav 33(6):530–538
Patel D, Hong X, Zhao G (2016). Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR 2016). IEEE, pp 2258–2263
Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745–1745
Pfister T, Li X, Zhao G, Pietikainen M (2011). Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops). IEEE, pp 868–875
Polikovsky S, Kameda, Y, Ohta Y (2009). Facial micro-expressions recognition using high-speed camera and 3D-gradient descriptor. In: 3rd international conference on crime detection and prevention (ICDP 2009). IET, pp 1–6
Russell TA, Chu E, Phillips ML (2006) A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br J Clin Psychol 45(4):579–583
Salter F, Grammer K, Rikowski A (2005) Sex differences in negotiating with powerful males. Hum Nat 16(3):306–321
Shabat AM, Tapamo JR (2018) Angled local directional pattern for texture analysis with an application to facial expression recognition. IET Comput Vis 12(5):603–608
Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011). Macro-and micro-expression spotting in long videos using Spatio-temporal strain. In: 2011 IEEE international conference on automatic face gesture recognition and workshops (FG 2011), pp 51–56
Takalkar M A, Xu M (2017). Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 international conference on digital image computing: techniques and applications (DICTA 2017). IEEE, pp 1–7
Wang Y, See J, Phan RCW, Oh YH (2014). Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision (ACCV). Springer, pp 525–537
Weiss J, Ekman P (2009) Telling lies : clues to deceit in the marketplace, politics, and marriage. New York: Norton. Am J Clin Hypn 53(4):287–288
Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):1–8
Yan WJ, Wu Q, Liang J, Chen YH, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230
Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013). CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–7
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis & Machine Intelligence 6:915–928
Zhou Z, Zhao G, Pietikainen M (2011). Towards a practical lipreading system. In: Proceedings of the 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 137–144
Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia 20(11):3160–3172
Funding
This work was supported by the National Key R&D Program of China (No. 2018YFC 2001700), National Natural Science Foundation of China (No. 61672093), Beijing Municipal Natural Science Foundation (No. L192005), and Advanced Innovation Center for Intelligent Robots and Systems Open Research Project (No.2018IRS01).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Pan, H., Xie, L., Lv, Z. et al. Hierarchical support vector machine for facial micro-expression recognition. Multimed Tools Appl 79, 31451–31465 (2020). https://doi.org/10.1007/s11042-020-09475-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09475-4