Region-based facial representation for real-time Action Units intensity detection across datasets

Theoretical Advances

Abstract

Most research on facial expressions recognition has focused on binary Action Units (AUs) detection, while graded changes in their intensity have rarely been considered. This paper proposes a method for the real-time detection of AUs intensity in terms of the Facial Action Coding System scale. It is grounded on a novel and robust anatomically based facial representation strategy, for which features are registered from a different region of interest depending on the AU considered. Real-time processing is achieved by combining Histogram of Gradients descriptors with linear kernel Support Vector Machines. Following this method, AU intensity detection models are built and validated through the DISFA database, outperforming previous approaches without real-time capabilities. An in-depth evaluation through three different databases (DISFA, BP4D and UNBC Shoulder-Pain) further demonstrates that the proposed method generalizes well across datasets. This study also brings insights about existing public corpora and their impact on AU intensity prediction.

Keywords

Facial expressions recognition FACS Action Units intensity detection Cross-dataset validation Real-time processing 

References

  1. 1.
    Agarwal S, Umer S (2015) Mp-feg: media player controlled by facial expressions and gestures. In: 5th National conference on computer vision, pattern recognition, image processing and graphics, pp 1–4Google Scholar
  2. 2.
    Baltrušaitis T, Mahmoud M, Robinson P (2015) Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: 11th IEEE international conference and workshops on automatic face and gesture recognition, vol 6, pp 1–6Google Scholar
  3. 3.
    Bingol D, Celik T, Omlin CW, Vadapalli HB (2014) Facial action unit intensity estimation using rotation invariant features and regression analysis. In: 2014 IEEE international conference on image processing, pp 1381–1385Google Scholar
  4. 4.
    Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: 5th Annual workshop on computational learning theory, pp 144–152. ACMGoogle Scholar
  5. 5.
    Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
  6. 6.
    Chew SW, Lucey P, Lucey S, Saragih J, Cohn JF, Matthews I, Sridharan S (2012) In the pursuit of effective affective computing: the relationship between features and registration. IEEE Trans Syst Man Cybern B Cybern 42(4):1006–1016CrossRefGoogle Scholar
  7. 7.
    Chu WS, Torre F, Cohn J (2013) Selective transfer machine for personalized facial action unit detection. In: IEEE conference on computer vision and pattern recognition, pp 3515–3522Google Scholar
  8. 8.
    Cohn JF, Ambadar Z, Ekman P (2007) Observer-based measurement of facial expression with the facial action coding system. The handbook of emotion elicitation and assessment, pp 203–221Google Scholar
  9. 9.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. IEEE Comput Soc Conf Comput Vis Pattern Recognit 1:886–893Google Scholar
  10. 10.
    De Moor K, Mazza F, Hupont I, Quintero MR, Mäki T, Varela M (2014) Chamber QoE: a multi-instrumental approach to explore affective aspects in relation to quality of experience. In: IS&T/SPIE electronic imaging. International society for optics and photonicsGoogle Scholar
  11. 11.
    Flores VC (2005) Artnatomy. www.artnatomia.net
  12. 12.
    Gehrig T, Al-Halah Z, Ekenel HK, Stiefelhagen R (2015) Action unit intensity estimation using hierarchical partial least squares. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition, vol 1, pp 1–6Google Scholar
  13. 13.
    Girard JM, Cohn JF, Mahoor MH, Mavadati S, Rosenwald DP (2013) Social risk and depression: evidence from manual and automatic facial expression analysis. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition, pp 1–8Google Scholar
  14. 14.
    Girard JM, Cohn JF, De la Torre F (2015) Estimating smile intensity: a better way. Pattern Recogn Lett 66:13–21CrossRefGoogle Scholar
  15. 15.
    Gonzalez I, Verhelst W, Oveneke M, Sahli H, Jiang D (2015) Framework for combination aware au intensity recognition. In: 2015 International conference on affective computing and intelligent interaction, pp 602–608Google Scholar
  16. 16.
    Hager JC, Ekman P, Friesen WV (2002) Facial action coding system. A Human Face, Salt Lake CityGoogle Scholar
  17. 17.
    Happy S, George A, Routray A (2012) A real time facial expression classification system using local binary patterns. In: 4th IEEE international conference on intelligent human computer interaction, pp 1–5Google Scholar
  18. 18.
    Hupont I, Cerezo E, Baldassarri S (2008) Facial emotional classifier for natural interaction. ELCVIA Electron Lett Comput Vis Image Anal 7(4):1–12Google Scholar
  19. 19.
    Jiang B, Martinez B, Valstar MF, Pantic M (2014) Decision level fusion of domain specific regions for facial action recognition. In: 2014 22nd international conference on pattern recognition, pp 1776–1781Google Scholar
  20. 20.
    Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: 2011 IEEE international conference on automatic face and gesture recognition and workshops, pp 314–321Google Scholar
  21. 21.
    Kaltwang S, Todorovic S, Pantic M (2016) Doubly sparse relevance vector machine for continuous facial behavior estimation. IEEE Trans Pattern Anal Mach Intell 38(9):1748–1761CrossRefGoogle Scholar
  22. 22.
    Li Y, Mavadati SM, Mahoor MH, Ji Q (2013) A unified probabilistic framework for measuring the intensity of spontaneous facial action units. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition, pp 1–7Google Scholar
  23. 23.
    Li Y, Mavadati SM, Mahoor MH, Zhao Y, Ji Q (2015) Measuring the intensity of spontaneous facial action units with dynamic Bayesian network. Pattern Recogn 48(11):3417–3427CrossRefGoogle Scholar
  24. 24.
    Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (2011) The computer expression recognition toolbox (CERT). In: 2011 IEEE international conference on automatic face and gesture recognition and workshops, pp 298–305Google Scholar
  25. 25.
    Littlewort GC, Bartlett MS, Lee K (2009) Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis Comput 27(12):1797–1803CrossRefGoogle Scholar
  26. 26.
    Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: the UNBC-McMaster shoulder pain expression archive database. In: 2011 IEEE international conference on automatic face and gesture recognition and workshops, pp 57–64Google Scholar
  27. 27.
    Mahoor MH, Cadavid S, Messinger DS, Cohn, JF (2009) A framework for automated measurement of the intensity of non-posed facial action units. In: IEEE computer society conference on computer vision and pattern recognition workshops, pp 74–80Google Scholar
  28. 28.
    Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) DISFA: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160CrossRefGoogle Scholar
  29. 29.
    Mohammadi MR, Fatemizadeh E, Mahoor MH (2016) Intensity estimation of spontaneous facial action units based on their sparsity properties. IEEE Trans Cybern 46(3):817–826CrossRefGoogle Scholar
  30. 30.
    Nakamura T, Maejima A, Morishima S (2014) Driver drowsiness estimation from facial expression features computer vision feature investigation using a CG model. In: 2014 IEEE international conference on computer vision theory and applications, vol 2, pp 207–214Google Scholar
  31. 31.
    Nicolle J, Bailly K, Chetouani M (2015) Facial action unit intensity prediction via hard multi-task metric learning for kernel regression. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition, vol 6, pp 1–6Google Scholar
  32. 32.
    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830MathSciNetMATHGoogle Scholar
  33. 33.
    Ren H, Li ZN (2014) Age estimation based on complexity-aware features. In: Computer vision–ACCV 2014, pp 115–128. SpringerGoogle Scholar
  34. 34.
    Rudovic O, Pavlovic V, Pantic M (2015) Context-sensitive dynamic ordinal regression for intensity estimation of facial action units. IEEE Trans Pattern Anal Mach Intell 37(5):944–958CrossRefGoogle Scholar
  35. 35.
    Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37(6):1113–1133CrossRefGoogle Scholar
  36. 36.
    Savran A, Sankur B, Bilge MT (2012) Regression-based intensity estimation of facial action units. Image Vis Comput 30(10):774–784CrossRefGoogle Scholar
  37. 37.
    Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86(2):420CrossRefGoogle Scholar
  38. 38.
    Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system–a case study. In: 2014 IEEE conference on computer vision and pattern recognition workshops, pp 132–137Google Scholar
  39. 39.
    Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system-a case study. In: IEEE conference on computer vision and pattern recognition workshops, pp 132–137Google Scholar
  40. 40.
    Tian Yl, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115CrossRefGoogle Scholar
  41. 41.
    Valstar MF, Almaev T, Girard JM, McKeown G, Mehu M, Yin L, Pantic M, Cohn JF (2015) FERA 2015-second facial expression recognition and analysis challenge. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition, vol 6, pp 1–8Google Scholar
  42. 42.
    Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Trans Syst Man Cybern B Cybern 42(1):28–43CrossRefGoogle Scholar
  43. 43.
    Velusamy S, Gopalakrishnan V, Anand B, Moogi P, Pandey B (2013) Improved feature representation for robust facial action unit detection. In: IEEE consumer communications and networking conference, pp 681–684Google Scholar
  44. 44.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154CrossRefGoogle Scholar
  45. 45.
    Werner P, Saxen F, Al-Hamadi A (2015) Handling data imbalance in automatic facial action intensity estimation. FERA, p 26Google Scholar
  46. 46.
    Wu T, Butko NJ, Ruvolo P, Whitehill J, Bartlett MS, Movellan JR (2011) Action unit recognition transfer across datasets. In: 2011 IEEE international conference on automatic face and gesture recognition and workshops, pp 889–896Google Scholar
  47. 47.
    Xiong X, Torre F (2013) Supervised descent method and its applications to face alignment. In: IEEE conference on computer vision and pattern recognition, pp 532–539Google Scholar
  48. 48.
    Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P, Girard JM (2014) BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis Comput 32(10):692–706CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd. 2017

Authors and Affiliations

  1. 1.Institut des Systèmes Intelligents et de RobotiqueUniversité Pierre et Marie Curie (Sorbonne Universités)ParisFrance

Personalised recommendations