Skip to main content
SpringerLink
Log in

Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition

  • Chapter
  • First Online:
Advances in Face Detection and Facial Image Analysis

Abstract

In this chapter we consider the problem of automatic facial expression analysis. Our take on this is that the field has reached a point where it needs to move away from considering experiments and applications under in-the-lab conditions, and move towards so-called in-the-wild scenarios. We assume throughout this chapter that the aim is to develop technology that can be deployed in practical applications under unconstrained conditions. While some first efforts in this direction have been reported very recently, it is still unclear what the right path to achieving accurate, informative, robust, and real-time facial expression analysis will be. To illuminate the journey ahead, we first provide in Sect. 1 an overview of the existing theories and specific problem formulations considered within the computer vision community. Then we describe in Sect. 2 the standard algorithmic pipeline which is common to most facial expression analysis algorithms. We include suggestions as to which of the current algorithms and approaches are most suited to the scenario considered. In Sect. 3 we describe our view of the remaining challenges, and the current opportunities within the field. This chapter is thus not intended as a review of different approaches, but rather a selection of what we believe are the most suitable state-of-the-art algorithms, and a selection of exemplars chosen to characterise a specific approach. We review in Sect. 4 some of the exciting opportunities for the application of automatic facial expression analysis to everyday practical problems and current commercial applications being exploited. Section 5 ends the chapter by summarising the major conclusions drawn.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    While FACS does not explicitly define temporal phases, there’s a large amount of consensus on how to code them. See e.g. [132].

  2. 2.

    http://www.humansensing.cs.cmu.edu/intraface/.

  3. 3.

    It is interesting to note that different methods define the set of landmarks to be detected differently. The widely most common nowadays are the 49 landmarks depicted in Fig. 3, The 66 landmarks that result from adding 17 landmarks laid on the face contour, and the set of 68 landmarks that result from adding 2 extra landmarks on the inner lip mouth corners.

  4. 4.

    http://www.humansensing.cs.cmu.edu/intraface/.

References

  1. T. Almaev, M. Valstar, Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition, in Affective Computing and Intelligent Interaction (2013)

    Google Scholar 

  2. Z. Ambadar, J.F. Cohn, L.I. Reed, All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J. Nonverbal Behav. 33, 17–34 (2009)

    Article  Google Scholar 

  3. American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders (DSM), 5th edn. (American Psychiatric Association, Washington, 2013)

    Google Scholar 

  4. A.B. Ashraf, S. Lucey, J.F. Cohn, T. Chen, Z. Ambadar, K.M. Prkachin, P.E. Solomon, The painful face - pain expression recognition using active appearance models. Image Vis. Comput. 27(12), 1788–1796 (2009)

    Article  Google Scholar 

  5. A. Asthana, S. Zafeiriou, S. Cheng, M. Pantic, Incremental face alignment in the wild, in Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  6. M.S. Aung, S. Kaltwang, B. Romera-Paredes, B. Martinez, A. Singh, M. Cella, M. Valstar, H. Meng, A. Kemp, M. Shafizadeh, A.C. Elkins, N. Kanakam, A. de Rothschild, N. Tyler, P.J. Watson, A.C. de C. Williams, M. Pantic, N. Bianchi-Berthouze, The automatic detection of chronic pain-related expression: requirements, challenges and a multimodal dataset. Trans. Affect. Comput. In Press

    Google Scholar 

  7. M.R. Bagby, A.G. Ryder, D.R. Schuller, M.B. Marshall, The Hamilton depression rating scale: has the gold standard become a lead weight? Am. J. Psychiatry 161, 2163–2177 (2004)

    Article  Google Scholar 

  8. T. Baltrušaitis, P. Robinson, L.P. Morency, 3D constrained local model for rigid and non-rigid facial tracking, in Computer Vision and Pattern Recognition (2012)

    Google Scholar 

  9. T. Baltrusaitis, P. Robinson, L.P. Morency, Continuous conditional neural fields for structured regression, in European Conference on Computer Vision (2014), pp. 593–608

    Google Scholar 

  10. T. Baltrušaitis, M. Mahmoud, P. Robinson, Cross-dataset learning and person-specific normalisation for automatic action unit detection, in Facial Expression Recognition and Analysis Challenge Workshop (2015)

    Google Scholar 

  11. L.M. Batrinca, G. Stratou, A. Shapiro, L. Morency, S. Scherer, Cicero - towards a multimodal virtual audience platform for public speaking training, in International Conference on Intelligent Virtual Agents (2013), pp. 116–128

    Google Scholar 

  12. T. Baur, I. Damian, P. Gebhard, K. Porayska-Pomsta, E. Andre, A job interview simulation: Social cue-based interaction with a virtual character, in International Conference on Social Computing (2013), pp. 220–227

    Google Scholar 

  13. J. Bazzo, M. Lamar, Recognizing facial actions using Gabor wavelets with neutral face average difference, in Automatic Face and Gesture Recognition (2004)

    Google Scholar 

  14. S. Bilakhia, A. Nijholt, S. Petridis, M. Pantic, The MAHNOB mimicry database - a database of naturalistic human interactions. Pattern Recogn. Lett. 66, 52–61 (2015)

    Article  Google Scholar 

  15. M.B. Blaschko, C.H. Lampert, Learning to localize objects with structured output regression, in European Conference on Computer Vision (2008)

    Google Scholar 

  16. L. Bourdev, J. Malik, Poselets: body part detectors trained using 3d human pose annotations, in International Conference on Computer Vision (2009)

    Google Scholar 

  17. H. Brugman, A. Russel, Annotating multimedia/multi-modal resources with ELAN, in International Conference on Language Resources and Evaluation (2004)

    Google Scholar 

  18. X.P. Burgos-Artizzu, P. Perona, P. Dollár, Robust face landmark estimation under occlusion, in International Conference on Computer Vision (2013), pp. 1513–1520

    Google Scholar 

  19. X. Cao, Y. Wei, F. Wen, J. Sun, Face alignment by explicit shape regression, in Computer Vision and Pattern Recognition (2012), pp. 2887–2894

    Google Scholar 

  20. S. Cheng, S. Zafeiriou, A. Asthana, M. Pantic, 3D facial geometric features for constrained local models, in International Conference on Image Processing (2014)

    Google Scholar 

  21. S. Chew, P. Lucey, S. Lucey, J. Saragih, J. Cohn, S. Sridharan, Person-independent facial expression detection using constrained local models, in Automatic Face and Gesture Recognition (2011), pp. 915–920

    Google Scholar 

  22. W.S. Chu, F. Zhou, F. De la Torre, Unsupervised temporal commonality discovery, in European Conference on Computer Vision (2012)

    Google Scholar 

  23. I. Cohen, N. Sebe, A. Garg, L.S. Chen, T.S. Huang, Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)

    Article  Google Scholar 

  24. J.F. Cohn, P. Ekman, Measuring facial actions, in The New Handbook of Methods in Nonverbal Behavior Research, ed. by J.A. Harrigan, R. Rosenthal, K. Scherer (Oxford University Press, New York, 2005), pp. 9–64

    Google Scholar 

  25. J. Cohn, K. Schmidt, The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multiresolution Inf. Process. 2(2), 121–132 (2004)

    Article  Google Scholar 

  26. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, M. Schröder, FEELTRACE: an instrument for recording perceived emotion in real time, in ISCA Tutorial and Research Workshop on Speech and Emotion (2000)

    Google Scholar 

  27. K.D. Craig, C.J. Patrick, Facial expression during induced pain. J. Pers. Soc. Psychol. 48(4), 1080–1091 (1985)

    Article  Google Scholar 

  28. N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in, Computer Vision and Pattern Recognition (2005), pp. 886–893

    Google Scholar 

  29. M. Dantone, J. Gall, G. Fanelli, L.J.V. Gool, Real-time facial feature detection using conditional regression forests, in Computer Vision and Pattern Recognition (2012), pp. 2578–2585

    Google Scholar 

  30. C. Darwin, The Expression of the Emotions in Man and Animals (John Murray, London, 1872)

    Book  Google Scholar 

  31. K. Dautenhahn, Getting to know each other – artificial social intelligence for autonomous robots. Robot. Auton. Syst. 16(2), 333–356 (1995)

    Article  Google Scholar 

  32. K. Dautenhahn, Socially intelligent robots: dimensions of human–robot interaction. Philos. Trans. R. Soc. B 362(1480), 679–704 (2007)

    Article  Google Scholar 

  33. K. Dautenhahn, I. Werry, Towards interactive robots in autism therapy: background, motivation and challenges. Pragmat. Cogn. 12(1), 1–35 (2004)

    Article  Google Scholar 

  34. F. de Rosis, C. Pelachaud, I. Poggi, V. Carofiglio, B.D. Carolis, From Greta’s mind to her face: modelling the dynamics of affective states in a conversational embodied agent. Int. J. Hum. Comput. Stud. 59(1–2), 81–118 (2003)

    Article  Google Scholar 

  35. B.M. DePaulo, J.J. Lindsay, B.E. Malone, L. Muhlenbruck, K. Charlton, H. Cooper, Cues to deception. Psychol. Bull. 129(1), 74 (2003)

    Google Scholar 

  36. A. Dhall, R. Goecke, S. Lucey, T. Gedeon, Collecting large richly annotated facial-expression databases from movies. IEEE MultiMedia 19(3), 34–41 (2012)

    Article  Google Scholar 

  37. X. Ding, W.S. Chu, F.D. la Torre, J.F. Cohn, Q. Wang, Facial action unit event detection by cascade of tasks, in International Conference on Computer Vision (2013)

    Google Scholar 

  38. S. Du, Y. Tao, A. Martinez, Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), 1454–1462 (2014)

    Article  Google Scholar 

  39. P. Ekman, W.V. Friesen, Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Google Scholar 

  40. P. Ekman, W. Friesen, Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17, 124–129 (1971)

    Article  Google Scholar 

  41. P. Ekman, W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement (Consulting Psychologists, Palo Alto, 1978)

    Google Scholar 

  42. P. Ekman, W. Friesen, J.C. Hager, in Facial Action Coding System (A Human Face, Salt Lake City, 2002)

    Google Scholar 

  43. F. Eyben, S. Petridis, B. Schuller, G. Tzimiropoulos, S. Zafeiriou, M. Pantic, Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks, in International Conference on Acoustics, Speech and Signal Processing (2011), pp. 5844–5847

    Google Scholar 

  44. P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object detection with discriminatively trained part-based models. Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)

    Article  Google Scholar 

  45. T. Fong, I. Nourbakhsh, K. Dautenhahn, A survey of socially interactive robots. Robot. Auton. Syst. 42(3), 143–166 (2003)

    Article  MATH  Google Scholar 

  46. D. Gatica-Perez, Automatic nonverbal analysis of social interaction in small groups: a review. Image Vis. Comput. 27(12), 1775–1787 (2009)

    Article  Google Scholar 

  47. A. Gudi, H.E. Tasli, T.M. den Uyl, A. Maroulis, Deep learning based FACS action unit occurrence and intensity estimation, in Facial Expression Recognition and Analysis Challenge (2015)

    Google Scholar 

  48. H. Gunes, B. Schuller, Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis. Comput. 31(2), 120–136 (2013)

    Article  Google Scholar 

  49. T. Hassner, S. Harel, E. Paz, R. Enbar, Effective face frontalization in unconstrained images, in Computer Vision and Pattern Recognition (2015)

    Google Scholar 

  50. H. Hung, D. Gatica-Perez, Estimating cohesion in small groups using audio-visual nonverbal behavior. Trans. Multimedia 12(6), 563–575 (2010)

    Article  Google Scholar 

  51. H. Hung, Y. Huang, G. Friedland, D. Gatica-Perez, Estimating dominance in multi-party meetings using speaker diarization. IEEE Trans. Audio Speech Lang. Process. 19(4), 847–860 (2011)

    Article  Google Scholar 

  52. M.E. Jabon, J.N. Bailenson, E. Pontikakis, L. Takayama, C. Nass, Facial expression analysis for predicting unsafe driving behavior. IEEE Pervasive Comput. 10(4), 84–95 (2011)

    Article  Google Scholar 

  53. V. Jain, E. Learned-Miller, FDDB: a benchmark for face detection in unconstrained settings. Technical Report UM-CS-2010-009, University of Massachusetts, Amherst (2010)

    Google Scholar 

  54. S. Jaiwand, B. Martinez, M. Valstar, Learning to combine local models for facial action unit detection, in Facial Expression Recognition and Analysis Challenge, in conj. with Face and Gesture Recognition (2015)

    Google Scholar 

  55. Q. Ji, X. Yang, Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8(5), 357–377 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  56. H. Jia, A.M. Martinez, Support vector machines in face recognition with occlusions, in Computer Vision and Pattern Recognition (2009), pp. 136–141

    Google Scholar 

  57. B. Jiang, M.F. Valstar, M. Pantic, Action unit detection using sparse appearance descriptors in space-time video volumes, in Automatic Face and Gesture Recognition (2011), pp. 314–321

    Google Scholar 

  58. B. Jiang, B. Martinez, M. Pantic, Parametric temporal alignment for the detection of facial action temporal segments, in British Machine Vision Conference (2014)

    Google Scholar 

  59. B. Jiang, B. Martinez, M.F. Valstar, M. Pantic, Decision level fusion of domain specific regions for facial action recognition, in International Conference on Pattern Recognition (2014)

    Google Scholar 

  60. B. Jiang, M.F. Valstar, B. Martinez, M. Pantic, Dynamic appearance descriptor approach to facial actions temporal modelling. Trans. Cybern. 44(2), 161–174 (2014)

    Article  Google Scholar 

  61. B. Jiang, B. Martinez, M. Pantic, Automatic analysis of facial actions, a survey. Trans. Affect. Comput. (under review)

    Google Scholar 

  62. S. Kaltwang, O. Rudovic, M. Pantic, Continuous pain intensity estimation from facial expressions, in Advances in Visual Computing (Springer, Heidelberg, 2012), pp. 368–377

    Google Scholar 

  63. S. Kaltwang, S. Todorovic, M. Pantic, Latent trees for estimating intensity of facial action units, in Computer Vision and Pattern Recognition (2015)

    Google Scholar 

  64. M. Kipp, ANVIL - a generic annotation tool for multimodal dialogue, in European Conference on Speech Communication and Technology (2001), pp. 1367–1370

    Google Scholar 

  65. S. Koelstra, I. Patras, Fusion of facial expressions and EEG for implicit affective tagging. Image Vis. Comput. 31(2), 164–174 (2013)

    Article  Google Scholar 

  66. S. Koelstra, M. Pantic, I. Patras, A dynamic texture based approach to recognition of facial actions and their temporal models. Trans. Pattern Anal. Mach. Intell. 32(11), 1940–1954 (2010)

    Article  Google Scholar 

  67. N. Komodakis, Efficient training for pairwise or higher order CRFs via dual decomposition, in Computer Vision and Pattern Recognition (2011), pp. 1841–1848

    Google Scholar 

  68. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012)

    Google Scholar 

  69. I. Leite, G. Castellano, A. Pereira, C. Martinho, A. Paiva, Empathic robots for long-term interaction. Int. J. Soc. Robot. 6(3), 329–341 (2014)

    Article  Google Scholar 

  70. G. Littlewort, M.S. Bartlett, I. Fasel, J. Susskind, J. Movellan, Dynamics of facial expression extracted automatically from video, in Image and Vision Computing (2004), pp. 615–625

    Google Scholar 

  71. G. Littlewort, J. Whitehill, T. Wu, I.R. Fasel, M.G. Frank, J.R. Movellan, M.S. Bartlett, The computer expression recognition toolbox (CERT), in Automatic Face and Gesture Recognition (2011), pp. 298–305

    Google Scholar 

  72. M. Liu, S. Shan, R. Wang, X. Chen, Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition, in Computer Vision and Pattern Recognition (2014), pp. 1749–1756

    Google Scholar 

  73. P. Liu, S. Han, Z. Meng, Y. Tong, Facial expression recognition via a boosted deep belief network, in Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  74. P. Lucey, J.F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, K.M. Prkachin, Automatically detecting pain in video through facial action units. Trans. Syst. Man Cybern. B 41(3), 664–674 (2011)

    Article  Google Scholar 

  75. P. Lucey, J.F. Cohn, K.M. Prkachin, P.E. Solomon, I. Matthews, Painful data: the UNBC-McMaster shoulder pain expression archive database, in Automatic Face and Gesture Recognition (2011)

    Google Scholar 

  76. M. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, Coding facial expressions with Gabor wavelets, in Automatic Face and Gesture Recognition (1998)

    Google Scholar 

  77. A. Maalej, B.B. Amor, M. Daoudi, A. Srivastava, S. Berretti, Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recogn. 44(8), 1581–1589 (2011)

    Article  Google Scholar 

  78. B. Martinez, M.F. Valstar, L21-based regression and prediction accumulation across views for robust facial landmark detection. Image Vis. Comput. In press

    Google Scholar 

  79. B. Martinez, M.F. Valstar, X. Binefa, M. Pantic, Local evidence aggregation for regression based facial point detection. Trans. Pattern Anal. Mach. Intell. 35(5), 1149–1163 (2013)

    Article  Google Scholar 

  80. M. Mathias, R. Benenson, M. Pedersoli, L. van Gool, Face detection without bells and whistles, in European Conference on Computer Vision (2014)

    Google Scholar 

  81. D. Matsumoto, More evidence for the universality of a contempt expression. Motiv. Emot. 16, 363–368 (1992)

    Article  Google Scholar 

  82. I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, D. Zhang, Automatic analysis of multimodal group actions in meetings. Trans. Pattern Anal. Mach. Intell. 27(3), 305–317 (2005)

    Article  Google Scholar 

  83. D. McDuff, R. El Kaliouby, E. Kodra, R. Picard, Measuring voter’s candidate preference based on affective responses to election debates, in Affective Computing and Intelligent Interaction (2013), pp. 369–374

    Google Scholar 

  84. D. McDuff, R. Kaliouby, T. Senechal, A, Amr, J.F. Cohn, R. Picard, Affectiva-MIT facial expression dataset (AM-FED): naturalistic and spontaneous facial expressions collected in-the-wild, in Computer Vision and Pattern Recognition Workshop (2013), pp. 881–888

    Google Scholar 

  85. D. McDuff, R. El Kaliouby, T. Senechal, D. Demirdjian, R. Picard, Automatic measurement of ad preferences from facial responses gathered over the internet. Image Vis. Comput. 32(10), 630–640 (2014)

    Article  Google Scholar 

  86. D. McDuff, R. Kaliouby, J. Cohn, R. Picard, Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. Trans. Affect. Comput. 6, 223–235 (2015)

    Article  Google Scholar 

  87. G. McKeown, I. Sneddon, Modeling continuous self-report measures of perceived emotion using generalized additive mixed models. Psychol. Methods 19(1), 155–74 (2014)

    Article  Google Scholar 

  88. G. McKeown, M. Valstar, R. Cowie, M. Pantic, M. Schroder, The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3, 5–17 (2012). doi:http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.20

    Google Scholar 

  89. L. Morency, I. de Kok, J. Gratch, Context-based recognition during human interactions: automatic feature selection and encoding dictionary, in International Conference on Multimodal Interaction (2008), pp. 181–188

    Google Scholar 

  90. R. Navarathna, P. Lucey, P. Carr, E. Carter, S. Sridharan, I. Matthews, Predicting movie ratings from audience behaviors, in IEEE Winter Conference on Applications of Computer Vision (2014), pp. 1058–1065

    Google Scholar 

  91. L.S. Nguyen, A. Marcos-Ramiro, M.M. Romera, D. Gatica-Perez, Multimodal analysis of body communication cues in employment interviews, in International Conference on Multimodal Interaction (2013), pp. 437–444

    Google Scholar 

  92. M.A. Nicolaou, H. Gunes, M. Pantic, Output-associative RVM regression for dimensional and continuous emotion prediction. Image Vis. Comput. 30(3), 186–196 (2012)

    Article  Google Scholar 

  93. M.A. Nicolaou, V. Pavlovic, M. Pantic, Dynamic probabilistic CCA for analysis of affective behaviour and fusion of continuous annotations. Trans. Pattern Anal. Mach. Intell. 36(7), 1299–1311 (2014)

    Article  Google Scholar 

  94. T. Ojala, M. Pietikainen, D. Harwood, A comparative study of texture measures with classification based on featured distribution. Pattern Recogn. 29(1), 51–59 (1996)

    Article  Google Scholar 

  95. J. Orozco, B. Martinez, M. Pantic, Empirical analysis of cascade deformable models for multi-view face detection. Image Vis. Comput. 42, 47–61 (2015)

    Article  Google Scholar 

  96. I. Pavlidis, N.L. Eberhardt, J.A. Levine, Human behaviour: seeing through the face of deception. Nature 415(6867), 35–35 (2002)

    Article  Google Scholar 

  97. P. Perakis, G. Passalis, T. Theoharis, I. Kakadiaris, 3D facial landmark detection under large yaw and expression variations. Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2013)

    Article  Google Scholar 

  98. S. Petridis, M. Pantic, Audiovisual discrimination between laughter and speech, in International Conference on Acoustics, Speech and Signal Processing (2008), pp. 5117–5120

    Google Scholar 

  99. S. Petridis, B. Martinez, M. Pantic, The MAHNOB laughter database. Image Vis. Comput. 31(2), 186–202 (2013)

    Article  Google Scholar 

  100. J.H. Pfeifer, M. Iacoboni, J.C. Mazziotta, M. Dapretto, Mirroring others’ emotions relates to empathy and interpersonal competence in children. NeuroImage 39(4), 2076–2085 (2008)

    Article  Google Scholar 

  101. T. Pfister, X. Li, G. Zhao, M. Pietikäinen, Recognising spontaneous facial micro-expressions, in International Conference on Computer Vision (2011), pp. 1449–1456

    Google Scholar 

  102. R.W. Picard, Affective Computing (MIT, Cambridge, 1997)

    Book  Google Scholar 

  103. K.M. Prkachin, P.E. Solomon, The structure, reliability and validity of pain expression: evidence from patients with shoulder pain. Pain 139, 267–274 (2008)

    Article  Google Scholar 

  104. O. Rudovic, M. Pantic, Shape-constrained Gaussian process regression for facial-point-based head-pose normalization, in International Conference on Computer Vision (2011), pp. 1495–1502

    Google Scholar 

  105. J.A. Russell, A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)

    Article  Google Scholar 

  106. G. Sandbach, S. Zafeiriou, M. Pantic, Binary pattern analysis for 3D facial action unit detection, in The British Machine Vision Conference (2012)

    Google Scholar 

  107. G. Sandbach, S. Zafeiriou, M. Pantic, Markov random field structures for facial action unit intensity estimation, in International Conference on Computer Vision Workshop (2013)

    Google Scholar 

  108. J.M. Saragih, S. Lucey, J.F. Cohn, Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  109. E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: a survey of registration, representation and recognition. Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

    Article  Google Scholar 

  110. K. Scherer, P. Ekman, Handbook of Methods in Nonverbal Behavior Research (Cambridge University Press, Cambridge, 1982)

    Google Scholar 

  111. M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. ter Maat, G. pain, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M.F. Valstar, M. Wöllmer, Building autonomous sensitive artificial listeners. Trans. Affect. Comput. 3(2), 165–183 (2012)

    Google Scholar 

  112. T. Senechal, V. Rapp, H. Salam, R. Seguier, K. Bailly, L. Prevost, Facial action recognition combining heterogeneous features via multi-kernel learning. IEEE Trans. Syst. Man Cybern. B 42(4), 993–1005 (2012)

    Article  Google Scholar 

  113. T. Sha, M. Song, J. Bu, C. Chen, D. Tao, Feature level analysis for 3D facial expression recognition. Neurocomputing 74(12–13), 2135–2141 (2011)

    Article  Google Scholar 

  114. C. Shan, S. Gong, P. McOwan, Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  115. P.E. Shrout, J.L. Fleiss, Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86(2), 420–428 (1979)

    Article  Google Scholar 

  116. T. Simon, M.H. Nguyen, F.D.L. Torre, J. Cohn, Action unit detection with segment-based SVMs, in Computer Vision and Pattern Recognition (2010), pp. 2737–2744

    Google Scholar 

  117. M. Soleymani, M. Pantic, Human-centered implicit tagging: overview and perspectives, in International Conference on Systems, Man, and Cybernetics (2012), pp. 3304–3309

    Google Scholar 

  118. M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, A multimodal database for affect recognition and implicit tagging. Trans. Affect. Comput. 3(1), 42–55 (2012)

    Article  Google Scholar 

  119. M. Soleymani, M. Larson, T. Pun, A. Hanjalic, Corpus development for affective video indexing. Trans. Multimedia 16(4), 1075–1089 (2014)

    Article  Google Scholar 

  120. M.J.L. Sullivan, P. Thibault, A. Savard, R. Catchlove, J. Kozey, W.D. Stanish, The influence of communication goals and physical demands on different dimensions of pain behavior. Pain 125(3), 270–277 (2006)

    Article  Google Scholar 

  121. X. Sun, J. Lichtenauer, M. Valstar, A. Nijholt, M. Pantic, A multimodal database for mimicry analysis, in Affective Computing and Intelligent Interaction (2011), pp. 367–376

    Google Scholar 

  122. M. Takahashi, M. Naemura, M. Fujii, S. Satoh, Estimation of attentiveness of people watching TV based on their emotional behaviors, in Affective Computing and Intelligent Interaction (2013), pp. 809–814

    Google Scholar 

  123. H. Tang, T. Huang, 3D facial expression recognition based on properties of line segments connecting facial feature points, in Automatic Face and Gesture Recognition (2008)

    Google Scholar 

  124. E. Taralova, F. De la Torre, M. Hebert, Motion words for video, in European Conference on Computer Vision (2014)

    Google Scholar 

  125. D. Tax, M.F. Valstar, M. Pantic, E. Hendrix, The detection of concept frames using clustering multi-instance learning, in International Conference on Pattern Recognition (2010), pp. 2917–2920

    Google Scholar 

  126. Y. Tong, J. Chen, Q. Ji, A unified probabilistic framework for spontaneous facial action modeling and understanding. Trans. Pattern Anal. Mach. Intell. 32(2), 258–273 (2010)

    Article  Google Scholar 

  127. F. Tsalakanidou, S. Malassiotis, Real-time 2D+3D facial action and expression recognition. Pattern Recogn. 43(5), 1763–1775 (2010)

    Article  Google Scholar 

  128. P. Tsiamyrtzis, J. Dowdall, D. Shastri, I. Pavlidis, M. Frank, P. Ekman, Imaging facial physiology for the detection of deceit. Int. J. Comput. Vis. 71(2), 197–214 (2007)

    Article  Google Scholar 

  129. G. Tzimiropoulos, Project-out cascaded regression with an application to face alignment, in Computer Vision and Pattern Recognition (2015), pp. 3659–3667

    Google Scholar 

  130. G. Tzimiropoulos, M. Pantic, Gauss-Newton deformable part models for face alignment in-the-wild, in Computer Vision and Pattern Recognition (2014), pp. 1851–1858

    Google Scholar 

  131. M. Valstar, Automatic behaviour understanding in medicine, in Workshop on Roadmapping the Future of Multimodal Interaction Research, including Business Opportunities and Challenges, RFMIR@ICMI (2014), pp. 57–60

    Google Scholar 

  132. M. Valstar, M. Pantic, Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. B 42(1), 28–43 (2012)

    Article  Google Scholar 

  133. M. Valstar, I. Patras, M. Pantic, Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data, in Computer Vision and Pattern Recognition Workshops (2005)

    Google Scholar 

  134. M.F. Valstar, B. Martinez, X. Binefa, M. Pantic, Facial point detection using boosted regression and graph models, in Computer Vision and Pattern Recognition (2010), pp. 2729–2736

    Google Scholar 

  135. M.F. Valstar, M. Mehu, B. Jiang, M. Pantic, K. Scherer, Meta – analysis of the first facial expression recognition challenge. IEEE Trans. Syst. Man Cybern. B 42(4), 966–979 (2012)

    Article  Google Scholar 

  136. M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, M. Pantic, AVEC 2014: 3D dimensional affect and depression recognition challenge, in International Workshop on Audio/Visual Emotion Challenge (2014), pp. 3–10

    Google Scholar 

  137. M.F. Valstar, T. Almaev, J.M. Girard, G. McKeown, M. Mehu, L. Yin, M. Pantic, J.F. Cohn, FERA 2015 - second facial expression recognition and analysis challenge, in Automatic Face and Gesture Recognition Workshop (2015)

    Google Scholar 

  138. L. van der Maaten, E. Hendriks, Action unit classification using active appearance models and conditional random fields. Cogn. Process. 13(2), 507–518 (2012)

    Article  Google Scholar 

  139. L. van der Maaten, M. Chen, S. Tyree, K.Q. Weinberger, Learning with marginalized corrupted features, in International Conference on Machine Learning (2013), pp. 410–418

    Google Scholar 

  140. A. Vinciarelli, M. Pantic, H. Bourlard, Social signal processing: survey of an emerging domain. Image Vis. Comput. 27(12), 1743–1759 (2009)

    Article  Google Scholar 

  141. A. Vinciarelli, M. Pantic, D. Heylen, C. Pelachaud, I. Poggi, F. D’Errico, M. Schröder, M.: Bridging the gap between social animal and unsocial machine: a survey of social signal processing. Trans. Affect. Comput. 3(1), 69–87 (2012)

    Google Scholar 

  142. P. Viola, M.J. Jones, Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  143. E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, J. Movellan, Drowsy driver detection through facial movement analysis, in IEEE International Conference on Human-Computer Interaction (2007), pp. 6–18

    Google Scholar 

  144. Z. Wang, Y. Li, S. Wang, Q. Ji, Capturing global semantic relationships for facial action unit recognition, in International Conference on Computer Vision (2013), pp. 3304–3311

    Google Scholar 

  145. S. Wang, Z. Liu, Y. Zhu, M. He, X. Chen, Q. Ji, Implicit video emotion tagging from audiences’ facial expression. Multimedia Tools Appl. 74(13), 4679–4706 (2015)

    Article  Google Scholar 

  146. G. Warren, E. Schertler, P. Bull, Detecting deception from emotional and unemotional cues. J. Nonverbal Behav. 33(1), 59–69 (2009)

    Article  Google Scholar 

  147. F. Weninger, Introducing CURRENNT: the munich open-source CUDA recurrent neural network toolkit. J. Mach. Learn. Res. 16, 547–551 (2015)

    MathSciNet  MATH  Google Scholar 

  148. J. Whitehill, Z. Serpell, Y. Lin, A. Foster, J.R. Movellan, The faces of engagement: automatic recognition of student engagement from facial expressions. Trans. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  149. M. Wöllmer, A. Metallinou, F. Eyben, B. Schuller, S.S. Narayanan, Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling, in Interspeech (2010), pp. 2362–2365

    Google Scholar 

  150. Q. Wu, X. Shen, X. Fu, The machine knows what you are hiding: an automatic micro-expression recognition system, in Affective Computing and Intelligent Interaction (2011), pp. 152–162

    Google Scholar 

  151. X. Xiong, F. De la Torre, Supervised descent method and its applications to face alignment, in Computer Vision and Pattern Recognition (2013)

    Google Scholar 

  152. J. Yan, Z. Lei, D. Yi, S.Z. Li, Learn to combine multiple hypotheses for accurate face alignment, in International Conference on Computer Vision Workshop (2013), pp. 392–396

    Google Scholar 

  153. W. Yan, Q. Wu, Y. Liu, S. Wang, X. Fu, CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces, in Automatic Face and Gesture Recognition (2013)

    Google Scholar 

  154. J. Yan, X. Zhang, Z. Lei, S.Z. Li, Face detection by structural models. Image Vis. Comput. 32(10), 790–799 (2014)

    Article  Google Scholar 

  155. P. Yang, Q. Liu, D.N. Metaxas, Boosting encoded dynamic features for facial expression recognition. Pattern Recogn. Lett. 30(2), 132–139 (2009)

    Article  Google Scholar 

  156. X. Yu, Z. Lin, J. Brandt, D. Metaxas, Consensus of regression for occlusion-robust facial feature localization, in European Conference on Computer Vision (2014), pp. 105–118

    Google Scholar 

  157. Z. Zeng, M. Pantic, G. Roisman, T.S. Huang et al., A survey of affect recognition methods: audio, visual, and spontaneous expressions. Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)

    Article  Google Scholar 

  158. X. Zhang, L. Yin, J.F. Cohn, Three dimensional binary edge feature representation for pain expression analysis, in Automatic Face and Gesture Recognition (2015)

    Google Scholar 

  159. G. Zhao, M. Pietikainen, Dynamic texture recognition using local binary patterns with an application to facial expressions. Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  160. L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, D.N. Metaxas, Learning active facial patches for expression analysis, in Computer Vision and Pattern Recognition (2012), pp. 2562–2569

    Google Scholar 

  161. X. Zhu, D. Ramanan, Face detection, pose estimation, and landmark localization in the wild, in Computer Vision and Pattern Recognition (2012), pp. 2879–2886

    Google Scholar 

  162. M. Zimmerman, I. Chelminski, M. Posternak, A review of studies of the Hamilton depression rating scale in healthy controls: implications for the definition of remission in treatment studies of depression. J. Nerv. Ment. Dis. 192(9), 595–601 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

The work of Dr. Valstar and Dr. Martinez is funded by European Union Horizon 2020 research and innovation programme under grant agreement No. 645378. The work of Dr. Valstar is also supported by MindTech Healthcare Technology Co-operative (NIHR-HTC).

Author information

Authors and Affiliations

  1. School of Computer Science, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, UK

    Brais Martinez & Michel F. Valstar

Authors
  1. Brais Martinez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michel F. Valstar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brais Martinez .

Editor information

Editors and Affiliations

  1. Automatic Control, Silesian University of Technology, Kawulok, Poland

    Michal Kawulok

  2. Computer Science, Louisiana State University in Shreveport, Shreveport, Louisiana, USA

    M. Emre Celebi

  3. Silesian University of Technology, Gliwice, Poland

    Bogdan Smolka

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Martinez, B., Valstar, M.F. (2016). Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition. In: Kawulok, M., Celebi, M., Smolka, B. (eds) Advances in Face Detection and Facial Image Analysis. Springer, Cham. https://doi.org/10.1007/978-3-319-25958-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25958-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-25956-7

  • Online ISBN: 978-3-319-25958-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation