Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition



In this chapter we consider the problem of automatic facial expression analysis. Our take on this is that the field has reached a point where it needs to move away from considering experiments and applications under in-the-lab conditions, and move towards so-called in-the-wild scenarios. We assume throughout this chapter that the aim is to develop technology that can be deployed in practical applications under unconstrained conditions. While some first efforts in this direction have been reported very recently, it is still unclear what the right path to achieving accurate, informative, robust, and real-time facial expression analysis will be. To illuminate the journey ahead, we first provide in Sect. 1 an overview of the existing theories and specific problem formulations considered within the computer vision community. Then we describe in Sect. 2 the standard algorithmic pipeline which is common to most facial expression analysis algorithms. We include suggestions as to which of the current algorithms and approaches are most suited to the scenario considered. In Sect. 3 we describe our view of the remaining challenges, and the current opportunities within the field. This chapter is thus not intended as a review of different approaches, but rather a selection of what we believe are the most suitable state-of-the-art algorithms, and a selection of exemplars chosen to characterise a specific approach. We review in Sect. 4 some of the exciting opportunities for the application of automatic facial expression analysis to everyday practical problems and current commercial applications being exploited. Section 5 ends the chapter by summarising the major conclusions drawn.


Facial Expression Facial Expression Recognition Face Appearance Multiple Kernel Learn Expressive Behaviour 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The work of Dr. Valstar and Dr. Martinez is funded by European Union Horizon 2020 research and innovation programme under grant agreement No. 645378. The work of Dr. Valstar is also supported by MindTech Healthcare Technology Co-operative (NIHR-HTC).


  1. 1.
    T. Almaev, M. Valstar, Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition, in Affective Computing and Intelligent Interaction (2013)Google Scholar
  2. 2.
    Z. Ambadar, J.F. Cohn, L.I. Reed, All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J. Nonverbal Behav. 33, 17–34 (2009)CrossRefGoogle Scholar
  3. 3.
    American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders (DSM), 5th edn. (American Psychiatric Association, Washington, 2013)Google Scholar
  4. 4.
    A.B. Ashraf, S. Lucey, J.F. Cohn, T. Chen, Z. Ambadar, K.M. Prkachin, P.E. Solomon, The painful face - pain expression recognition using active appearance models. Image Vis. Comput. 27(12), 1788–1796 (2009)CrossRefGoogle Scholar
  5. 5.
    A. Asthana, S. Zafeiriou, S. Cheng, M. Pantic, Incremental face alignment in the wild, in Computer Vision and Pattern Recognition (2014)Google Scholar
  6. 6.
    M.S. Aung, S. Kaltwang, B. Romera-Paredes, B. Martinez, A. Singh, M. Cella, M. Valstar, H. Meng, A. Kemp, M. Shafizadeh, A.C. Elkins, N. Kanakam, A. de Rothschild, N. Tyler, P.J. Watson, A.C. de C. Williams, M. Pantic, N. Bianchi-Berthouze, The automatic detection of chronic pain-related expression: requirements, challenges and a multimodal dataset. Trans. Affect. Comput. In PressGoogle Scholar
  7. 7.
    M.R. Bagby, A.G. Ryder, D.R. Schuller, M.B. Marshall, The Hamilton depression rating scale: has the gold standard become a lead weight? Am. J. Psychiatry 161, 2163–2177 (2004)CrossRefGoogle Scholar
  8. 8.
    T. Baltrušaitis, P. Robinson, L.P. Morency, 3D constrained local model for rigid and non-rigid facial tracking, in Computer Vision and Pattern Recognition (2012)Google Scholar
  9. 9.
    T. Baltrusaitis, P. Robinson, L.P. Morency, Continuous conditional neural fields for structured regression, in European Conference on Computer Vision (2014), pp. 593–608Google Scholar
  10. 10.
    T. Baltrušaitis, M. Mahmoud, P. Robinson, Cross-dataset learning and person-specific normalisation for automatic action unit detection, in Facial Expression Recognition and Analysis Challenge Workshop (2015)Google Scholar
  11. 11.
    L.M. Batrinca, G. Stratou, A. Shapiro, L. Morency, S. Scherer, Cicero - towards a multimodal virtual audience platform for public speaking training, in International Conference on Intelligent Virtual Agents (2013), pp. 116–128Google Scholar
  12. 12.
    T. Baur, I. Damian, P. Gebhard, K. Porayska-Pomsta, E. Andre, A job interview simulation: Social cue-based interaction with a virtual character, in International Conference on Social Computing (2013), pp. 220–227Google Scholar
  13. 13.
    J. Bazzo, M. Lamar, Recognizing facial actions using Gabor wavelets with neutral face average difference, in Automatic Face and Gesture Recognition (2004)Google Scholar
  14. 14.
    S. Bilakhia, A. Nijholt, S. Petridis, M. Pantic, The MAHNOB mimicry database - a database of naturalistic human interactions. Pattern Recogn. Lett. 66, 52–61 (2015)CrossRefGoogle Scholar
  15. 15.
    M.B. Blaschko, C.H. Lampert, Learning to localize objects with structured output regression, in European Conference on Computer Vision (2008)Google Scholar
  16. 16.
    L. Bourdev, J. Malik, Poselets: body part detectors trained using 3d human pose annotations, in International Conference on Computer Vision (2009)Google Scholar
  17. 17.
    H. Brugman, A. Russel, Annotating multimedia/multi-modal resources with ELAN, in International Conference on Language Resources and Evaluation (2004)Google Scholar
  18. 18.
    X.P. Burgos-Artizzu, P. Perona, P. Dollár, Robust face landmark estimation under occlusion, in International Conference on Computer Vision (2013), pp. 1513–1520Google Scholar
  19. 19.
    X. Cao, Y. Wei, F. Wen, J. Sun, Face alignment by explicit shape regression, in Computer Vision and Pattern Recognition (2012), pp. 2887–2894Google Scholar
  20. 20.
    S. Cheng, S. Zafeiriou, A. Asthana, M. Pantic, 3D facial geometric features for constrained local models, in International Conference on Image Processing (2014)Google Scholar
  21. 21.
    S. Chew, P. Lucey, S. Lucey, J. Saragih, J. Cohn, S. Sridharan, Person-independent facial expression detection using constrained local models, in Automatic Face and Gesture Recognition (2011), pp. 915–920Google Scholar
  22. 22.
    W.S. Chu, F. Zhou, F. De la Torre, Unsupervised temporal commonality discovery, in European Conference on Computer Vision (2012)Google Scholar
  23. 23.
    I. Cohen, N. Sebe, A. Garg, L.S. Chen, T.S. Huang, Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)CrossRefGoogle Scholar
  24. 24.
    J.F. Cohn, P. Ekman, Measuring facial actions, in The New Handbook of Methods in Nonverbal Behavior Research, ed. by J.A. Harrigan, R. Rosenthal, K. Scherer (Oxford University Press, New York, 2005), pp. 9–64Google Scholar
  25. 25.
    J. Cohn, K. Schmidt, The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multiresolution Inf. Process. 2(2), 121–132 (2004)CrossRefGoogle Scholar
  26. 26.
    R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, M. Schröder, FEELTRACE: an instrument for recording perceived emotion in real time, in ISCA Tutorial and Research Workshop on Speech and Emotion (2000)Google Scholar
  27. 27.
    K.D. Craig, C.J. Patrick, Facial expression during induced pain. J. Pers. Soc. Psychol. 48(4), 1080–1091 (1985)CrossRefGoogle Scholar
  28. 28.
    N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in, Computer Vision and Pattern Recognition (2005), pp. 886–893Google Scholar
  29. 29.
    M. Dantone, J. Gall, G. Fanelli, L.J.V. Gool, Real-time facial feature detection using conditional regression forests, in Computer Vision and Pattern Recognition (2012), pp. 2578–2585Google Scholar
  30. 30.
    C. Darwin, The Expression of the Emotions in Man and Animals (John Murray, London, 1872)CrossRefGoogle Scholar
  31. 31.
    K. Dautenhahn, Getting to know each other – artificial social intelligence for autonomous robots. Robot. Auton. Syst. 16(2), 333–356 (1995)CrossRefGoogle Scholar
  32. 32.
    K. Dautenhahn, Socially intelligent robots: dimensions of human–robot interaction. Philos. Trans. R. Soc. B 362(1480), 679–704 (2007)CrossRefGoogle Scholar
  33. 33.
    K. Dautenhahn, I. Werry, Towards interactive robots in autism therapy: background, motivation and challenges. Pragmat. Cogn. 12(1), 1–35 (2004)CrossRefGoogle Scholar
  34. 34.
    F. de Rosis, C. Pelachaud, I. Poggi, V. Carofiglio, B.D. Carolis, From Greta’s mind to her face: modelling the dynamics of affective states in a conversational embodied agent. Int. J. Hum. Comput. Stud. 59(1–2), 81–118 (2003)CrossRefGoogle Scholar
  35. 35.
    B.M. DePaulo, J.J. Lindsay, B.E. Malone, L. Muhlenbruck, K. Charlton, H. Cooper, Cues to deception. Psychol. Bull. 129(1), 74 (2003)Google Scholar
  36. 36.
    A. Dhall, R. Goecke, S. Lucey, T. Gedeon, Collecting large richly annotated facial-expression databases from movies. IEEE MultiMedia 19(3), 34–41 (2012)CrossRefGoogle Scholar
  37. 37.
    X. Ding, W.S. Chu, F.D. la Torre, J.F. Cohn, Q. Wang, Facial action unit event detection by cascade of tasks, in International Conference on Computer Vision (2013)Google Scholar
  38. 38.
    S. Du, Y. Tao, A. Martinez, Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), 1454–1462 (2014)CrossRefGoogle Scholar
  39. 39.
    P. Ekman, W.V. Friesen, Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)Google Scholar
  40. 40.
    P. Ekman, W. Friesen, Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17, 124–129 (1971)CrossRefGoogle Scholar
  41. 41.
    P. Ekman, W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement (Consulting Psychologists, Palo Alto, 1978)Google Scholar
  42. 42.
    P. Ekman, W. Friesen, J.C. Hager, in Facial Action Coding System (A Human Face, Salt Lake City, 2002)Google Scholar
  43. 43.
    F. Eyben, S. Petridis, B. Schuller, G. Tzimiropoulos, S. Zafeiriou, M. Pantic, Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks, in International Conference on Acoustics, Speech and Signal Processing (2011), pp. 5844–5847Google Scholar
  44. 44.
    P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object detection with discriminatively trained part-based models. Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)CrossRefGoogle Scholar
  45. 45.
    T. Fong, I. Nourbakhsh, K. Dautenhahn, A survey of socially interactive robots. Robot. Auton. Syst. 42(3), 143–166 (2003)zbMATHCrossRefGoogle Scholar
  46. 46.
    D. Gatica-Perez, Automatic nonverbal analysis of social interaction in small groups: a review. Image Vis. Comput. 27(12), 1775–1787 (2009)CrossRefGoogle Scholar
  47. 47.
    A. Gudi, H.E. Tasli, T.M. den Uyl, A. Maroulis, Deep learning based FACS action unit occurrence and intensity estimation, in Facial Expression Recognition and Analysis Challenge (2015)Google Scholar
  48. 48.
    H. Gunes, B. Schuller, Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis. Comput. 31(2), 120–136 (2013)CrossRefGoogle Scholar
  49. 49.
    T. Hassner, S. Harel, E. Paz, R. Enbar, Effective face frontalization in unconstrained images, in Computer Vision and Pattern Recognition (2015)Google Scholar
  50. 50.
    H. Hung, D. Gatica-Perez, Estimating cohesion in small groups using audio-visual nonverbal behavior. Trans. Multimedia 12(6), 563–575 (2010)CrossRefGoogle Scholar
  51. 51.
    H. Hung, Y. Huang, G. Friedland, D. Gatica-Perez, Estimating dominance in multi-party meetings using speaker diarization. IEEE Trans. Audio Speech Lang. Process. 19(4), 847–860 (2011)CrossRefGoogle Scholar
  52. 52.
    M.E. Jabon, J.N. Bailenson, E. Pontikakis, L. Takayama, C. Nass, Facial expression analysis for predicting unsafe driving behavior. IEEE Pervasive Comput. 10(4), 84–95 (2011)CrossRefGoogle Scholar
  53. 53.
    V. Jain, E. Learned-Miller, FDDB: a benchmark for face detection in unconstrained settings. Technical Report UM-CS-2010-009, University of Massachusetts, Amherst (2010)Google Scholar
  54. 54.
    S. Jaiwand, B. Martinez, M. Valstar, Learning to combine local models for facial action unit detection, in Facial Expression Recognition and Analysis Challenge, in conj. with Face and Gesture Recognition (2015)Google Scholar
  55. 55.
    Q. Ji, X. Yang, Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8(5), 357–377 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  56. 56.
    H. Jia, A.M. Martinez, Support vector machines in face recognition with occlusions, in Computer Vision and Pattern Recognition (2009), pp. 136–141Google Scholar
  57. 57.
    B. Jiang, M.F. Valstar, M. Pantic, Action unit detection using sparse appearance descriptors in space-time video volumes, in Automatic Face and Gesture Recognition (2011), pp. 314–321Google Scholar
  58. 58.
    B. Jiang, B. Martinez, M. Pantic, Parametric temporal alignment for the detection of facial action temporal segments, in British Machine Vision Conference (2014)Google Scholar
  59. 59.
    B. Jiang, B. Martinez, M.F. Valstar, M. Pantic, Decision level fusion of domain specific regions for facial action recognition, in International Conference on Pattern Recognition (2014)Google Scholar
  60. 60.
    B. Jiang, M.F. Valstar, B. Martinez, M. Pantic, Dynamic appearance descriptor approach to facial actions temporal modelling. Trans. Cybern. 44(2), 161–174 (2014)CrossRefGoogle Scholar
  61. 61.
    B. Jiang, B. Martinez, M. Pantic, Automatic analysis of facial actions, a survey. Trans. Affect. Comput. (under review)Google Scholar
  62. 62.
    S. Kaltwang, O. Rudovic, M. Pantic, Continuous pain intensity estimation from facial expressions, in Advances in Visual Computing (Springer, Heidelberg, 2012), pp. 368–377Google Scholar
  63. 63.
    S. Kaltwang, S. Todorovic, M. Pantic, Latent trees for estimating intensity of facial action units, in Computer Vision and Pattern Recognition (2015)Google Scholar
  64. 64.
    M. Kipp, ANVIL - a generic annotation tool for multimodal dialogue, in European Conference on Speech Communication and Technology (2001), pp. 1367–1370Google Scholar
  65. 65.
    S. Koelstra, I. Patras, Fusion of facial expressions and EEG for implicit affective tagging. Image Vis. Comput. 31(2), 164–174 (2013)CrossRefGoogle Scholar
  66. 66.
    S. Koelstra, M. Pantic, I. Patras, A dynamic texture based approach to recognition of facial actions and their temporal models. Trans. Pattern Anal. Mach. Intell. 32(11), 1940–1954 (2010)CrossRefGoogle Scholar
  67. 67.
    N. Komodakis, Efficient training for pairwise or higher order CRFs via dual decomposition, in Computer Vision and Pattern Recognition (2011), pp. 1841–1848Google Scholar
  68. 68.
    A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012)Google Scholar
  69. 69.
    I. Leite, G. Castellano, A. Pereira, C. Martinho, A. Paiva, Empathic robots for long-term interaction. Int. J. Soc. Robot. 6(3), 329–341 (2014)CrossRefGoogle Scholar
  70. 70.
    G. Littlewort, M.S. Bartlett, I. Fasel, J. Susskind, J. Movellan, Dynamics of facial expression extracted automatically from video, in Image and Vision Computing (2004), pp. 615–625Google Scholar
  71. 71.
    G. Littlewort, J. Whitehill, T. Wu, I.R. Fasel, M.G. Frank, J.R. Movellan, M.S. Bartlett, The computer expression recognition toolbox (CERT), in Automatic Face and Gesture Recognition (2011), pp. 298–305Google Scholar
  72. 72.
    M. Liu, S. Shan, R. Wang, X. Chen, Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition, in Computer Vision and Pattern Recognition (2014), pp. 1749–1756Google Scholar
  73. 73.
    P. Liu, S. Han, Z. Meng, Y. Tong, Facial expression recognition via a boosted deep belief network, in Computer Vision and Pattern Recognition (2014)Google Scholar
  74. 74.
    P. Lucey, J.F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, K.M. Prkachin, Automatically detecting pain in video through facial action units. Trans. Syst. Man Cybern. B 41(3), 664–674 (2011)CrossRefGoogle Scholar
  75. 75.
    P. Lucey, J.F. Cohn, K.M. Prkachin, P.E. Solomon, I. Matthews, Painful data: the UNBC-McMaster shoulder pain expression archive database, in Automatic Face and Gesture Recognition (2011)Google Scholar
  76. 76.
    M. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, Coding facial expressions with Gabor wavelets, in Automatic Face and Gesture Recognition (1998)Google Scholar
  77. 77.
    A. Maalej, B.B. Amor, M. Daoudi, A. Srivastava, S. Berretti, Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recogn. 44(8), 1581–1589 (2011)CrossRefGoogle Scholar
  78. 78.
    B. Martinez, M.F. Valstar, L21-based regression and prediction accumulation across views for robust facial landmark detection. Image Vis. Comput. In pressGoogle Scholar
  79. 79.
    B. Martinez, M.F. Valstar, X. Binefa, M. Pantic, Local evidence aggregation for regression based facial point detection. Trans. Pattern Anal. Mach. Intell. 35(5), 1149–1163 (2013)CrossRefGoogle Scholar
  80. 80.
    M. Mathias, R. Benenson, M. Pedersoli, L. van Gool, Face detection without bells and whistles, in European Conference on Computer Vision (2014)Google Scholar
  81. 81.
    D. Matsumoto, More evidence for the universality of a contempt expression. Motiv. Emot. 16, 363–368 (1992)CrossRefGoogle Scholar
  82. 82.
    I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, D. Zhang, Automatic analysis of multimodal group actions in meetings. Trans. Pattern Anal. Mach. Intell. 27(3), 305–317 (2005)CrossRefGoogle Scholar
  83. 83.
    D. McDuff, R. El Kaliouby, E. Kodra, R. Picard, Measuring voter’s candidate preference based on affective responses to election debates, in Affective Computing and Intelligent Interaction (2013), pp. 369–374Google Scholar
  84. 84.
    D. McDuff, R. Kaliouby, T. Senechal, A, Amr, J.F. Cohn, R. Picard, Affectiva-MIT facial expression dataset (AM-FED): naturalistic and spontaneous facial expressions collected in-the-wild, in Computer Vision and Pattern Recognition Workshop (2013), pp. 881–888Google Scholar
  85. 85.
    D. McDuff, R. El Kaliouby, T. Senechal, D. Demirdjian, R. Picard, Automatic measurement of ad preferences from facial responses gathered over the internet. Image Vis. Comput. 32(10), 630–640 (2014)CrossRefGoogle Scholar
  86. 86.
    D. McDuff, R. Kaliouby, J. Cohn, R. Picard, Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. Trans. Affect. Comput. 6, 223–235 (2015)CrossRefGoogle Scholar
  87. 87.
    G. McKeown, I. Sneddon, Modeling continuous self-report measures of perceived emotion using generalized additive mixed models. Psychol. Methods 19(1), 155–74 (2014)CrossRefGoogle Scholar
  88. 88.
    G. McKeown, M. Valstar, R. Cowie, M. Pantic, M. Schroder, The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3, 5–17 (2012). doi: Google Scholar
  89. 89.
    L. Morency, I. de Kok, J. Gratch, Context-based recognition during human interactions: automatic feature selection and encoding dictionary, in International Conference on Multimodal Interaction (2008), pp. 181–188Google Scholar
  90. 90.
    R. Navarathna, P. Lucey, P. Carr, E. Carter, S. Sridharan, I. Matthews, Predicting movie ratings from audience behaviors, in IEEE Winter Conference on Applications of Computer Vision (2014), pp. 1058–1065Google Scholar
  91. 91.
    L.S. Nguyen, A. Marcos-Ramiro, M.M. Romera, D. Gatica-Perez, Multimodal analysis of body communication cues in employment interviews, in International Conference on Multimodal Interaction (2013), pp. 437–444Google Scholar
  92. 92.
    M.A. Nicolaou, H. Gunes, M. Pantic, Output-associative RVM regression for dimensional and continuous emotion prediction. Image Vis. Comput. 30(3), 186–196 (2012)CrossRefGoogle Scholar
  93. 93.
    M.A. Nicolaou, V. Pavlovic, M. Pantic, Dynamic probabilistic CCA for analysis of affective behaviour and fusion of continuous annotations. Trans. Pattern Anal. Mach. Intell. 36(7), 1299–1311 (2014)CrossRefGoogle Scholar
  94. 94.
    T. Ojala, M. Pietikainen, D. Harwood, A comparative study of texture measures with classification based on featured distribution. Pattern Recogn. 29(1), 51–59 (1996)CrossRefGoogle Scholar
  95. 95.
    J. Orozco, B. Martinez, M. Pantic, Empirical analysis of cascade deformable models for multi-view face detection. Image Vis. Comput. 42, 47–61 (2015)CrossRefGoogle Scholar
  96. 96.
    I. Pavlidis, N.L. Eberhardt, J.A. Levine, Human behaviour: seeing through the face of deception. Nature 415(6867), 35–35 (2002)CrossRefGoogle Scholar
  97. 97.
    P. Perakis, G. Passalis, T. Theoharis, I. Kakadiaris, 3D facial landmark detection under large yaw and expression variations. Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2013)CrossRefGoogle Scholar
  98. 98.
    S. Petridis, M. Pantic, Audiovisual discrimination between laughter and speech, in International Conference on Acoustics, Speech and Signal Processing (2008), pp. 5117–5120Google Scholar
  99. 99.
    S. Petridis, B. Martinez, M. Pantic, The MAHNOB laughter database. Image Vis. Comput. 31(2), 186–202 (2013)CrossRefGoogle Scholar
  100. 100.
    J.H. Pfeifer, M. Iacoboni, J.C. Mazziotta, M. Dapretto, Mirroring others’ emotions relates to empathy and interpersonal competence in children. NeuroImage 39(4), 2076–2085 (2008)CrossRefGoogle Scholar
  101. 101.
    T. Pfister, X. Li, G. Zhao, M. Pietikäinen, Recognising spontaneous facial micro-expressions, in International Conference on Computer Vision (2011), pp. 1449–1456Google Scholar
  102. 102.
    R.W. Picard, Affective Computing (MIT, Cambridge, 1997)CrossRefGoogle Scholar
  103. 103.
    K.M. Prkachin, P.E. Solomon, The structure, reliability and validity of pain expression: evidence from patients with shoulder pain. Pain 139, 267–274 (2008)CrossRefGoogle Scholar
  104. 104.
    O. Rudovic, M. Pantic, Shape-constrained Gaussian process regression for facial-point-based head-pose normalization, in International Conference on Computer Vision (2011), pp. 1495–1502Google Scholar
  105. 105.
    J.A. Russell, A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)CrossRefGoogle Scholar
  106. 106.
    G. Sandbach, S. Zafeiriou, M. Pantic, Binary pattern analysis for 3D facial action unit detection, in The British Machine Vision Conference (2012)Google Scholar
  107. 107.
    G. Sandbach, S. Zafeiriou, M. Pantic, Markov random field structures for facial action unit intensity estimation, in International Conference on Computer Vision Workshop (2013)Google Scholar
  108. 108.
    J.M. Saragih, S. Lucey, J.F. Cohn, Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  109. 109.
    E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: a survey of registration, representation and recognition. Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)CrossRefGoogle Scholar
  110. 110.
    K. Scherer, P. Ekman, Handbook of Methods in Nonverbal Behavior Research (Cambridge University Press, Cambridge, 1982)Google Scholar
  111. 111.
    M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. ter Maat, G. pain, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M.F. Valstar, M. Wöllmer, Building autonomous sensitive artificial listeners. Trans. Affect. Comput. 3(2), 165–183 (2012)Google Scholar
  112. 112.
    T. Senechal, V. Rapp, H. Salam, R. Seguier, K. Bailly, L. Prevost, Facial action recognition combining heterogeneous features via multi-kernel learning. IEEE Trans. Syst. Man Cybern. B 42(4), 993–1005 (2012)CrossRefGoogle Scholar
  113. 113.
    T. Sha, M. Song, J. Bu, C. Chen, D. Tao, Feature level analysis for 3D facial expression recognition. Neurocomputing 74(12–13), 2135–2141 (2011)CrossRefGoogle Scholar
  114. 114.
    C. Shan, S. Gong, P. McOwan, Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  115. 115.
    P.E. Shrout, J.L. Fleiss, Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86(2), 420–428 (1979)CrossRefGoogle Scholar
  116. 116.
    T. Simon, M.H. Nguyen, F.D.L. Torre, J. Cohn, Action unit detection with segment-based SVMs, in Computer Vision and Pattern Recognition (2010), pp. 2737–2744Google Scholar
  117. 117.
    M. Soleymani, M. Pantic, Human-centered implicit tagging: overview and perspectives, in International Conference on Systems, Man, and Cybernetics (2012), pp. 3304–3309Google Scholar
  118. 118.
    M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, A multimodal database for affect recognition and implicit tagging. Trans. Affect. Comput. 3(1), 42–55 (2012)CrossRefGoogle Scholar
  119. 119.
    M. Soleymani, M. Larson, T. Pun, A. Hanjalic, Corpus development for affective video indexing. Trans. Multimedia 16(4), 1075–1089 (2014)CrossRefGoogle Scholar
  120. 120.
    M.J.L. Sullivan, P. Thibault, A. Savard, R. Catchlove, J. Kozey, W.D. Stanish, The influence of communication goals and physical demands on different dimensions of pain behavior. Pain 125(3), 270–277 (2006)CrossRefGoogle Scholar
  121. 121.
    X. Sun, J. Lichtenauer, M. Valstar, A. Nijholt, M. Pantic, A multimodal database for mimicry analysis, in Affective Computing and Intelligent Interaction (2011), pp. 367–376Google Scholar
  122. 122.
    M. Takahashi, M. Naemura, M. Fujii, S. Satoh, Estimation of attentiveness of people watching TV based on their emotional behaviors, in Affective Computing and Intelligent Interaction (2013), pp. 809–814Google Scholar
  123. 123.
    H. Tang, T. Huang, 3D facial expression recognition based on properties of line segments connecting facial feature points, in Automatic Face and Gesture Recognition (2008)Google Scholar
  124. 124.
    E. Taralova, F. De la Torre, M. Hebert, Motion words for video, in European Conference on Computer Vision (2014)Google Scholar
  125. 125.
    D. Tax, M.F. Valstar, M. Pantic, E. Hendrix, The detection of concept frames using clustering multi-instance learning, in International Conference on Pattern Recognition (2010), pp. 2917–2920Google Scholar
  126. 126.
    Y. Tong, J. Chen, Q. Ji, A unified probabilistic framework for spontaneous facial action modeling and understanding. Trans. Pattern Anal. Mach. Intell. 32(2), 258–273 (2010)CrossRefGoogle Scholar
  127. 127.
    F. Tsalakanidou, S. Malassiotis, Real-time 2D+3D facial action and expression recognition. Pattern Recogn. 43(5), 1763–1775 (2010)CrossRefGoogle Scholar
  128. 128.
    P. Tsiamyrtzis, J. Dowdall, D. Shastri, I. Pavlidis, M. Frank, P. Ekman, Imaging facial physiology for the detection of deceit. Int. J. Comput. Vis. 71(2), 197–214 (2007)CrossRefGoogle Scholar
  129. 129.
    G. Tzimiropoulos, Project-out cascaded regression with an application to face alignment, in Computer Vision and Pattern Recognition (2015), pp. 3659–3667Google Scholar
  130. 130.
    G. Tzimiropoulos, M. Pantic, Gauss-Newton deformable part models for face alignment in-the-wild, in Computer Vision and Pattern Recognition (2014), pp. 1851–1858Google Scholar
  131. 131.
    M. Valstar, Automatic behaviour understanding in medicine, in Workshop on Roadmapping the Future of Multimodal Interaction Research, including Business Opportunities and Challenges, RFMIR@ICMI (2014), pp. 57–60Google Scholar
  132. 132.
    M. Valstar, M. Pantic, Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. B 42(1), 28–43 (2012)CrossRefGoogle Scholar
  133. 133.
    M. Valstar, I. Patras, M. Pantic, Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data, in Computer Vision and Pattern Recognition Workshops (2005)Google Scholar
  134. 134.
    M.F. Valstar, B. Martinez, X. Binefa, M. Pantic, Facial point detection using boosted regression and graph models, in Computer Vision and Pattern Recognition (2010), pp. 2729–2736Google Scholar
  135. 135.
    M.F. Valstar, M. Mehu, B. Jiang, M. Pantic, K. Scherer, Meta – analysis of the first facial expression recognition challenge. IEEE Trans. Syst. Man Cybern. B 42(4), 966–979 (2012)CrossRefGoogle Scholar
  136. 136.
    M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, M. Pantic, AVEC 2014: 3D dimensional affect and depression recognition challenge, in International Workshop on Audio/Visual Emotion Challenge (2014), pp. 3–10Google Scholar
  137. 137.
    M.F. Valstar, T. Almaev, J.M. Girard, G. McKeown, M. Mehu, L. Yin, M. Pantic, J.F. Cohn, FERA 2015 - second facial expression recognition and analysis challenge, in Automatic Face and Gesture Recognition Workshop (2015)Google Scholar
  138. 138.
    L. van der Maaten, E. Hendriks, Action unit classification using active appearance models and conditional random fields. Cogn. Process. 13(2), 507–518 (2012)CrossRefGoogle Scholar
  139. 139.
    L. van der Maaten, M. Chen, S. Tyree, K.Q. Weinberger, Learning with marginalized corrupted features, in International Conference on Machine Learning (2013), pp. 410–418Google Scholar
  140. 140.
    A. Vinciarelli, M. Pantic, H. Bourlard, Social signal processing: survey of an emerging domain. Image Vis. Comput. 27(12), 1743–1759 (2009)CrossRefGoogle Scholar
  141. 141.
    A. Vinciarelli, M. Pantic, D. Heylen, C. Pelachaud, I. Poggi, F. D’Errico, M. Schröder, M.: Bridging the gap between social animal and unsocial machine: a survey of social signal processing. Trans. Affect. Comput. 3(1), 69–87 (2012)Google Scholar
  142. 142.
    P. Viola, M.J. Jones, Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  143. 143.
    E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, J. Movellan, Drowsy driver detection through facial movement analysis, in IEEE International Conference on Human-Computer Interaction (2007), pp. 6–18Google Scholar
  144. 144.
    Z. Wang, Y. Li, S. Wang, Q. Ji, Capturing global semantic relationships for facial action unit recognition, in International Conference on Computer Vision (2013), pp. 3304–3311Google Scholar
  145. 145.
    S. Wang, Z. Liu, Y. Zhu, M. He, X. Chen, Q. Ji, Implicit video emotion tagging from audiences’ facial expression. Multimedia Tools Appl. 74(13), 4679–4706 (2015)CrossRefGoogle Scholar
  146. 146.
    G. Warren, E. Schertler, P. Bull, Detecting deception from emotional and unemotional cues. J. Nonverbal Behav. 33(1), 59–69 (2009)CrossRefGoogle Scholar
  147. 147.
    F. Weninger, Introducing CURRENNT: the munich open-source CUDA recurrent neural network toolkit. J. Mach. Learn. Res. 16, 547–551 (2015)MathSciNetzbMATHGoogle Scholar
  148. 148.
    J. Whitehill, Z. Serpell, Y. Lin, A. Foster, J.R. Movellan, The faces of engagement: automatic recognition of student engagement from facial expressions. Trans. Affect. Comput. 5(1), 86–98 (2014)CrossRefGoogle Scholar
  149. 149.
    M. Wöllmer, A. Metallinou, F. Eyben, B. Schuller, S.S. Narayanan, Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling, in Interspeech (2010), pp. 2362–2365Google Scholar
  150. 150.
    Q. Wu, X. Shen, X. Fu, The machine knows what you are hiding: an automatic micro-expression recognition system, in Affective Computing and Intelligent Interaction (2011), pp. 152–162Google Scholar
  151. 151.
    X. Xiong, F. De la Torre, Supervised descent method and its applications to face alignment, in Computer Vision and Pattern Recognition (2013)Google Scholar
  152. 152.
    J. Yan, Z. Lei, D. Yi, S.Z. Li, Learn to combine multiple hypotheses for accurate face alignment, in International Conference on Computer Vision Workshop (2013), pp. 392–396Google Scholar
  153. 153.
    W. Yan, Q. Wu, Y. Liu, S. Wang, X. Fu, CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces, in Automatic Face and Gesture Recognition (2013)Google Scholar
  154. 154.
    J. Yan, X. Zhang, Z. Lei, S.Z. Li, Face detection by structural models. Image Vis. Comput. 32(10), 790–799 (2014)CrossRefGoogle Scholar
  155. 155.
    P. Yang, Q. Liu, D.N. Metaxas, Boosting encoded dynamic features for facial expression recognition. Pattern Recogn. Lett. 30(2), 132–139 (2009)CrossRefGoogle Scholar
  156. 156.
    X. Yu, Z. Lin, J. Brandt, D. Metaxas, Consensus of regression for occlusion-robust facial feature localization, in European Conference on Computer Vision (2014), pp. 105–118Google Scholar
  157. 157.
    Z. Zeng, M. Pantic, G. Roisman, T.S. Huang et al., A survey of affect recognition methods: audio, visual, and spontaneous expressions. Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)CrossRefGoogle Scholar
  158. 158.
    X. Zhang, L. Yin, J.F. Cohn, Three dimensional binary edge feature representation for pain expression analysis, in Automatic Face and Gesture Recognition (2015)Google Scholar
  159. 159.
    G. Zhao, M. Pietikainen, Dynamic texture recognition using local binary patterns with an application to facial expressions. Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)CrossRefGoogle Scholar
  160. 160.
    L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, D.N. Metaxas, Learning active facial patches for expression analysis, in Computer Vision and Pattern Recognition (2012), pp. 2562–2569Google Scholar
  161. 161.
    X. Zhu, D. Ramanan, Face detection, pose estimation, and landmark localization in the wild, in Computer Vision and Pattern Recognition (2012), pp. 2879–2886Google Scholar
  162. 162.
    M. Zimmerman, I. Chelminski, M. Posternak, A review of studies of the Hamilton depression rating scale in healthy controls: implications for the definition of remission in treatment studies of depression. J. Nerv. Ment. Dis. 192(9), 595–601 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of Computer ScienceNottinghamUK

Personalised recommendations