Advertisement

Behavior Research Methods

, Volume 50, Issue 4, pp 1446–1460 | Cite as

Facial expression analysis with AFFDEX and FACET: A validation study

  • Sabrina Stöckli
  • Michael Schulte-Mecklenbeck
  • Stefan Borer
  • Andrea C. Samson
Article

Abstract

The goal of this study was to validate AFFDEX and FACET, two algorithms classifying emotions from facial expressions, in iMotions’s software suite. In Study 1, pictures of standardized emotional facial expressions from three databases, the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP), the Amsterdam Dynamic Facial Expression Set (ADFES), and the Radboud Faces Database (RaFD), were classified with both modules. Accuracy (Matching Scores) was computed to assess and compare the classification quality. Results show a large variance in accuracy across emotions and databases, with a performance advantage for FACET over AFFDEX. In Study 2, 110 participants’ facial expressions were measured while being exposed to emotionally evocative pictures from the International Affective Picture System (IAPS), the Geneva Affective Picture Database (GAPED) and the Radboud Faces Database (RaFD). Accuracy again differed for distinct emotions, and FACET performed better. Overall, iMotions can achieve acceptable accuracy for standardized pictures of prototypical (vs. natural) facial expressions, but performs worse for more natural facial expressions. We discuss potential sources for limited validity and suggest research directions in the broader context of emotion research.

Keywords

Emotion classification Facial expression FACS AFFDEX FACET 

Supplementary material

13428_2017_996_MOESM1_ESM.docx (216 kb)
ESM 1 (DOCX 215 kb)

References

  1. Aragón, O. R., Clark, M. S., Dyer, R. L., & Bargh, J. A. (2015). Dimorphous expressions of positive emotion: Displays of both care and aggression in response to cute stimuli. Psychological Science, 26(3) 1–15.CrossRefGoogle Scholar
  2. Aviezer, H., Trope, Y., & Todorov, A. (2012). Holistic person processing: Faces with bodies tell the whole story. Journal of Personality and Social Psychology, 103(1), 20–37.CrossRefPubMedGoogle Scholar
  3. Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. Proceedings from 2016 I.E. Winter Conference on Applications of Computer Vision (WACV) (pp. 1–10). IEEE.Google Scholar
  4. Barrett, L. F., Mesquita, B., & Gendron, M. (2011). Context in emotion perception. Current Directions in Psychological Science, 20(5), 286–290.CrossRefGoogle Scholar
  5. Barrett L. F., & Wager T. D. (2006). The structure of emotion: evidence from neuroimaging studies. Current Directions in Psychological Science, 15, 79–83.  https://doi.org/10.1111/j.0963-7214.2006.00411 CrossRefGoogle Scholar
  6. Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36(02), 253–263.CrossRefPubMedGoogle Scholar
  7. Beumer, G. M., Tao, Q., Bazen, A. M., & Veldhuis, R. N. (2006). A landmark paper in face recognition. In 7th International Conference on Automatic Face and Gesture Recognition, (pp. 73–78). IEEE.Google Scholar
  8. Bonanno, G., & Keltner, D. (2004). Brief Report: The coherence of emotion systems: Comparing “on-line” measures of appraisal and facial expressions, and self-report. Cognition and Emotion, 18(3), 431–444.CrossRefGoogle Scholar
  9. Calvo, M. G., Gutiérrez-García, A., Fernández-Martín, A., & Nummenmaa, L. (2014). Recognition of facial expressions of emotion is related to their frequency in everyday life. Journal of Nonverbal Behavior, 38(4), 549–567.CrossRefGoogle Scholar
  10. Carretié, L., Mercado, F., Tapia, M., & Hinojosa, J. A. (2001). Emotion, attention, and the ‘negativity bias’, studied through event-related potentials. International Journal of Psychophysiology, 41(1), 75–85.CrossRefPubMedGoogle Scholar
  11. Coan, J. A., & Allen, J. J. (Eds.). (2007). Handbook of emotion elicitation and assessment. Oxford, UK: Oxford University Press.Google Scholar
  12. Cohn, J. F., & Sayette, M. A. (2010). Spontaneous facial expression in a small group can be automatically measured: An initial demonstration. Behavior Research Methods, 42(4), 1079–1086.CrossRefPubMedPubMedCentralGoogle Scholar
  13. Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001). Active appearance models. IEEE Transactions on pattern analysis and machine intelligence, 23(6), 681–685.CrossRefGoogle Scholar
  14. Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 43(2), 468–477.CrossRefPubMedGoogle Scholar
  15. den Uyl, M. J., & van Kuilenburg, H. (2005). The FaceReader: Online facial expression recognition. In Proceedings of Measuring Behavior 2005 (pp. 589–590).Google Scholar
  16. Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), 1454–1462.CrossRefGoogle Scholar
  17. Ekman, P. (1992a). An argument for basic emotions. Cognition & Emotion, 6(3–4), 169–200.CrossRefGoogle Scholar
  18. Ekman, P. (1992b). Are there basic emotions? Psychological Review, 99(3), 550-553.CrossRefPubMedGoogle Scholar
  19. Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1(1), 56–75.CrossRefGoogle Scholar
  20. Ekman, P., & Friesen, W. V. (1982). Felt, false, and miserable smiles. Journal of Nonverbal Behavior, 6(4), 238–252.CrossRefGoogle Scholar
  21. Ekman, P., Friesen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125–1134.CrossRefGoogle Scholar
  22. Ekman, P., & Friesen, W. V. (2003). Unmasking the face: A guide to recognizing emotions from facial clues. Cambridge, MA: Malor Books.Google Scholar
  23. Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K. … Tzavaras, A. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712–717.CrossRefPubMedGoogle Scholar
  24. Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30(1), 527–554.CrossRefGoogle Scholar
  25. Elfenbein, H. A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological Bulletin, 128(2), 203-235.CrossRefPubMedGoogle Scholar
  26. El Kaliouby, R., & Robinson, P. (2005). Real-time inference of complex mental states from facial expressions and head gestures. In B. Kisačanin, V. Pavlović & T. S. Huang (Eds), Real-time vision for human-computer interaction (pp. 181–200). New York: Springer.Google Scholar
  27. Ellsworth, P. C., & Scherer, K. R. (2003). Appraisal processes in emotion. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 572–595). New York: Oxford University Press.Google Scholar
  28. Fredrickson, B. L., & Levenson, R. W. (1998). Positive emotions speed recovery from the cardiovascular sequelae of negative emotions. Cognition & Emotion, 12(2), 191–220.CrossRefGoogle Scholar
  29. Gotlib, I. H., Krasnoperova, E., Yue, D. N., & Joormann, J. (2004). Attentional biases for negative interpersonal stimuli in clinical depression. Journal of Abnormal Psychology, 113(1), 127–135.CrossRefGoogle Scholar
  30. Greenwald, M. K., Cook, E. W., & Lang, P. J. (1989). Affective judgment and psychophysiological response: Dimensional covariation in the evaluation of pictorial stimuli. Journal of Psychophysiology, 3(1), 51–64.Google Scholar
  31. Gross, J. J. (2002). Emotion regulation: Affective, cognitive, and social consequences. Psychophysiology, 39(3), 281–291.CrossRefPubMedGoogle Scholar
  32. Hager J. C., P. Ekman, & W. V. Friesen (2002). Facial action coding system. Salt Lake City, UT: A Human Face.Google Scholar
  33. Hess, U., Banse, R., & Kappas, A. (1995). The intensity of facial expression is determined by underlying affective state and social situation. Journal of Personality and Social Psychology, 69(2), 280–288.CrossRefGoogle Scholar
  34. Huang, C.-N., Chen, C.-H., & Chung, H.-Y. (2004). The review of applications and measurements in facial electromyography. Journal of Medical and Biological Engineering, 25(1), 15–20.Google Scholar
  35. Huang, Y.-X., & Luo, Y.-J. (2006). Temporal course of emotional negativity bias: An ERP study. Neuroscience Letters, 398(1), 91–96.  https://doi.org/10.1016/j.neulet.2005.12.074 CrossRefPubMedGoogle Scholar
  36. Hwang, H. C., & Matsumoto, D. (2016). Facial expressions. In D. Matsumoto, H. C. Hwang, & M. G. Frank (Eds.), APA handbook of nonverbal communication (pp. 257–287). Washington, DC, US: American Psychological Association.  https://doi.org/10.1037/14669-010 CrossRefGoogle Scholar
  37. iMotions (2016). Facial Expression Analysis: The definitive guide. Retrieved from https://imotions.com/facialexpression-guide-ebook/
  38. Kim, H., Somerville, L. H., Johnstone, T., Polis, S., Alexander, A. L., Shin, L. M., & Whalen, P. J. (2004). Contextual modulation of amygdala responsivity to surprised faces. Journal of Cognitive Neuroscience, 16(10), 1730–1745.CrossRefPubMedGoogle Scholar
  39. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377–1388.Google Scholar
  40. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1999). International affective picture system (IAPS): Technical manual and affective ratings. Gainesville, FL: The Center for Research in Psychophysiology, University of Florida. Retrieved from http://www2.unifesp.br/dpsicobio/Nova_versao_pagina_psicobio/adap/instructions.pdf Google Scholar
  41. Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273.  https://doi.org/10.1111/j.1469-8986.1993.tb03352.x CrossRefPubMedGoogle Scholar
  42. Larsen, J. T., Norris, C. J., & Cacioppo, J. T. (2003). Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii. Psychophysiology, 40(5), 776–785.CrossRefPubMedGoogle Scholar
  43. Lee, H., Choi, Y. S., Lee, S., & Park, I. P. (2012). Towards unobtrusive emotion recognition for affective social communication. In Consumer Communications and Networking Conference (CCNC), 2012 IEEE (pp. 260-264). IEEE.  https://doi.org/10.1109/CCNC.2012.6181098
  44. Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7(4), 227–236.Google Scholar
  45. Lien, J. J., Kanade, T., Cohn, J. F., & Li, C.-C. (1998). Automated facial expression recognition based on FACS action units. Proceedings from Third IEEE International Conference on Automatic Face and Gesture Recognition (pp. 390–395). IEEE. Retrieved from http://ieeexplore.ieee.org/abstract/document/670980/
  46. Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J., & Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. Image and Vision Computing, 24(6), 615–625.  https://doi.org/10.1016/j.imavis.2005.09.011 CrossRefGoogle Scholar
  47. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M. (2011). The computer expression recognition toolbox (CERT). Proceedings from 2011 I.E. International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), (pp. 298–305). IEEE.  https://doi.org/10.1109/FG.2011.5771414
  48. Manera, V., Samson, A. C., Pehrs, C., Lee, I. A., & Gross, J. J. (2014). The eyes have it: The role of attention in cognitive reappraisal of social stimuli. Emotion, 14(5), 833–900.CrossRefPubMedGoogle Scholar
  49. McDuff, D., El Kaliouby, R., Kassam, K., & Picard, R. (2010). Affect valence inference from facial action unit spectrograms. Proceedings from 2010 I.E. Computer Society Conference on Computer Vision and Pattern Recognition — Workshops (pp. 17–24). IEEE.  https://doi.org/10.1109/CVPRW.2010.5543833
  50. McDuff, D., El Kaliouby, R., Cohn, J. F., & Picard, R. W. (2015). Predicting ad liking and purchase intent: Large-scale analysis of facial responses to ads. Affective Computing, IEEE Transactions, 6(3), 223–235.Google Scholar
  51. McDuff, D. (2016). Discovering Facial Expressions for States of Amused, Persuaded, Informed, Sentimental and Inspired. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (pp. 71–75). New York, NY, USA: ACM.  https://doi.org/10.1145/2993148.2993192 Google Scholar
  52. McEwan, K., Gilbert, P., Dandeneau, S., Lipka, S., Maratos, F., Paterson, K. B., & Baldwin, M. (2014). Facial expressions depicting compassionate and critical emotions: The development and validation of a new emotional face stimulus set. PloS one, 9(2), 1–8.CrossRefGoogle Scholar
  53. Meiselman, H. L. (2016). Emotion measurement. Cambridge, UK: Woodhead.CrossRefGoogle Scholar
  54. Mortillaro, M., Meuleman, B., & Scherer, K. R. (2015). Automated Recognition of Emotion Appraisals. In J. Vallverdu (Ed.), Handbook of Research on Synthesizing Human Emotion in Intelligent Systems and Robotics (pp. 338-351). Hershey, PA: IGI Global.CrossRefGoogle Scholar
  55. Naab, P. J., & Russell, J. A. (2007). Judgments of emotion from spontaneous facial expressions of New Guineans. Emotion, 7, 736–744.CrossRefPubMedGoogle Scholar
  56. Nelson, N. L., & Russell, J. A. (2013). Universality revisited. Emotion Review, 5(1), 8–15.CrossRefGoogle Scholar
  57. Neta, M., Davis, F. C., & Whalen, P. J. (2011). Valence resolution of ambiguous facial expressions using an emotional oddball task. Emotion, 11(6), 1425–1433.CrossRefPubMedPubMedCentralGoogle Scholar
  58. Öhman, A., Lundqvist, D., & Esteves, F. (2001). The face in the crowd revisited: a threat advantage with schematic stimuli. Journal of Personality and Social Psychology, 80(3), 381–396.CrossRefPubMedGoogle Scholar
  59. Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., & Ohme, R. K. (2015). Warsaw set of emotional facial expression pictures: A validation study of facial display photographs. Frontiers in Psychology, 5, 1–8.CrossRefGoogle Scholar
  60. Ortony, A., & Turner, T. J. (1990). What's basic about basic emotions?. Psychological Review, 97(3), 315–331.CrossRefPubMedGoogle Scholar
  61. O'Toole, A. Phillips, P. J. Narvekar, A. Jiang, F. Ayyad, J. (2008). Face recognition algorithms and the “other-race” effect. Journal of Vision, 8(6). https://journalofvision.org/8/6/256/
  62. Reiman, E. M., Lane, R. D., Ahern, G. L., Schwartz, G. E., Davidson, R. J., Friston, K. J., … Chen, K. (1997). Neuroanatomical correlates of externally and internally generated human emotion. American Journal of Psychiatry, 154(7), 918–925.CrossRefPubMedGoogle Scholar
  63. Reisenzein, R., Studtmann, M., & Horstmann, G. (2013). Coherence between emotion and facial expression: Evidence from laboratory experiments. Emotion Review, 5(1), 16–23.  https://doi.org/10.1177/1754073912457228 CrossRefGoogle Scholar
  64. Roseman, I. J., & Smith, C. A. (2001). Appraisal theory. Appraisal theory: Overview, assumptions, varieties, controversies. In K. R. Scherer, A. Schorr & T. Johnstone (Eds.), Appraisal processes in emotion (pp. 3–19). Oxford, UK: Oxford University Press.Google Scholar
  65. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172.CrossRefPubMedGoogle Scholar
  66. Scherer, K. R., & Ellgring, H. (2007). Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion, 7, 113–130.CrossRefPubMedGoogle Scholar
  67. Scherer, K. R., Wranik, T., Sangsue, J., Tran, V., & Scherer, U. (2004). Emotions in everyday life: Probability of occurrence, risk factors, appraisal and reaction patterns. Social Science Information, 43, 499–570.CrossRefGoogle Scholar
  68. Scherer, K. R. (2005). What are emotions? And how can they be measured?. Social Science Information, 44(4), 695–729.CrossRefGoogle Scholar
  69. Schlochtermeier, L. H., Pehrs, C., Kuchinke, L., Kappelhoff, H., & Jacobs, A. M. (2015). Emotion processing in different media types: realism, complexity and immersion. Journal of Systems and Integrative Neuroscience, 1, 41–47.CrossRefGoogle Scholar
  70. Schulte-Mecklenbeck, M., Johnson, J.G., Böckenholt, U., Goldstein, D., Russo, J., Sullivan, N., & Willemsen, M. (2017). Process tracing methods in decision making: On growing up in the 70ties. Current Directions in Psychological Science, 26(5), 442–450.  https://doi.org/10.1177/0963721417708229
  71. Stets, J. E., & Turner, J. H. (2014). Handbook of the Sociology of Emotions (Vol. 2). Heidelberg, Germany: Springer.Google Scholar
  72. Swinton, R., & El Kaliouby, R. (2012). Measuring emotions through a mobile device across borders, ages, genders and more. ESOMAR. Retrieved from http://www.affectiva.com/wp-content/uploads/2014/09/Measuring_Emotion_Through_Mobile_Esomar.pdf
  73. Taggart, R. W., Dressler, M., Kumar, P., Khan, S., & Coppola, J. F. (n.d.). Determining emotions via facial expression analysis software. Retrieved from http://csis.pace.edu/~ctappert/srd2016/2016PDF/c2.pdf
  74. Terzis, V., Moridis, C. N., & Economides, A. A. (2010). Measuring instant emotions during a self-assessment test: the use of FaceReader. In Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research (p. 18). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=1931362
  75. Vallverdu, J. (2014). Handbook of research on synthesizing human emotion in intelligent systems and robotics. Hershey PA, USA: IGI Global.Google Scholar
  76. Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011). The first facial expression recognition and analysis challenge. Proceedings from 2011 I.E. International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), (pp. 921–926). IEEE. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5771374
  77. van der Schalk, J., Hawk, S. T., Fischer, A. H., & Doosje, B. (2011). Moving faces, looking places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES). Emotion, 11(4), 907-920.CrossRefPubMedGoogle Scholar
  78. van Kuilenburg, H., Wiering, M., & den Uyl, M. (2005). A model based method for automatic facial expression recognition. In European Conference on Machine Learning (pp. 194–205). Springer.  https://doi.org/10.1007/11564096_22
  79. Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 I.E. Computer Society Conference on, 1(1), 1–9.Google Scholar
  80. Wegrzyn M, Vogt M, Kireclioglu B, Schneider J, & J. Kissler (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE, 12(5).  https://doi.org/10.1371/journal.pone.0177239
  81. Wolf, K. (2015). Measuring facial expression of emotion. Dialogues in Clinical Neuroscience, 17(4), 457–462.PubMedPubMedCentralGoogle Scholar
  82. Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.CrossRefPubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • Sabrina Stöckli
    • 1
  • Michael Schulte-Mecklenbeck
    • 1
    • 2
  • Stefan Borer
    • 1
  • Andrea C. Samson
    • 3
    • 4
  1. 1.Institute of Marketing and Management, Department of Consumer BehaviorUniversity of BernBernSwitzerland
  2. 2.Max Planck Institute for Human DevelopmentBerlinGermany
  3. 3.Swiss Center for Affective SciencesUniversity of GenevaGenevaSwitzerland
  4. 4.Department of Psychiatry and Behavioral ScienceStanford University School of MedicineStanfordUSA

Personalised recommendations