Skip to main content

Advertisement

SpringerLink
Log in

A Critique of Automated Approaches to Code Facial Expressions: What Do Researchers Need to Know?

  • SHORT REVIEW
  • Published:
Affective Science Aims and scope Submit manuscript

Abstract

Facial expression recognition software is becoming more commonly used by affective scientists to measure facial expressions. Although the use of this software has exciting implications, there are persistent and concerning issues regarding the validity and reliability of these programs. In this paper, we highlight three of these issues: biases of the programs against certain skin colors and genders; the common inability of these programs to capture facial expressions made in non-idealized conditions (e.g., “in the wild”); and programs being forced to adopt the underlying assumptions of the specific theory of emotion on which each software is based. We then discuss three directions for the future of affective science in the area of automated facial coding. First, researchers need to be cognizant of exactly how and on which data sets the machine learning algorithms underlying these programs are being trained. In addition, there are several ethical considerations, such as privacy and data storage, surrounding the use of facial expression recognition programs. Finally, researchers should consider collecting additional emotion data, such as body language, and combine these data with facial expression data in order to achieve a more comprehensive picture of complex human emotions. Facial expression recognition programs are an excellent method of collecting facial expression data, but affective scientists should ensure that they recognize the limitations and ethical implications of these programs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. There is currently a lack of agreement within the field of affective science regarding the definition of emotion (for example, see Mulligan & Scherer [2012]). For the purposes of this paper, we use the current definition from the Merriam-Webster dictionary: emotion is “a conscious mental reaction (such as anger or fear) subjectively experienced as strong feeling usually directed toward a specific object and typically accompanied by physiological and behavioral changes in the body” (Merriam-Webster, 2023).

References

  • Benitez-Quiroz, C. F., Wang, Y., & Martinez, A. M. (2017). Recognition of action units in the wild with deep nets and a new global-local loss. Proceedings of the international conference on computer vision (pp. 3970–3979).

  • Birhane, A. (2021). The impossibility of automating ambiguity. Artificial Life, 27, 44–61.

    Article  PubMed  Google Scholar 

  • Birhane, A. (2022). The unseen Black faces of AI algorithms. Nature, 610, 451–452.

    Article  PubMed  Google Scholar 

  • Birhane, A., Isaac, W., Prabhakaran, V., Díaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022). Power to the people? Opportunities and challenges for participatory AI. Equity and access in algorithms, mechanisms, and optimization (pp. 1–8).

  • Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. 2018 conference on fairness, accountability, and transparency, 81 (pp. 77–91).

  • Cabitza, F., Campagner, A., & Mattioli, M. (2022). The unbearable (technical) unreliability of automated facial emotion recognition. Big Data & Society, 9(2), 20539517221129548.

    Article  Google Scholar 

  • Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1, 18–37.

    Article  Google Scholar 

  • Clore, G. L., & Ortony, A. (2013). Psychological construction in the OCC model of emotion. Emotion Review, 5, 335–343.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cross, M. P., Hunter, J. F., Smith, J. R., Twidwell, R. E., & Pressman, S. D. (2022). Comparing, differentiating, and applying affective facial coding techniques for the assessment of positive emotion. The Journal of Positive Psychology.

  • Degutyte, Z., & Astell, A. (2021). The role of eye gaze in regulating turn taking in conversations: A systematized review of methods and findings. Frontiers in Psychology, 12, 616471.

    Article  PubMed  PubMed Central  Google Scholar 

  • D’Mello, S., Kappas, A., & Gratch, J. (2017). The affective computing approach to affect measurement. Emotion Review, 10, 174–183.

    Article  Google Scholar 

  • Dhall, A., Goecke, R., Gedeon, T., & Sebe, N. (2016). Emotion recognition in the wild. Journal on Multimodal User Interfaces, 10(2), 95–97.

    Article  Google Scholar 

  • Do, N. T., Kim, S. H., Yang, H. J., Lee, G. S., & Yeom, S. (2021). Context-aware emotion recognition in the wild using spatio-temporal and temporal-pyramid models. Sensors, 21(7), 2344.

    Article  PubMed  PubMed Central  Google Scholar 

  • Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111, E1454–E1462.

    Article  Google Scholar 

  • Dupré, D., Krumhuber, E. G., Küster, D., & McKeown, G. J. (2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PloS one, 15, e0231968.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ekman, P. (1992). Are there basic emotions? Psychological Review, 99, 550–553.

    Article  PubMed  Google Scholar 

  • Ekman, P., & Cordaro, D. (2011). What is meant by calling emotions basic. Emotion Review, 3, 364–370.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1, 56–75.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1978). Facial action coding system. Consulting Psychologist Press.

    Google Scholar 

  • Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164, 86–88.

    Article  PubMed  Google Scholar 

  • Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: Manual. A human face. Research Nexus.

  • Feldman Barrett, L., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 1, 1–68.

    Article  Google Scholar 

  • Hostetter, A. B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137, 297–315.

    Article  PubMed  Google Scholar 

  • Klare, B., Burge, M., Klontz, J., Vorder, B. R., & Jain, A. (2012). Face recognition performance: Role of demographic information. IEEE Transactions on Information, Forensic, & Security, 7(6), 1789–1801.

    Article  Google Scholar 

  • Küntzler, T., Höfling, T. T. A., & Alpers, G. W. (2021). Automatic facial expression recognition in standardized and non-standardized emotional expressions. Frontiers in Psychology, 12, 627561.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lindquist, K. A., Jackson, J. C., Leshin, J., Satpute, A. B., & Gendron, M. (2022). The cultural evolution of emotion. Nature Reviews Psychology, 1, 669–681.

    Article  Google Scholar 

  • Market Research Report (2022). Emotion detection and recognition market by component (software (facial expression recognition, speech & recognition), services), application area, end user, vertical and region - Global forecast to 2027. Retrieved on December 8, 2022, from https://www.marketsandmarkets.com/Market-Reports/emotion-detection-recognition-market-23376176.html

  • Martinez, A. M. (2019). The promises and perils of automated facial coding in studying children’s emotions. Developmental Psychology, 55, 1965–1981.

    Article  PubMed  PubMed Central  Google Scholar 

  • Merriam-Webster (2023). Emotion. Retrieved on May 23, 2023, from https://www.merriam-webster.com/dictionary/emotion

  • Mulligan, K., & Scherer, K. R. (2012). Toward a working definition of emotion. Emotion Review, 4, 345–357.

    Article  Google Scholar 

  • Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., & Ohme, R. K. (2013). Warsaw set of emotional facial expression pictures: A validation study of facial display photographs. Frontiers in Psychology, 5, 1516.

    Google Scholar 

  • Rhue, L. (2018). Racial influence on automated perceptions of emotions. SSRN.

  • Saurav, S., Saini, R., & Singh, S. (2021). EmNet: A deep integrated convolutional neural network for facial emotion recognition in the wild. Applied Intelligence, 51(8), 5543–5570.

    Article  Google Scholar 

  • Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40, 227–256.

    Article  Google Scholar 

  • Skiendziel, T., Rösch, A. G., & Schultheiss, O. C. (2019). Assessing the convergent validity between automated emotion recognition software Noldus FaceReader 7 and Facial Action Coding System scoring. PLoS One, 14, e0223905.

    Article  PubMed  PubMed Central  Google Scholar 

  • Smith, M., & Miller, S. (2021). The ethical application of biometric facial recognition technology. AI & Society, 37, 167–175.

    Article  Google Scholar 

  • Srinivasan, R., & Martinez, A. M. (2018). Cross-cultural and cultural-specific production and perception of facial expressions of emotion in the wild. IEEE Transactions on Affective Computing, 12, 707–721.

    Article  Google Scholar 

  • Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., & Samson, A. C. (2017). Facial expression analysis with AFFDEX and FACET: a validation study. Behavior Research Methods, 50, 1446–1460.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biobehavioral Health, Pennsylvania State University, University Park, PA, USA

    Marie P. Cross

  2. Basic Biobehavioral and Psychological Sciences Branch, National Cancer Institute, Rockville, MD, USA

    Amanda M. Acevedo

  3. Department of Psychology, Chapman University, Orange, CA, USA

    John F. Hunter

Authors
  1. Marie P. Cross
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amanda M. Acevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John F. Hunter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marie P. Cross.

Ethics declarations

Funding

Not applicable

Conflict of interest

The authors declare no competing interests.

Data availability

Not applicable

Code availability

Not applicable

Authors' contribution

Not applicable

Ethics approval

Not applicable

Consent to participate

Not applicable

Consent for publication

Not applicable

Additional information

Handling Editor: Linda Camras

Dr. Acevedo contributed to this article in her personal capacity. The views expressed are the author’s own and do not reflect the view of the National Institutes of Health, the Department of Health and Human Services, or the US Government.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cross, M.P., Acevedo, A.M. & Hunter, J.F. A Critique of Automated Approaches to Code Facial Expressions: What Do Researchers Need to Know?. Affec Sci 4, 500–505 (2023). https://doi.org/10.1007/s42761-023-00195-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42761-023-00195-0

Keywords

Search

Navigation