Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder
Facial expression is impaired in autism spectrum disorder (ASD), but rarely systematically studied. We focus on the ability of individuals with ASD to produce facial expressions of emotions in response to a verbal prompt. We used the Janssen Autism Knowledge Engine (JAKE®), including automated facial expression analysis software (FACET) to measure facial expressions in individuals with ASD (n = 144) and a typically developing (TD) comparison group (n = 41). Differences in ability to produce facial expressions were observed between ASD and TD groups, demonstrated by activation of facial action units (happy, scared, surprised, disgusted, but not angry or sad). Activation of facial action units correlated with parent-reported social communication skills. This approach has potential for diagnostic and response to intervention measures.
Trial Registration NCT02299700.
KeywordsASD Emotion Facial Expression Automated FACET
The authors thank the study participants and the following investigators for their participation in this study: Arizona Christopher J. Smith, PhD; California Bennett Leventhal, MD and Robert Hendren; Connecticut (at the time of study conduct): Frederick Shic, PhD Massachusetts Jean Frazier, MD New Jersey Yvette Janvier, MD; New York Russell Tobe, MD; North Carolina Geraldine Dawson, PhD; Pennsylvania Judith S. Miller, PhD; Washington Bryan King, MD.
JM, AB, SN, NVM, FS, MG, GD, BL, RH and GP were involved in study design. FS, GD, RH, and BL were involved in data collection. DL, NVM, and JM performed the analysis. All authors were involved in interpretation of the results, drafting the manuscript, and read and approved the final manuscript. All authors had full access to all the data in the study and take responsibility for integrity of the data and the accuracy of the data analysis. All authors meet ICMJE criteria and all those who fulfilled those criteria are listed as authors.
This study was funded by Janssen Research & Development, LLC, USA. We acknowledge Sandra Norris, PharmD of the Norris Communications Group LLC for medical writing support, and Ellen Baum, PhD (Janssen Research & Development, LLC) for additional editorial support for the development of this manuscript.
Compliance with Ethical Standards
Conflict of interest
Joseph Manfredonia, Abigail Bangerter, Seth Ness, David Lewin, Andrew Skalkin, Nikolay V. Manyakov and Gahan Pandina are. Robert Hendren received reimbursement for consultation from Janssen Research & Development, LLC. Matthew Goodwin has received research and consulting funding from Janssen Research & Development, LLC. Geraldine Dawson is on the Scientific Advisory Boards of Janssen Research and Development, LLC and Akili, Inc., a consultant to Roche, has received grant funding from Janssen Research and Development, L.L.C. and PerkinElmer, and receives royalties from Guildford Press and Oxford University Press. Bennett Leventhal has received research grant funding from the NIH, is a consultant to Janssen Research and Development, LLC and the Illinois Children’s Healthcare Foundation, and is a board member of the Brain Research Foundation. Frederick Shic is on the Scientific Advisory Board of and is a consultant to Janssen Research and Development, LLC, and has received grant funding from Janssen Research and Development LLC, and Roche.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
- Bangerter, A., Ness, S., Aman, M. G., Esbensen, A. J., Goodwin, M. S., Dawson, G., et al. (2017). Autism Behavior Inventory: A novel tool for assessing core and associated symptoms of autism spectrum disorder. Journal of Child and Adolescent Psychopharmacology, 27(9), 814–822.Google Scholar
- Bartlett, M., Littlewort, G., Wu, T., & Movellan, J. (2008). Computer expression recognition toolbox. Presented at the 8th IEEE international Conference on Automatic Face and Gesture Recognition (FG 2008), September 17–19, 2008, Amsterdam, The Netherlands. pp. 1–2.Google Scholar
- Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36, 253–263.Google Scholar
- Berenbaum, H., & Rotter, A. (1992). The relationship between spontaneous facial expressions of emotion and voluntary control of facial muscles. Journal of Nonverbal Behavior, 16, 179–190.Google Scholar
- Constantino, J. N., & Gruber, C. P. (2012). Social responsiveness scale (SRS). Torrance: Western Psychological Services.Google Scholar
- Darwin, C. (1872). The expression of emotion in animals and man. London: Methuen.(1877), A biographical sketch of an infant. Mind, 2, 285–294.Google Scholar
- Deckers, A., Muris, P., & Roelofs, J. (2017). Being on your own or feeling lonely? Loneliness and other social variables in youths with autism spectrum disorders. Child Psychiatry & Human Development, 48, 828–839.Google Scholar
- Denham, S. A. (1986). Social cognition, prosocial behavior, and emotion in preschoolers: contextual validation. Child Development, 57(1), 194–201.Google Scholar
- Diano, M., Celeghin, A., Bagnis, A., & Tamietto, M. (2017). Amygdala response to emotional stimuli without awareness: facts and interpretations. Frontiers in Psychology, 7, 2029.Google Scholar
- Eack, S. M., Mazefsky, C. A., & Minshew, N. J. (2015). Misinterpretation of facial expressions of emotion in verbal adults with autism spectrum disorder. Autism, 19(3), 308–315.Google Scholar
- Ekman, P. (1992). An argument for basic emotions. Cognition & emotion, 6(3–4), 169–200.Google Scholar
- Ekman, P., & Friesen, W. V. (1978). Manual for the facial action coding system. Palo Alto: Consulting Psychologists Press.Google Scholar
- Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System. Manual and Investigator’s Guide. Salt Lake City, UT: Research Nexus.Google Scholar
- Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30(1), 527–554. https://doi.org/10.1146/annurev.ps.30.020179.002523.Google Scholar
- Elliott, R., Zahn, R., Deakin, J. W., & Anderson, I. M. (2011). Affective cognition and its disruption in mood disorders. Neuropsychopharmacology, 36(1), 153–182.Google Scholar
- Farroni, T., Menon, E., Rigato, S., & Johnson, M. H. (2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4(1), 2–13.Google Scholar
- Freedman, N. (1972). The Analysis of Movement Behavior During the Clinical Interview. In A. W. Siegman & B. Pope (Eds.), Studies in dyadic communication (Vol. 7, pp. 153–175). Amsterdam: Elsevier B.V.Google Scholar
- Friesen, W., & Ekman, P. (1983). EMFACS-7: Emotional Facial Action Coding System. California: Unpublished manual, University of California.Google Scholar
- Gordon, I., Pierce, M. D., Bartlett, M. S., & Tanaka, J. W. (2014). Training facial expression production in children on the autism spectrum. Journal of Autism and Developmental Disorders, 44(10), 2486–2498.Google Scholar
- Grossman, R. B., Edelson, L. R., & Tager-Flusberg, H. (2013). Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. Journal of Speech, Language, and Hearing Research, 56(3), 1035–1044.Google Scholar
- Guarnera, M., Hichy, Z., Cascio, M. I., & Carrubba, S. (2015). Facial expressions and ability to recognize emotions from eyes or mouth in children. Europe’s Journal of Psychology, 11(2), 183.Google Scholar
- Gunnery, S. D., Hall, J. A., & Ruben, M. A. (2013). The deliberate Duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior, 37(1), 29–41.Google Scholar
- iMotions, A./S. (2016). Facial Expression Analysis. Retrieved 10 November, 2017, from https://imotions.com/guides.
- Jack, R. E., Garrod, O. G., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244.Google Scholar
- Kasari, C., Sigman, M., Mundy, P., & Yirmiya, N. (1990). Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. Journal of Autism and Developmental Disorders, 20(1), 87–100.Google Scholar
- Kim, J. A., Szatmari, P., Bryson, S. E., Streiner, D. L., & Wilson, F. J. (2000). The prevalence of anxiety and mood problems among children with autism and Asperger syndrome. Autism, 4(2), 117–132.Google Scholar
- Langdell, T. (1981). Face perception: An approach to the study of autism PhD Thesis, University of London.Google Scholar
- Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., et al. (2011). The computer expression recognition toolbox (CERT). Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), (pp. 298–305). IEEE.Google Scholar
- Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. (2012). Autism diagnostic observation schedule: ADOS-2. Los Angeles: Western Psychological Services.Google Scholar
- MacDonald, H., Rutter, M., Howlin, P., Rios, P., Conteur, A. L., Evered, C., et al. (1989). Recognition and expression of emotional cues by autistic and normal adults. Journal of Child Psychology and Psychiatry, 30, 865–877. https://doi.org/10.1111/j.1469-7610.1989.tb00288.x.Google Scholar
- McIntosh, D. N., Reichmann-Decker, A., Winkielman, P., & Wilbarger, J. L. (2006). When the social mirror breaks: Deficits in automatic, but not voluntary, mimicry of emotional facial expressions in autism. Developmental Science, 9(3), 295–302. https://doi.org/10.1111/j.1467-7687.2006.00492.x.Google Scholar
- Orsmond, G. I., Krauss, M. W., & Seltzer, M. M. (2004). Peer relationships and social and recreational activities among adolescents and adults with autism. Journal of Autism and Developmental Disorders, 34(3), 245–256.Google Scholar
- Press, C., Richardson, D., & Bird, G. (2010). Intact imitation of emotional facial actions in autism spectrum conditions. Neuropsychologia, 48(11), 3291–3297.Google Scholar
- Rodgers, J. D., Thomeer, M. L., Lopata, C., Volker, M. A., Lee, G. K., McDonald, C. A., et al. (2016). RCT of a psychosocial treatment for children with high-functioning ASD: supplemental analyses of treatment effects on facial emotion encoding. Journal of Developmental and Physical Disabilities, 27(2), 207–221. https://doi.org/10.1007/s10882-014-9409-x.Google Scholar
- Rutter, M., Bailey, A., & Lord, C. (2003). The social communication questionnaire: Manual. Oxford: Western Psychological Services.Google Scholar
- Stel, M., & van Knippenberg, A. (2008). The role of facial mimicry in the recognition of affect. Psychological Science, 19(10), 984–985.Google Scholar
- Stewart, M. E., Barnard, L., Pearson, J., Hasan, R., & O’Brien, G. (2006). Presentation of depression in autism and Asperger syndrome: A review. Autism, 10(1), 103–116.Google Scholar
- Tanaka, J. W., Wolf, J. M., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., et al. (2010). Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s Face It! program. Journal of Child Psychology and Psychiatry, 51(8), 944–952. https://doi.org/10.1111/j.1469-7610.2010.02258.x.Google Scholar
- Tell, D., Davidson, D., & Camras, L. A. (2014). Recognition of emotion from facial expressions with direct or averted eye gaze and varying expression intensities in children with autism disorder and typically developing children. Autism Research and Treatment. https://doi.org/10.1155/2014/816137.Google Scholar
- Thomeer, M. L., Lopata, C., Volker, M. A., Toomey, J. A., Lee, G. K., Smerbeck, A. M., et al. (2012). Randomized clinical trial replication of a psychosocial treatment for children with high-functioning autism spectrum disorders. Psychology in the Schools, 49(10), 942–954. https://doi.org/10.1002/pits.21647.Google Scholar
- Thomeer, M. L., Smith, R. A., Lopata, C., Volker, M. A., Lipinski, A. M., & Rodgers, J. D., et al. (2015). Randomized controlled trial of Mind Reading and in vivo rehearsal for high-functioning children with ASD. Journal of Autism and Developmental Disorders, 45(7), 2115–2127. https://doi.org/10.1007/s10803-015-2374-0.Google Scholar
- Tomkins, S. S., & McCarter, R. (1964). What and where are the primary affects? Some evidence for a theory. Perceptual and Motor Skills, 18(1), 119–158.Google Scholar
- Tracy, J. L., Robins, R. W., Schriber, R. A., & Solomon, M. (2011). Is emotion recognition impaired in individuals with autism spectrum disorders? Journal of Autism and Developmental Disorders, 41(1), 102–109.Google Scholar
- Trevisan, D. A., Bowering, M., & Birmingham, E. (2016). Alexithymia, but not autism spectrum disorder, may be related to the production of emotional facial expressions. Molecular Autism, 7(1), 46.Google Scholar
- Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., & Kissler, J. (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE, 12(5), e0177239.Google Scholar
- Whitaker, L., & Widen, S. C. (2018). How does emotional intelligence relate to adolescents’ interpretation of cues for disgust? Cognition and Emotion, 32(5), 1097–1104.Google Scholar
- Widen, S. C., Pochedly, J. T., & Russell, J. A. (2015). The development of emotion concepts: A story superiority effect in older children and adolescents. Journal of Experimental Child Psychology, 131, 186–192.Google Scholar
- Winkielman, P., McIntosh, D. N., & Oberman, L. (2009). Embodied and disembodied emotion processing: Learning from and about typical and autistic individuals. Emotion Review, 1(2), 178–190.Google Scholar
- Wolf, K. (2015). Measuring facial expression of emotion. Dialogues in Clinical Neuroscience, 17(4), 457.Google Scholar
- Yirmiya, N., Kasari, C., Sigman, M., & Mundy, P. (1989). Facial expressions of affect in autistic, mentally retarded and normal children. Journal of Child Psychology and Psychiatry, 30(5), 725–735. https://doi.org/10.1111/j.1469-7610.1989.tb00785.x.Google Scholar