Journal of Autism and Developmental Disorders

, Volume 49, Issue 1, pp 279–293 | Cite as

Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder

  • Joseph Manfredonia
  • Abigail Bangerter
  • Nikolay V. Manyakov
  • Seth Ness
  • David Lewin
  • Andrew Skalkin
  • Matthew Boice
  • Matthew S. Goodwin
  • Geraldine Dawson
  • Robert Hendren
  • Bennett Leventhal
  • Frederick Shic
  • Gahan Pandina
Original Paper


Facial expression is impaired in autism spectrum disorder (ASD), but rarely systematically studied. We focus on the ability of individuals with ASD to produce facial expressions of emotions in response to a verbal prompt. We used the Janssen Autism Knowledge Engine (JAKE®), including automated facial expression analysis software (FACET) to measure facial expressions in individuals with ASD (n = 144) and a typically developing (TD) comparison group (n = 41). Differences in ability to produce facial expressions were observed between ASD and TD groups, demonstrated by activation of facial action units (happy, scared, surprised, disgusted, but not angry or sad). Activation of facial action units correlated with parent-reported social communication skills. This approach has potential for diagnostic and response to intervention measures.

Trial Registration NCT02299700.


ASD Emotion Facial Expression Automated FACET 



The authors thank the study participants and the following investigators for their participation in this study: Arizona Christopher J. Smith, PhD; California Bennett Leventhal, MD and Robert Hendren; Connecticut (at the time of study conduct): Frederick Shic, PhD Massachusetts Jean Frazier, MD New Jersey Yvette Janvier, MD; New York Russell Tobe, MD; North Carolina Geraldine Dawson, PhD; Pennsylvania Judith S. Miller, PhD; Washington Bryan King, MD.

Author Contributions

JM, AB, SN, NVM, FS, MG, GD, BL, RH and GP were involved in study design. FS, GD, RH, and BL were involved in data collection. DL, NVM, and JM performed the analysis. All authors were involved in interpretation of the results, drafting the manuscript, and read and approved the final manuscript. All authors had full access to all the data in the study and take responsibility for integrity of the data and the accuracy of the data analysis. All authors meet ICMJE criteria and all those who fulfilled those criteria are listed as authors.


This study was funded by Janssen Research & Development, LLC, USA. We acknowledge Sandra Norris, PharmD of the Norris Communications Group LLC for medical writing support, and Ellen Baum, PhD (Janssen Research & Development, LLC) for additional editorial support for the development of this manuscript.

Compliance with Ethical Standards

Conflict of interest

Joseph Manfredonia, Abigail Bangerter, Seth Ness, David Lewin, Andrew Skalkin, Nikolay V. Manyakov and Gahan Pandina are. Robert Hendren received reimbursement for consultation from Janssen Research & Development, LLC. Matthew Goodwin has received research and consulting funding from Janssen Research & Development, LLC. Geraldine Dawson is on the Scientific Advisory Boards of Janssen Research and Development, LLC and Akili, Inc., a consultant to Roche, has received grant funding from Janssen Research and Development, L.L.C. and PerkinElmer, and receives royalties from Guildford Press and Oxford University Press. Bennett Leventhal has received research grant funding from the NIH, is a consultant to Janssen Research and Development, LLC and the Illinois Children’s Healthcare Foundation, and is a board member of the Brain Research Foundation. Frederick Shic is on the Scientific Advisory Board of and is a consultant to Janssen Research and Development, LLC, and has received grant funding from Janssen Research and Development LLC, and Roche.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.


  1. Bangerter, A., Ness, S., Aman, M. G., Esbensen, A. J., Goodwin, M. S., Dawson, G., et al. (2017). Autism Behavior Inventory: A novel tool for assessing core and associated symptoms of autism spectrum disorder. Journal of Child and Adolescent Psychopharmacology, 27(9), 814–822.Google Scholar
  2. Bartlett, M., Littlewort, G., Wu, T., & Movellan, J. (2008). Computer expression recognition toolbox. Presented at the 8th IEEE international Conference on Automatic Face and Gesture Recognition (FG 2008), September 17–19, 2008, Amsterdam, The Netherlands. pp. 1–2.Google Scholar
  3. Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36, 253–263.Google Scholar
  4. Beadle-Brown, J. D., & Whiten, A. (2004). Elicited imitation in children and adults with autism: Is there a deficit? Journal of Intellectual and Developmental Disability, 29(2), 147–163. Scholar
  5. Begeer, S., Koot, H. M., Rieffe, C., Meerum Terwogt, M., & Stegge, H. (2008). Emotional competence in children with autism: Diagnostic criteria and empirical evidence. Developmental Review. Scholar
  6. Berenbaum, H., & Rotter, A. (1992). The relationship between spontaneous facial expressions of emotion and voluntary control of facial muscles. Journal of Nonverbal Behavior, 16, 179–190.Google Scholar
  7. Brewer, R., Biotti, F., Catmur, C., Press, C., Happé, F., Cook, R., et al. (2016). Can neurotypical individuals read autistic facial expressions? A typical production of emotional facial expressions in autism spectrum disorders. Autism Research. Scholar
  8. Constantino, J. N., & Gruber, C. P. (2012). Social responsiveness scale (SRS). Torrance: Western Psychological Services.Google Scholar
  9. Darwin, C. (1872). The expression of emotion in animals and man. London: Methuen.(1877), A biographical sketch of an infant. Mind, 2, 285–294.Google Scholar
  10. Deckers, A., Muris, P., & Roelofs, J. (2017). Being on your own or feeling lonely? Loneliness and other social variables in youths with autism spectrum disorders. Child Psychiatry & Human Development, 48, 828–839.Google Scholar
  11. Denham, S. A. (1986). Social cognition, prosocial behavior, and emotion in preschoolers: contextual validation. Child Development, 57(1), 194–201.Google Scholar
  12. Diano, M., Celeghin, A., Bagnis, A., & Tamietto, M. (2017). Amygdala response to emotional stimuli without awareness: facts and interpretations. Frontiers in Psychology, 7, 2029.Google Scholar
  13. Eack, S. M., Mazefsky, C. A., & Minshew, N. J. (2015). Misinterpretation of facial expressions of emotion in verbal adults with autism spectrum disorder. Autism, 19(3), 308–315.Google Scholar
  14. Ekman, P. (1992). An argument for basic emotions. Cognition & emotion, 6(3–4), 169–200.Google Scholar
  15. Ekman, P., & Friesen, W. V. (1978). Manual for the facial action coding system. Palo Alto: Consulting Psychologists Press.Google Scholar
  16. Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System. Manual and Investigator’s Guide. Salt Lake City, UT: Research Nexus.Google Scholar
  17. Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30(1), 527–554. Scholar
  18. Elliott, R., Zahn, R., Deakin, J. W., & Anderson, I. M. (2011). Affective cognition and its disruption in mood disorders. Neuropsychopharmacology, 36(1), 153–182.Google Scholar
  19. Farroni, T., Menon, E., Rigato, S., & Johnson, M. H. (2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4(1), 2–13.Google Scholar
  20. Faso, D. J., Sasson, N. J., & Pinkham, A. E. (2015). Evaluating posed and evoked facial expressions of emotion from adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 45(1), 75–89. Scholar
  21. Freedman, N. (1972). The Analysis of Movement Behavior During the Clinical Interview. In A. W. Siegman & B. Pope (Eds.), Studies in dyadic communication (Vol. 7, pp. 153–175). Amsterdam: Elsevier B.V.Google Scholar
  22. Friesen, W., & Ekman, P. (1983). EMFACS-7: Emotional Facial Action Coding System. California: Unpublished manual, University of California.Google Scholar
  23. Frith, C. & Series, B. (2009). Role of facial expressions in social interactions. Philosophical Transactions of the Royal Society of London: Biological Sciences, 364(1535), 3453–3458. Scholar
  24. Gordon, I., Pierce, M. D., Bartlett, M. S., & Tanaka, J. W. (2014). Training facial expression production in children on the autism spectrum. Journal of Autism and Developmental Disorders, 44(10), 2486–2498.Google Scholar
  25. Grossman, R. B., Edelson, L. R., & Tager-Flusberg, H. (2013). Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. Journal of Speech, Language, and Hearing Research, 56(3), 1035–1044.Google Scholar
  26. Guarnera, M., Hichy, Z., Cascio, M. I., & Carrubba, S. (2015). Facial expressions and ability to recognize emotions from eyes or mouth in children. Europe’s Journal of Psychology, 11(2), 183.Google Scholar
  27. Gunnery, S. D., Hall, J. A., & Ruben, M. A. (2013). The deliberate Duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior, 37(1), 29–41.Google Scholar
  28. Harms, M. B., Martin, A., & Wallace, G. L. (2010). Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychology Review, 20(3), 290–322. Scholar
  29. iMotions, A./S. (2016). Facial Expression Analysis. Retrieved 10 November, 2017, from
  30. Jack, R. E., Garrod, O. G., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244.Google Scholar
  31. Kasari, C., Sigman, M., Mundy, P., & Yirmiya, N. (1990). Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. Journal of Autism and Developmental Disorders, 20(1), 87–100.Google Scholar
  32. Kim, J. A., Szatmari, P., Bryson, S. E., Streiner, D. L., & Wilson, F. J. (2000). The prevalence of anxiety and mood problems among children with autism and Asperger syndrome. Autism, 4(2), 117–132.Google Scholar
  33. Langdell, T. (1981). Face perception: An approach to the study of autism PhD Thesis, University of London.Google Scholar
  34. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., et al. (2011). The computer expression recognition toolbox (CERT). Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), (pp. 298–305). IEEE.Google Scholar
  35. Loomes, R., Hull, L., & Mandy, W. P. L. (2017). What is the male-to-female ratio in autism spectrum disorder? A systematic review and meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 56(6), 466–474. Scholar
  36. Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. (2012). Autism diagnostic observation schedule: ADOS-2. Los Angeles: Western Psychological Services.Google Scholar
  37. Loveland, K. A., Tunali-Kotoski, B., Pearson, D. A., & Brelsford, K. A. (1994). Imitation and expression of facial affect in autism. Development & Psychopathology, 6(3), 433–444. Scholar
  38. MacDonald, H., Rutter, M., Howlin, P., Rios, P., Conteur, A. L., Evered, C., et al. (1989). Recognition and expression of emotional cues by autistic and normal adults. Journal of Child Psychology and Psychiatry, 30, 865–877. Scholar
  39. McIntosh, D. N., Reichmann-Decker, A., Winkielman, P., & Wilbarger, J. L. (2006). When the social mirror breaks: Deficits in automatic, but not voluntary, mimicry of emotional facial expressions in autism. Developmental Science, 9(3), 295–302. Scholar
  40. Namba, S., Makihara, S., Kabir, R. S., Miyatani, M., & Nakao, T. (2016). Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, 35, 1–13. Scholar
  41. Ness, S. L., Manyakov, N. V., Bangerter, A., Lewin, D., Jagannatha, S., Boice, M., et al. (2017). JAKE® multimodal data capture system: Insights from an observational study of autism spectrum disorder. Frontiers in Neuroscience, 11, 517. Scholar
  42. Orsmond, G. I., Krauss, M. W., & Seltzer, M. M. (2004). Peer relationships and social and recreational activities among adolescents and adults with autism. Journal of Autism and Developmental Disorders, 34(3), 245–256.Google Scholar
  43. Press, C., Richardson, D., & Bird, G. (2010). Intact imitation of emotional facial actions in autism spectrum conditions. Neuropsychologia, 48(11), 3291–3297.Google Scholar
  44. Rodgers, J. D., Thomeer, M. L., Lopata, C., Volker, M. A., Lee, G. K., McDonald, C. A., et al. (2016). RCT of a psychosocial treatment for children with high-functioning ASD: supplemental analyses of treatment effects on facial emotion encoding. Journal of Developmental and Physical Disabilities, 27(2), 207–221. Scholar
  45. Rutter, M., Bailey, A., & Lord, C. (2003). The social communication questionnaire: Manual. Oxford: Western Psychological Services.Google Scholar
  46. Sheppard, E., Pillai, D., Wong, G. T. L., Ropar, D., & Mitchell, P. (2016). How easy is it to read the minds of people with autism spectrum disorder? Journal of Autism and Developmental Disorders. Scholar
  47. Snow, M. E., Hertzig, M. E., & Shapiro, T. (1987). Expression of emotion in young autistic children. Journal of the American Academy of Child & Adolescent Psychiatry, 26(6), 836–838. Scholar
  48. Stagg, S. D., Slavny, R., Hand, C., Cardoso, A., & Smith, P. (2014). Does facial expressivity count? How typically developing children respond initially to children with autism. Autism, 18(6), 704–711. Scholar
  49. Stel, M., & van Knippenberg, A. (2008). The role of facial mimicry in the recognition of affect. Psychological Science, 19(10), 984–985.Google Scholar
  50. Stewart, M. E., Barnard, L., Pearson, J., Hasan, R., & O’Brien, G. (2006). Presentation of depression in autism and Asperger syndrome: A review. Autism, 10(1), 103–116.Google Scholar
  51. Tanaka, J. W., Wolf, J. M., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., et al. (2010). Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s Face It! program. Journal of Child Psychology and Psychiatry, 51(8), 944–952. Scholar
  52. Tell, D., Davidson, D., & Camras, L. A. (2014). Recognition of emotion from facial expressions with direct or averted eye gaze and varying expression intensities in children with autism disorder and typically developing children. Autism Research and Treatment. Scholar
  53. Thomeer, M. L., Lopata, C., Volker, M. A., Toomey, J. A., Lee, G. K., Smerbeck, A. M., et al. (2012). Randomized clinical trial replication of a psychosocial treatment for children with high-functioning autism spectrum disorders. Psychology in the Schools, 49(10), 942–954. Scholar
  54. Thomeer, M. L., Smith, R. A., Lopata, C., Volker, M. A., Lipinski, A. M., & Rodgers, J. D., et al. (2015). Randomized controlled trial of Mind Reading and in vivo rehearsal for high-functioning children with ASD. Journal of Autism and Developmental Disorders, 45(7), 2115–2127. Scholar
  55. Tomkins, S. S., & McCarter, R. (1964). What and where are the primary affects? Some evidence for a theory. Perceptual and Motor Skills, 18(1), 119–158.Google Scholar
  56. Tracy, J. L., Robins, R. W., Schriber, R. A., & Solomon, M. (2011). Is emotion recognition impaired in individuals with autism spectrum disorders? Journal of Autism and Developmental Disorders, 41(1), 102–109.Google Scholar
  57. Trevisan, D. A., Bowering, M., & Birmingham, E. (2016). Alexithymia, but not autism spectrum disorder, may be related to the production of emotional facial expressions. Molecular Autism, 7(1), 46.Google Scholar
  58. Uljarevic, M., & Hamilton, A. (2013). Recognition of emotions in autism: A formal meta-analysis. Journal of Autism and Developmental Disorders, 43(7), 1517–1526. Scholar
  59. Volker, M. A., Lopata, C., Smith, D. A., & Thomeer, M. L. (2009). Facial encoding of children with high-functioning autism spectrum disorders. Focus on Autism and Other Developmental Disabilities, 24(4), 195–204. Scholar
  60. Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., & Kissler, J. (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE, 12(5), e0177239.Google Scholar
  61. Whitaker, L., & Widen, S. C. (2018). How does emotional intelligence relate to adolescents’ interpretation of cues for disgust? Cognition and Emotion, 32(5), 1097–1104.Google Scholar
  62. Widen, S. C., Pochedly, J. T., & Russell, J. A. (2015). The development of emotion concepts: A story superiority effect in older children and adolescents. Journal of Experimental Child Psychology, 131, 186–192.Google Scholar
  63. Winkielman, P., McIntosh, D. N., & Oberman, L. (2009). Embodied and disembodied emotion processing: Learning from and about typical and autistic individuals. Emotion Review, 1(2), 178–190.Google Scholar
  64. Wolf, K. (2015). Measuring facial expression of emotion. Dialogues in Clinical Neuroscience, 17(4), 457.Google Scholar
  65. Yirmiya, N., Kasari, C., Sigman, M., & Mundy, P. (1989). Facial expressions of affect in autistic, mentally retarded and normal children. Journal of Child Psychology and Psychiatry, 30(5), 725–735. Scholar
  66. Yoshimura, S., Sato, W., Uono, S., & Toichi, M. (2015). Impaired overt facial mimicry in response to dynamic facial expressions in high-functioning autism spectrum disorders. Journal of Autism and Developmental Disorders, 45(5), 1318–1328. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Joseph Manfredonia
    • 1
  • Abigail Bangerter
    • 2
  • Nikolay V. Manyakov
    • 3
  • Seth Ness
    • 2
  • David Lewin
    • 4
  • Andrew Skalkin
    • 5
  • Matthew Boice
    • 2
  • Matthew S. Goodwin
    • 6
  • Geraldine Dawson
    • 7
  • Robert Hendren
    • 8
  • Bennett Leventhal
    • 8
  • Frederick Shic
    • 9
  • Gahan Pandina
    • 2
  1. 1.Department of Advanced Analytics, Johnson & JohnsonJanssen Research & Development, LLCSomervilleUSA
  2. 2.Janssen Research & Development, LLCTitusvilleUSA
  3. 3.Janssen Research & DevelopmentBeerseBelgium
  4. 4.Janssen Research & Development, LLCRaritanUSA
  5. 5.Janssen Research & Development, LLCSpring HouseUSA
  6. 6.Department of Health Sciences, Bouvé College of Health SciencesNortheastern UniversityBostonUSA
  7. 7.Duke Center for Autism and Brain DevelopmentDuke University School of MedicineDurhamUSA
  8. 8.Benioff Children’s HospitalUniversity of California, San FranciscoSan FranciscoUSA
  9. 9.Department of Pediatrics, Center for Child Health, Behavior and Development, Seattle Children’s Research InstituteUniversity of WashingtonSeattleUSA

Personalised recommendations