Skip to main content
SpringerLink
Log in

Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder

  • Original Paper
  • Published:
Journal of Autism and Developmental Disorders Aims and scope Submit manuscript

Abstract

Facial expression is impaired in autism spectrum disorder (ASD), but rarely systematically studied. We focus on the ability of individuals with ASD to produce facial expressions of emotions in response to a verbal prompt. We used the Janssen Autism Knowledge Engine (JAKE®), including automated facial expression analysis software (FACET) to measure facial expressions in individuals with ASD (n = 144) and a typically developing (TD) comparison group (n = 41). Differences in ability to produce facial expressions were observed between ASD and TD groups, demonstrated by activation of facial action units (happy, scared, surprised, disgusted, but not angry or sad). Activation of facial action units correlated with parent-reported social communication skills. This approach has potential for diagnostic and response to intervention measures.

Trial Registration NCT02299700.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Bangerter, A., Ness, S., Aman, M. G., Esbensen, A. J., Goodwin, M. S., Dawson, G., et al. (2017). Autism Behavior Inventory: A novel tool for assessing core and associated symptoms of autism spectrum disorder. Journal of Child and Adolescent Psychopharmacology, 27(9), 814–822.

    Article  Google Scholar 

  • Bartlett, M., Littlewort, G., Wu, T., & Movellan, J. (2008). Computer expression recognition toolbox. Presented at the 8th IEEE international Conference on Automatic Face and Gesture Recognition (FG 2008), September 17–19, 2008, Amsterdam, The Netherlands. pp. 1–2.

  • Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36, 253–263.

    Article  Google Scholar 

  • Beadle-Brown, J. D., & Whiten, A. (2004). Elicited imitation in children and adults with autism: Is there a deficit? Journal of Intellectual and Developmental Disability, 29(2), 147–163. https://doi.org/10.1080/13668250410001709494.

    Article  Google Scholar 

  • Begeer, S., Koot, H. M., Rieffe, C., Meerum Terwogt, M., & Stegge, H. (2008). Emotional competence in children with autism: Diagnostic criteria and empirical evidence. Developmental Review. https://doi.org/10.1016/j.dr.2007.09.001.

    Article  Google Scholar 

  • Berenbaum, H., & Rotter, A. (1992). The relationship between spontaneous facial expressions of emotion and voluntary control of facial muscles. Journal of Nonverbal Behavior, 16, 179–190.

    Article  Google Scholar 

  • Brewer, R., Biotti, F., Catmur, C., Press, C., Happé, F., Cook, R., et al. (2016). Can neurotypical individuals read autistic facial expressions? A typical production of emotional facial expressions in autism spectrum disorders. Autism Research. https://doi.org/10.1002/aur.1508.

    Article  PubMed  Google Scholar 

  • Constantino, J. N., & Gruber, C. P. (2012). Social responsiveness scale (SRS). Torrance: Western Psychological Services.

    Google Scholar 

  • Darwin, C. (1872). The expression of emotion in animals and man. London: Methuen.(1877), A biographical sketch of an infant. Mind, 2, 285–294.

    Google Scholar 

  • Deckers, A., Muris, P., & Roelofs, J. (2017). Being on your own or feeling lonely? Loneliness and other social variables in youths with autism spectrum disorders. Child Psychiatry & Human Development, 48, 828–839.

    Article  Google Scholar 

  • Denham, S. A. (1986). Social cognition, prosocial behavior, and emotion in preschoolers: contextual validation. Child Development, 57(1), 194–201.

    Article  Google Scholar 

  • Diano, M., Celeghin, A., Bagnis, A., & Tamietto, M. (2017). Amygdala response to emotional stimuli without awareness: facts and interpretations. Frontiers in Psychology, 7, 2029.

    Article  Google Scholar 

  • Eack, S. M., Mazefsky, C. A., & Minshew, N. J. (2015). Misinterpretation of facial expressions of emotion in verbal adults with autism spectrum disorder. Autism, 19(3), 308–315.

    Article  Google Scholar 

  • Ekman, P. (1992). An argument for basic emotions. Cognition & emotion, 6(3–4), 169–200.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1978). Manual for the facial action coding system. Palo Alto: Consulting Psychologists Press.

    Google Scholar 

  • Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System. Manual and Investigator’s Guide. Salt Lake City, UT: Research Nexus.

    Google Scholar 

  • Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30(1), 527–554. https://doi.org/10.1146/annurev.ps.30.020179.002523.

    Article  Google Scholar 

  • Elliott, R., Zahn, R., Deakin, J. W., & Anderson, I. M. (2011). Affective cognition and its disruption in mood disorders. Neuropsychopharmacology, 36(1), 153–182.

    Article  Google Scholar 

  • Farroni, T., Menon, E., Rigato, S., & Johnson, M. H. (2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4(1), 2–13.

    Article  Google Scholar 

  • Faso, D. J., Sasson, N. J., & Pinkham, A. E. (2015). Evaluating posed and evoked facial expressions of emotion from adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 45(1), 75–89. https://doi.org/10.1007/s10803-014-2194-7.

    Article  PubMed  Google Scholar 

  • Freedman, N. (1972). The Analysis of Movement Behavior During the Clinical Interview. In A. W. Siegman & B. Pope (Eds.), Studies in dyadic communication (Vol. 7, pp. 153–175). Amsterdam: Elsevier B.V.

    Chapter  Google Scholar 

  • Friesen, W., & Ekman, P. (1983). EMFACS-7: Emotional Facial Action Coding System. California: Unpublished manual, University of California.

    Google Scholar 

  • Frith, C. & Series, B. (2009). Role of facial expressions in social interactions. Philosophical Transactions of the Royal Society of London: Biological Sciences, 364(1535), 3453–3458. https://doi.org/10.1098/rstb.2009.0142.

    Article  PubMed  Google Scholar 

  • Gordon, I., Pierce, M. D., Bartlett, M. S., & Tanaka, J. W. (2014). Training facial expression production in children on the autism spectrum. Journal of Autism and Developmental Disorders, 44(10), 2486–2498.

    Article  Google Scholar 

  • Grossman, R. B., Edelson, L. R., & Tager-Flusberg, H. (2013). Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. Journal of Speech, Language, and Hearing Research, 56(3), 1035–1044.

    Article  Google Scholar 

  • Guarnera, M., Hichy, Z., Cascio, M. I., & Carrubba, S. (2015). Facial expressions and ability to recognize emotions from eyes or mouth in children. Europe’s Journal of Psychology, 11(2), 183.

    Article  Google Scholar 

  • Gunnery, S. D., Hall, J. A., & Ruben, M. A. (2013). The deliberate Duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior, 37(1), 29–41.

    Article  Google Scholar 

  • Harms, M. B., Martin, A., & Wallace, G. L. (2010). Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychology Review, 20(3), 290–322. https://doi.org/10.1007/s11065-010-9138-6.

    Article  Google Scholar 

  • iMotions, A./S. (2016). Facial Expression Analysis. Retrieved 10 November, 2017, from https://imotions.com/guides.

  • Jack, R. E., Garrod, O. G., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244.

    Article  Google Scholar 

  • Kasari, C., Sigman, M., Mundy, P., & Yirmiya, N. (1990). Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. Journal of Autism and Developmental Disorders, 20(1), 87–100.

    Article  Google Scholar 

  • Kim, J. A., Szatmari, P., Bryson, S. E., Streiner, D. L., & Wilson, F. J. (2000). The prevalence of anxiety and mood problems among children with autism and Asperger syndrome. Autism, 4(2), 117–132.

    Article  Google Scholar 

  • Langdell, T. (1981). Face perception: An approach to the study of autism PhD Thesis, University of London.

  • Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., et al. (2011). The computer expression recognition toolbox (CERT). Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), (pp. 298–305). IEEE.

  • Loomes, R., Hull, L., & Mandy, W. P. L. (2017). What is the male-to-female ratio in autism spectrum disorder? A systematic review and meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 56(6), 466–474. https://doi.org/10.1016/j.jaac.2017.03.013.

    Article  Google Scholar 

  • Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. (2012). Autism diagnostic observation schedule: ADOS-2. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Loveland, K. A., Tunali-Kotoski, B., Pearson, D. A., & Brelsford, K. A. (1994). Imitation and expression of facial affect in autism. Development & Psychopathology, 6(3), 433–444. https://doi.org/10.1017/S0954579400006039.

    Article  Google Scholar 

  • MacDonald, H., Rutter, M., Howlin, P., Rios, P., Conteur, A. L., Evered, C., et al. (1989). Recognition and expression of emotional cues by autistic and normal adults. Journal of Child Psychology and Psychiatry, 30, 865–877. https://doi.org/10.1111/j.1469-7610.1989.tb00288.x.

    Article  PubMed  Google Scholar 

  • McIntosh, D. N., Reichmann-Decker, A., Winkielman, P., & Wilbarger, J. L. (2006). When the social mirror breaks: Deficits in automatic, but not voluntary, mimicry of emotional facial expressions in autism. Developmental Science, 9(3), 295–302. https://doi.org/10.1111/j.1467-7687.2006.00492.x.

    Article  PubMed  Google Scholar 

  • Namba, S., Makihara, S., Kabir, R. S., Miyatani, M., & Nakao, T. (2016). Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, 35, 1–13. https://doi.org/10.1007/s12144-016-9448-9.

    Article  Google Scholar 

  • Ness, S. L., Manyakov, N. V., Bangerter, A., Lewin, D., Jagannatha, S., Boice, M., et al. (2017). JAKE® multimodal data capture system: Insights from an observational study of autism spectrum disorder. Frontiers in Neuroscience, 11, 517. https://doi.org/10.3389/fnins.2017.00517.

    Article  PubMed  PubMed Central  Google Scholar 

  • Orsmond, G. I., Krauss, M. W., & Seltzer, M. M. (2004). Peer relationships and social and recreational activities among adolescents and adults with autism. Journal of Autism and Developmental Disorders, 34(3), 245–256.

    Article  Google Scholar 

  • Press, C., Richardson, D., & Bird, G. (2010). Intact imitation of emotional facial actions in autism spectrum conditions. Neuropsychologia, 48(11), 3291–3297.

    Article  Google Scholar 

  • Rodgers, J. D., Thomeer, M. L., Lopata, C., Volker, M. A., Lee, G. K., McDonald, C. A., et al. (2016). RCT of a psychosocial treatment for children with high-functioning ASD: supplemental analyses of treatment effects on facial emotion encoding. Journal of Developmental and Physical Disabilities, 27(2), 207–221. https://doi.org/10.1007/s10882-014-9409-x.

    Article  Google Scholar 

  • Rutter, M., Bailey, A., & Lord, C. (2003). The social communication questionnaire: Manual. Oxford: Western Psychological Services.

    Google Scholar 

  • Sheppard, E., Pillai, D., Wong, G. T. L., Ropar, D., & Mitchell, P. (2016). How easy is it to read the minds of people with autism spectrum disorder? Journal of Autism and Developmental Disorders. https://doi.org/10.1007/s10803-015-2662-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Snow, M. E., Hertzig, M. E., & Shapiro, T. (1987). Expression of emotion in young autistic children. Journal of the American Academy of Child & Adolescent Psychiatry, 26(6), 836–838. https://doi.org/10.1097/00004583-198726060-00006.

    Article  Google Scholar 

  • Stagg, S. D., Slavny, R., Hand, C., Cardoso, A., & Smith, P. (2014). Does facial expressivity count? How typically developing children respond initially to children with autism. Autism, 18(6), 704–711. https://doi.org/10.1177/1362361313492392.

    Article  PubMed  Google Scholar 

  • Stel, M., & van Knippenberg, A. (2008). The role of facial mimicry in the recognition of affect. Psychological Science, 19(10), 984–985.

    Article  Google Scholar 

  • Stewart, M. E., Barnard, L., Pearson, J., Hasan, R., & O’Brien, G. (2006). Presentation of depression in autism and Asperger syndrome: A review. Autism, 10(1), 103–116.

    Article  Google Scholar 

  • Tanaka, J. W., Wolf, J. M., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., et al. (2010). Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s Face It! program. Journal of Child Psychology and Psychiatry, 51(8), 944–952. https://doi.org/10.1111/j.1469-7610.2010.02258.x.

    Article  PubMed  Google Scholar 

  • Tell, D., Davidson, D., & Camras, L. A. (2014). Recognition of emotion from facial expressions with direct or averted eye gaze and varying expression intensities in children with autism disorder and typically developing children. Autism Research and Treatment. https://doi.org/10.1155/2014/816137.

    Article  PubMed  PubMed Central  Google Scholar 

  • Thomeer, M. L., Lopata, C., Volker, M. A., Toomey, J. A., Lee, G. K., Smerbeck, A. M., et al. (2012). Randomized clinical trial replication of a psychosocial treatment for children with high-functioning autism spectrum disorders. Psychology in the Schools, 49(10), 942–954. https://doi.org/10.1002/pits.21647.

    Article  Google Scholar 

  • Thomeer, M. L., Smith, R. A., Lopata, C., Volker, M. A., Lipinski, A. M., & Rodgers, J. D., et al. (2015). Randomized controlled trial of Mind Reading and in vivo rehearsal for high-functioning children with ASD. Journal of Autism and Developmental Disorders, 45(7), 2115–2127. https://doi.org/10.1007/s10803-015-2374-0.

    Article  PubMed  Google Scholar 

  • Tomkins, S. S., & McCarter, R. (1964). What and where are the primary affects? Some evidence for a theory. Perceptual and Motor Skills, 18(1), 119–158.

    Article  Google Scholar 

  • Tracy, J. L., Robins, R. W., Schriber, R. A., & Solomon, M. (2011). Is emotion recognition impaired in individuals with autism spectrum disorders? Journal of Autism and Developmental Disorders, 41(1), 102–109.

    Article  Google Scholar 

  • Trevisan, D. A., Bowering, M., & Birmingham, E. (2016). Alexithymia, but not autism spectrum disorder, may be related to the production of emotional facial expressions. Molecular Autism, 7(1), 46.

    Article  Google Scholar 

  • Uljarevic, M., & Hamilton, A. (2013). Recognition of emotions in autism: A formal meta-analysis. Journal of Autism and Developmental Disorders, 43(7), 1517–1526. https://doi.org/10.1007/s10803-012-1695-5.

    Article  Google Scholar 

  • Volker, M. A., Lopata, C., Smith, D. A., & Thomeer, M. L. (2009). Facial encoding of children with high-functioning autism spectrum disorders. Focus on Autism and Other Developmental Disabilities, 24(4), 195–204. https://doi.org/10.1177/1088357609347325.

    Article  Google Scholar 

  • Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., & Kissler, J. (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE, 12(5), e0177239.

    Article  Google Scholar 

  • Whitaker, L., & Widen, S. C. (2018). How does emotional intelligence relate to adolescents’ interpretation of cues for disgust? Cognition and Emotion, 32(5), 1097–1104.

    Article  Google Scholar 

  • Widen, S. C., Pochedly, J. T., & Russell, J. A. (2015). The development of emotion concepts: A story superiority effect in older children and adolescents. Journal of Experimental Child Psychology, 131, 186–192.

    Article  Google Scholar 

  • Winkielman, P., McIntosh, D. N., & Oberman, L. (2009). Embodied and disembodied emotion processing: Learning from and about typical and autistic individuals. Emotion Review, 1(2), 178–190.

    Article  Google Scholar 

  • Wolf, K. (2015). Measuring facial expression of emotion. Dialogues in Clinical Neuroscience, 17(4), 457.

    PubMed  PubMed Central  Google Scholar 

  • Yirmiya, N., Kasari, C., Sigman, M., & Mundy, P. (1989). Facial expressions of affect in autistic, mentally retarded and normal children. Journal of Child Psychology and Psychiatry, 30(5), 725–735. https://doi.org/10.1111/j.1469-7610.1989.tb00785.x.

    Article  PubMed  Google Scholar 

  • Yoshimura, S., Sato, W., Uono, S., & Toichi, M. (2015). Impaired overt facial mimicry in response to dynamic facial expressions in high-functioning autism spectrum disorders. Journal of Autism and Developmental Disorders, 45(5), 1318–1328. https://doi.org/10.1007/s10803-014-2291-7.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The authors thank the study participants and the following investigators for their participation in this study: Arizona Christopher J. Smith, PhD; California Bennett Leventhal, MD and Robert Hendren; Connecticut (at the time of study conduct): Frederick Shic, PhD Massachusetts Jean Frazier, MD New Jersey Yvette Janvier, MD; New York Russell Tobe, MD; North Carolina Geraldine Dawson, PhD; Pennsylvania Judith S. Miller, PhD; Washington Bryan King, MD.

Funding

This study was funded by Janssen Research & Development, LLC, USA. We acknowledge Sandra Norris, PharmD of the Norris Communications Group LLC for medical writing support, and Ellen Baum, PhD (Janssen Research & Development, LLC) for additional editorial support for the development of this manuscript.

Author information

Author notes

  1. Joseph Manfredonia

    Present address: Department of Advanced Analytics, Johnson & Johnson, Janssen Research & Development, LLC, 555 Route 22, Somerville, NJ, 08807, USA

  2. Frederick Shic

    Present address: Department of Pediatrics, Center for Child Health, Behavior and Development, Seattle Children’s Research Institute, University of Washington, 2001 8th Ave Suite #400, Seattle, WA, 98121, USA

Authors and Affiliations

  1. Janssen Research & Development, LLC, 1125 Trenton-Harbourton Road, Titusville, NJ, 08560, USA

    Abigail Bangerter, Seth Ness, Matthew Boice & Gahan Pandina

  2. Janssen Research & Development, Turnhoutseweg 30, 2340, Beerse, Belgium

    Nikolay V. Manyakov

  3. Janssen Research & Development, LLC, 920 Rte. 202, Raritan, NJ, 08869-0602, USA

    David Lewin

  4. Janssen Research & Development, LLC, PO Box 776, Welsh & McKean Roads, Spring House, PA, 19477-0776, USA

    Andrew Skalkin

  5. Department of Health Sciences, Bouvé College of Health Sciences, Northeastern University, 312E Robinson Hall, 360 Huntington Avenue, Boston, MA, 02115, USA

    Matthew S. Goodwin

  6. Duke Center for Autism and Brain Development, Duke University School of Medicine, 2608 Erwin Road, Suite 30, Durham, NC, 27705, USA

    Geraldine Dawson

  7. Benioff Children’s Hospital, University of California, San Francisco, 401 Parnassus Ave, Langley Porter, San Francisco, CA, 94143-0984, USA

    Robert Hendren & Bennett Leventhal

Authors
  1. Joseph Manfredonia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abigail Bangerter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikolay V. Manyakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seth Ness
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Lewin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrew Skalkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Matthew Boice
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Matthew S. Goodwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Geraldine Dawson
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Robert Hendren
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Bennett Leventhal
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Frederick Shic
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Gahan Pandina
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JM, AB, SN, NVM, FS, MG, GD, BL, RH and GP were involved in study design. FS, GD, RH, and BL were involved in data collection. DL, NVM, and JM performed the analysis. All authors were involved in interpretation of the results, drafting the manuscript, and read and approved the final manuscript. All authors had full access to all the data in the study and take responsibility for integrity of the data and the accuracy of the data analysis. All authors meet ICMJE criteria and all those who fulfilled those criteria are listed as authors.

Corresponding author

Correspondence to Joseph Manfredonia.

Ethics declarations

Conflict of interest

Joseph Manfredonia, Abigail Bangerter, Seth Ness, David Lewin, Andrew Skalkin, Nikolay V. Manyakov and Gahan Pandina are. Robert Hendren received reimbursement for consultation from Janssen Research & Development, LLC. Matthew Goodwin has received research and consulting funding from Janssen Research & Development, LLC. Geraldine Dawson is on the Scientific Advisory Boards of Janssen Research and Development, LLC and Akili, Inc., a consultant to Roche, has received grant funding from Janssen Research and Development, L.L.C. and PerkinElmer, and receives royalties from Guildford Press and Oxford University Press. Bennett Leventhal has received research grant funding from the NIH, is a consultant to Janssen Research and Development, LLC and the Illinois Children’s Healthcare Foundation, and is a board member of the Brain Research Foundation. Frederick Shic is on the Scientific Advisory Board of and is a consultant to Janssen Research and Development, LLC, and has received grant funding from Janssen Research and Development LLC, and Roche.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Manfredonia, J., Bangerter, A., Manyakov, N.V. et al. Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder. J Autism Dev Disord 49, 279–293 (2019). https://doi.org/10.1007/s10803-018-3757-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-018-3757-9

Keywords

Search

Navigation