Automated vs Human Recognition of Emotional Facial Expressions of High-Functioning Children with Autism in a Diagnostic-Technological Context: Explorations via a Bottom-Up Approach

  • Miklos Gyori
  • Zsófia Borsos
  • Krisztina Stefanik
  • Zoltán Jakab
  • Fanni Varga
  • Judit Csákvári
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10896)


Early detection of autism spectrum conditions (ASC) is an important goal. Automated facial expression recognition is a promising approach and has implications for assistive and educational technologies, too. This study was an initial exploration of (1) the inter-rater reliability of human recognition of facial emotions of high functioning (HF) children with ASC; (2) the relationship between human and automated recognition of facial emotions; and (3) a ‘bottom-up’ approach on identifying ASC/typical development (TD) differences, from a screening serious game context. Thirteen HF, kindergarten-age children with ASC and 13 children with TD, matched along age and IQ, participated. Emotion recognition was administered on video-recordings from sessions of their playing with the serious game. Results showed lack of inter-rater reliability in human coding, confirming some advantages of machine coding. The simple bottom-up cross-sectional exploratory analysis did not reveal any ASC/TD difference. This is in contrast with our and others’ previous results, indicating such differences when aggregating emotion data from wider time-windows in machine-coded data-sets. This suggests that this second approach may be a more promising one to identify autism-specific emotion expression patterns.


Autism spectrum conditions Emotional facial expressions Screening Serious game 


Ethical Approval and Acknowledgements

This research was approved by the Research Ethics Committee of the ‘Barczi Gusztav’ Faculty of Special Education, ELTE University, Budapest, Hungary. Some elements were funded by a grant within the EIT ICT Labs Hungarian Node (PI: András Lőrincz), and via a TÁMOP grant, co-financed by the European Union and the government of Hungary (TÁMOP 4.2.1./B-09/KMR-2010-0003). The work of M Gyori, Zs Borsos and K Stefanik was supported by a grant from the Hungarian Academy of Sciences within its Content Pedagogy Programme. Authors wish to thank András Lőrincz and Tibor Gregorics for their contributions and support in earlier phases of the project.


  1. 1.
    World Health Organization: International Classification of Diseases and Disorders, 10th edn. Author, Geneva (1993)Google Scholar
  2. 2.
    APA [American Psychiatric Association]: Diagnostic and Statistical Manual of Mental Disorders, 5th edn, (DSM-5). American Psychiatric Association, Washington DC (2013)Google Scholar
  3. 3.
    Chawarska, K., Macari, S., Volkmar, F., Kim, S.H., Shic, F.: Autism in infancy and early childhood. In: Volkmar, F., Rogers, S., Paul, R., Pelphrey, K. (eds.) Handbook of Autism and Pervasive Developmental Disorders. Willey, Hoboken (2014)Google Scholar
  4. 4.
    Brett, D., Warnell, F., McConachie, H., Parr, J.R.: Factors affecting age at ASD diagnosis in UK: no evidence that diagnosis age has decreased between 2004 and 2014. J. Autism Dev. Disord. 46, 1974–1984 (2016)CrossRefGoogle Scholar
  5. 5.
    Daniels, A.M., Mandell, D.S.: Explaining differences in age at autism spectrum disorder diagnosis: a critical review. Autism 18, 583–597 (2014)CrossRefGoogle Scholar
  6. 6.
    Bölte, S., Bartl-Pokorny, K.D., Jonsson, U., Berggren, S., Zhang, D., Kostrzewa, E., et al.: How can clinicians detect and treat autism early? Methodological trends of technology use in research. Acta Paediatr. 105(2), 137–144 (2016)CrossRefGoogle Scholar
  7. 7.
    Borsos, Z., Gyori, M.: Can automated facial expression analysis show differences between autism and typical functioning? In: Cudd, P., de Witte, L. (eds.) Harnessing the Power of Technology to Improve Lives. Studies in Health Technology and Informatics, pp. 797–804. IOS Press, Amsterdam (2017)Google Scholar
  8. 8.
    Owada, K., Kojima, M., Yassin, W., Kuroda, M., Kawakobo, Y., Kuwabara, H., Kano, Y., Yamasue, H.: Computer-analyzed facial expression as a surrogate marker for autism spectrum social core symptoms. PLoS ONE 13(1), e0190442 (2018)CrossRefGoogle Scholar
  9. 9.
    Gyori, M., Borsos, Z., Stefanik, K., Csákvári, J.: Data quality as a bottleneck in developing a social-serious-game-based multi-modal system for early screening for ‘High Functioning’ cases of autism spectrum condition. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9759, pp. 358–366. Springer, Cham (2016). Scholar
  10. 10.
    Grossard, C., Grynspan, O., Serret, S., Jouen, A.-L., Bailly, K., Cohen, D.: Serious games to teach social interactions and emotions to individuals with autism spectrum disorders (ASD). Comput. Educ. 113, 195–211 (2017)CrossRefGoogle Scholar
  11. 11.
    Gyori, M., Borsos, Z., Stefanik, K.: Evidence-based development and first usability testing of a social serious game based multi-modal system for early screening for atypical socio-cognitive development. In: Sik-Lányi, C., Hoogerwerf, E.J., Miesenberger, K. (eds.) Assistive Technology: Building Bridges. Studies in Health Technology and Informatics, pp. 48–54. IOS Press, Amsterdam (2015)Google Scholar
  12. 12.
    Rutter, M., Bailey, A., Lord, C.: The Social Communication Questionnaire: Manual. Western Psychological Services, Los Angeles (2003)Google Scholar
  13. 13.
    Lord, C., Rutter, M., DiLavore, P.C., Risi, S.: Autism Diagnostic Observation Schedule. Western Psychological Services, Los Angeles (1999)Google Scholar
  14. 14.
    Le Couteur, A., Lord, C., Rutter, M.: The Autism Diagnostic Interview-Revised. Western Psychological Services, Los Angeles (2003)Google Scholar
  15. 15.
    Sodian, B., Frith, U.: Deception and sabotage in autistic, retarded and normal children. J. Child Psychol. Psychiatry 33, 591–605 (1992)CrossRefGoogle Scholar
  16. 16.
    Kring, A.M., Sloan, D.M.: The facial expression coding system (FACES): development, validation, and utility. Psychol. Assess. 19(2), 210–224 (2007)CrossRefGoogle Scholar
  17. 17.
    Cohn, J.F., Ekman, P.: Measuring facial action. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) The New Handbook of Methods in Nonverbal Behavior Research, pp. 9–64. Oxford University Press (2005)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.ELTE UniversityBudapestHungary
  2. 2.MTA-ELTE ‘Autism in Education’ Research GroupBudapestHungary

Personalised recommendations