Automated vs Human Recognition of Emotional Facial Expressions of High-Functioning Children with Autism in a Diagnostic-Technological Context: Explorations via a Bottom-Up Approach
Early detection of autism spectrum conditions (ASC) is an important goal. Automated facial expression recognition is a promising approach and has implications for assistive and educational technologies, too. This study was an initial exploration of (1) the inter-rater reliability of human recognition of facial emotions of high functioning (HF) children with ASC; (2) the relationship between human and automated recognition of facial emotions; and (3) a ‘bottom-up’ approach on identifying ASC/typical development (TD) differences, from a screening serious game context. Thirteen HF, kindergarten-age children with ASC and 13 children with TD, matched along age and IQ, participated. Emotion recognition was administered on video-recordings from sessions of their playing with the serious game. Results showed lack of inter-rater reliability in human coding, confirming some advantages of machine coding. The simple bottom-up cross-sectional exploratory analysis did not reveal any ASC/TD difference. This is in contrast with our and others’ previous results, indicating such differences when aggregating emotion data from wider time-windows in machine-coded data-sets. This suggests that this second approach may be a more promising one to identify autism-specific emotion expression patterns.
KeywordsAutism spectrum conditions Emotional facial expressions Screening Serious game
Ethical Approval and Acknowledgements
This research was approved by the Research Ethics Committee of the ‘Barczi Gusztav’ Faculty of Special Education, ELTE University, Budapest, Hungary. Some elements were funded by a grant within the EIT ICT Labs Hungarian Node (PI: András Lőrincz), and via a TÁMOP grant, co-financed by the European Union and the government of Hungary (TÁMOP 4.2.1./B-09/KMR-2010-0003). The work of M Gyori, Zs Borsos and K Stefanik was supported by a grant from the Hungarian Academy of Sciences within its Content Pedagogy Programme. Authors wish to thank András Lőrincz and Tibor Gregorics for their contributions and support in earlier phases of the project.
- 1.World Health Organization: International Classification of Diseases and Disorders, 10th edn. Author, Geneva (1993)Google Scholar
- 2.APA [American Psychiatric Association]: Diagnostic and Statistical Manual of Mental Disorders, 5th edn, (DSM-5). American Psychiatric Association, Washington DC (2013)Google Scholar
- 3.Chawarska, K., Macari, S., Volkmar, F., Kim, S.H., Shic, F.: Autism in infancy and early childhood. In: Volkmar, F., Rogers, S., Paul, R., Pelphrey, K. (eds.) Handbook of Autism and Pervasive Developmental Disorders. Willey, Hoboken (2014)Google Scholar
- 7.Borsos, Z., Gyori, M.: Can automated facial expression analysis show differences between autism and typical functioning? In: Cudd, P., de Witte, L. (eds.) Harnessing the Power of Technology to Improve Lives. Studies in Health Technology and Informatics, pp. 797–804. IOS Press, Amsterdam (2017)Google Scholar
- 9.Gyori, M., Borsos, Z., Stefanik, K., Csákvári, J.: Data quality as a bottleneck in developing a social-serious-game-based multi-modal system for early screening for ‘High Functioning’ cases of autism spectrum condition. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9759, pp. 358–366. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41267-2_51CrossRefGoogle Scholar
- 11.Gyori, M., Borsos, Z., Stefanik, K.: Evidence-based development and first usability testing of a social serious game based multi-modal system for early screening for atypical socio-cognitive development. In: Sik-Lányi, C., Hoogerwerf, E.J., Miesenberger, K. (eds.) Assistive Technology: Building Bridges. Studies in Health Technology and Informatics, pp. 48–54. IOS Press, Amsterdam (2015)Google Scholar
- 12.Rutter, M., Bailey, A., Lord, C.: The Social Communication Questionnaire: Manual. Western Psychological Services, Los Angeles (2003)Google Scholar
- 13.Lord, C., Rutter, M., DiLavore, P.C., Risi, S.: Autism Diagnostic Observation Schedule. Western Psychological Services, Los Angeles (1999)Google Scholar
- 14.Le Couteur, A., Lord, C., Rutter, M.: The Autism Diagnostic Interview-Revised. Western Psychological Services, Los Angeles (2003)Google Scholar
- 17.Cohn, J.F., Ekman, P.: Measuring facial action. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) The New Handbook of Methods in Nonverbal Behavior Research, pp. 9–64. Oxford University Press (2005)Google Scholar