Detecting Autism by Analyzing a Simulated Social Interaction

  • Hanna DrimallaEmail author
  • Niels Landwehr
  • Irina Baskow
  • Behnoush Behnia
  • Stefan Roepke
  • Isabel Dziobek
  • Tobias Scheffer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11051)


Diagnosing autism spectrum conditions takes several hours by well-trained practitioners; therefore, standardized questionnaires are widely used for first-level screening. Questionnaires as a diagnostic tool, however, rely on self-reflection—which is typically impaired in individuals with autism spectrum condition. We develop an alternative screening mechanism in which subjects engage in a simulated social interaction. During this interaction, the subjects’ voice, eye gaze, and facial expression are tracked, and features are extracted that serve as input to a predictive model. We find that a random-forest classifier on these features can detect autism spectrum condition accurately and functionally independently of diagnostic questionnaires. We also find that a regression model estimates the severity of the condition more accurately than the reference screening method.



This work was partially funded by the German Science Foundation under grant LA3270/1-1.


  1. 1.
    Mattila, M.L., et al.: Autism spectrum disorders according to DSM-IV-TR and comparison with DSM-5 draft criteria: an epidemiological study. J. Am. Acad. Child Adolesc. Psychiatry 50(6), 583–592.e11 (2011)CrossRefGoogle Scholar
  2. 2.
    American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders: DSM-5, 5th edn. American Psychiatric Association, Arlington (2013)CrossRefGoogle Scholar
  3. 3.
    Newschaffer, C.J., et al.: The epidemiology of autism spectrum disorders. Annu. Rev. Public Health 28, 235–258 (2007)CrossRefGoogle Scholar
  4. 4.
    Barnard, J., Harvey, V., Potter, D.: Ignored or Ineligible? The reality for adults with autism spectrum disorders. National Autistic Society (2001)Google Scholar
  5. 5.
    Harms, M.B., Martin, A., Wallace, G.L.: Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychol. Rev. 20(3), 290–322 (2010)CrossRefGoogle Scholar
  6. 6.
    White, S.W., Ollendick, T.H., Bray, B.C.: College students on the autism spectrum: prevalence and associated problems. Autism Int. J. Res. Pract. 15(6), 683–701 (2011)CrossRefGoogle Scholar
  7. 7.
    Lehnhardt, F.G., Gawronski, A., Volpert, K., Schilbach, L., Tepest, R., Vogeley, K.: Das psychosoziale funktionsniveau spätdiagnostizierter patienten mit autismus- spektrum-störungen–eine retrospektive untersuchung im erwachsenenalter. Fortschr. Neurol. Psychiatr. 80(2), 88–97 (2012)CrossRefGoogle Scholar
  8. 8.
    Lord, C., et al.: The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J. Autism Dev. Disord. 30(3), 205–223 (2000)CrossRefGoogle Scholar
  9. 9.
    Lord, C., Rutter, M., Le Couteur, A.: Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J. Autism Dev. Disord. 24(5), 659–685 (1994)CrossRefGoogle Scholar
  10. 10.
    World Health Organization: The ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines. ICD-10 Classification of Mental and Behavioural Disorders/World Health Organization. World Health Organization (1992)Google Scholar
  11. 11.
    Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., Clubley, E.: The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians. J. Autism Dev. Disord. 31(1), 5–17 (2001)CrossRefGoogle Scholar
  12. 12.
    Woodbury-Smith, M.R., Robinson, J., Wheelwright, S., Baron-Cohen, S.: Screening adults for asperger syndrome using the AQ: a preliminary study of its diagnostic validity in clinical practice. J. Autism Dev. Disord. 35(3), 331–335 (2005)CrossRefGoogle Scholar
  13. 13.
    Van de Mortel, T.F., et al.: Faking it: social desirability response bias in self-report research. Aust. J. Adv. Nurs. 25(4), 40 (2008)Google Scholar
  14. 14.
    Happé, F.: Theory of mind and the self. Ann. N. Y. Acad. Sci. 1001(1), 134–144 (2003)CrossRefGoogle Scholar
  15. 15.
    Minshew, N.J., Meyer, J., Goldstein, G.: Abstract reasoning in autism: a disassociation between concept formation and concept identification. Neuropsychology 16(3), 327 (2002)CrossRefGoogle Scholar
  16. 16.
    Crippa, A., et al.: Use of machine learning to identify children with autism and their motor abnormalities. J. Autism Dev. Disord. 45(7), 2146–2156 (2015)CrossRefGoogle Scholar
  17. 17.
    Hashemi, J., et al.: A computer vision approach for the assessment of autism-related behavioral markers. In: 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1–7. IEEE (2012)Google Scholar
  18. 18.
    Bryson, S.E., Zwaigenbaum, L.: Autism observation scale for infants. In: Patel, V., Preedy, V., Martin, C. (eds.) Comprehensive Guide to Autism, pp. 299–310. Springer, New York (2014). Scholar
  19. 19.
    Liu, W., Li, M., Yi, L.: Identifying children with autism spectrum disorder based on their face processing abnormality: a machine learning framework. Autism Res. Off. J. Int. Soc. Autism Res. 9(8), 888–898 (2016)CrossRefGoogle Scholar
  20. 20.
    Pelphrey, K.A., Sasson, N.J., Reznick, J.S., Paul, G., Goldman, B.D., Piven, J.: Visual scanning of faces in autism. J. Autism Dev. Disord. 32(4), 249–261 (2002)CrossRefGoogle Scholar
  21. 21.
    Gliga, T., Bedford, R., Charman, T., Johnson, M.H.: Enhanced visual search in infancy predicts emerging autism symptoms. Curr. Biol. CB 25(13), 1727–1730 (2015)CrossRefGoogle Scholar
  22. 22.
    Nakai, Y., Takiguchi, T., Matsui, G., Yamaoka, N., Takada, S.: Detecting abnormal word utterances in children with autism spectrum disorders: machine-learning-based voice analysis versus speech therapists. Percept. Mot. Ski. 124(5), 961–973 (2017)CrossRefGoogle Scholar
  23. 23.
    Nasir, M., Jati, A., Shivakumar, P.G., Nallan Chakravarthula, S., Georgiou, P.: Multimodal and multiresolution depression detection from speech and facial landmark features. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 43–50. ACM (2016)Google Scholar
  24. 24.
    Laksana, E., Baltrušaitis, T., Morency, L.P., Pestian, J.P.: Investigating facial behavior indicators of suicidal ideation. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 770–777. IEEE (2017)Google Scholar
  25. 25.
    Tron, T., Peled, A., Grinsphoon, A., Weinshall, D.: Automated facial expressions analysis in schizophrenia: a continuous dynamic approach. In: Serino, S., Matic, A., Giakoumis, D., Lopez, G., Cipresso, P. (eds.) MindCare 2015. CCIS, vol. 604, pp. 72–81. Springer, Cham (2016). Scholar
  26. 26.
    Moore, E., Clements, M.A., Peifer, J.W., Weisser, L.: Critical analysis of the impact of glottal features in the classification of clinical depression in speech. IEEE Trans. Bio-Med. Eng. 55(1), 96–107 (2008)CrossRefGoogle Scholar
  27. 27.
    Cohn, J.F., et al.: Detecting depression from facial actions and vocal prosody. In: Staff, I. (ed.) 2009 3rd International Conference on Affective Computing and Intelligent Interaction, pp. 1–7. IEEE (2009)Google Scholar
  28. 28.
    Alghowinem, S., Goecke, R., Cohn, J.F., Wagner, M., Parker, G., Breakspear, M.: Cross-cultural detection of depression from nonverbal behaviour. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops, vol. 1 (2015)Google Scholar
  29. 29.
    Seibt, B., Mühlberger, A., Likowski, K., Weyers, P.: Facial mimicry in its social setting. Front. Psychol. 6, 1122 (2015)CrossRefGoogle Scholar
  30. 30.
    McIntosh, D.N., Reichmann-Decker, A., Winkielman, P., Wilbarger, J.L.: When the social mirror breaks: deficits in automatic, but not voluntary, mimicry of emotional facial expressions in autism. Dev. Sci. 9(3), 295–302 (2006)CrossRefGoogle Scholar
  31. 31.
    Stagg, S.D., Slavny, R., Hand, C., Cardoso, A., Smith, P.: Does facial expressivity count? How typically developing children respond initially to children with autism. Autism 18(6), 704–711 (2014)CrossRefGoogle Scholar
  32. 32.
    Grossman, R.B., Edelson, L.R., Tager-Flusberg, H.: Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. J. Speech Lang. Hear. Res. 56(3), 1035–1044 (2013)CrossRefGoogle Scholar
  33. 33.
    Zhao, S., Uono, S., Yoshimura, S., Kubota, Y., Toichi, M.: Atypical gaze cueing pattern in a complex environment in individuals with ASD. J. Autism Dev. Disord. 47(7), 1978–1986 (2017)CrossRefGoogle Scholar
  34. 34.
    Wieckowski, A.T., White, S.W.: Eye-gaze analysis of facial emotion recognition and expression in adolescents with ASD. J. Clin. Child Adolesc. Psychol. 46(1), 110–124 (2017). The official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53CrossRefGoogle Scholar
  35. 35.
    Madipakkam, A.R., Rothkirch, M., Dziobek, I., Sterzer, P.: Unconscious avoidance of eye contact in autism spectrum disorder. Sci. Rep. 7(1), 13378 (2017)CrossRefGoogle Scholar
  36. 36.
    Shriberg, L.D., Paul, R., McSweeny, J.L., Klin, A., Cohen, D.J., Volkmar, F.R.: Speech and prosody characteristics of adolescents and adults with high-functioning autism and asperger syndrome. J. Speech Lang. Hear. Res. 44(5), 1097–1115 (2001)CrossRefGoogle Scholar
  37. 37.
    Sharda, M., et al.: Sounds of melodypitch patterns of speech in autism. Neurosci. Lett. 478(1), 42–45 (2010)CrossRefGoogle Scholar
  38. 38.
    Diehl, J.J., Watson, D., Bennetto, L., McDonough, J., Gunlogson, C.: An acoustic analysis of prosody in high-functioning autism. Appl. Psycholinguist. 30(3), 385–404 (2009)CrossRefGoogle Scholar
  39. 39.
    Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  40. 40.
    Fasel, B., Luettin, J.: Automatic facial expression analysis: a survey. Pattern Recognit. 36(1), 259–275 (2003)CrossRefGoogle Scholar
  41. 41.
    Tian, Y.L., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)CrossRefGoogle Scholar
  42. 42.
    Baltrusaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)Google Scholar
  43. 43.
    Baltrusaitis, T., Robinson, P., Morency, L.P.: Constrained local neural fields for robust facial landmark detection in the wild. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 354–361 (2013)Google Scholar
  44. 44.
    McKeown, G., Valstar, M.F., Cowie, R., Pantic, M.: The semaine corpus of emotionally coloured character interactions. In: 2010 IEEE International Conference on Multimedia and Expo, pp. 1079–1084. IEEE (2010)Google Scholar
  45. 45.
    Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: DISFA: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151–160 (2013)CrossRefGoogle Scholar
  46. 46.
    Zhang, X., et al.: BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 692–706 (2014)CrossRefGoogle Scholar
  47. 47.
    Baltrusaitis, T., Mahmoud, M., Robinson, P.: Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2015)Google Scholar
  48. 48.
    Valstar, M.F., et al.: FERA 2015 - second facial expression recognition and analysis challenge. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2015)Google Scholar
  49. 49.
    Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Mpiigaze: Real-world dataset and deep appearance-based gaze estimation. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 162–175 (2019)CrossRefGoogle Scholar
  50. 50.
    Wood, E., Bulling, A.: EyeTab: model-based gaze estimation on unmodified tablet computers. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 207–210. ACM (2014)Google Scholar
  51. 51.
    Wood, E., Baltrusaitis, T., Zhang, X., Sugano, Y., Robinson, P., Bulling, A.: Rendering of eyes for eye-shape registration and gaze estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3756–3764 (2015)Google Scholar
  52. 52.
    McFee, B., et al.: librosa: Audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, pp. 18–25 (2015)Google Scholar
  53. 53.
    Ittichaichareon, C., Suksri, S., Yingthawornsuk, T.: Speech recognition using MFCC. In: International Conference on Computer Graphics, Simulation and Modeling (ICGSM 2012), pp. 28–29, July 2012Google Scholar
  54. 54.
    Marchi, E., Schuller, B., Batliner, A., Fridenzon, S., Tal, S., Golan, O.: Emotion in the speech of children with autism spectrum conditions: prosody and everything else. In: Proceedings 3rd Workshop on Child, Computer and Interaction (WOCCI 2012), Satellite Event of INTERSPEECH 2012 (2012)Google Scholar
  55. 55.
    Hoekstra, R.A., Bartels, M., Cath, D.C., Boomsma, D.I.: Factor structure, reliability and criterion validity of the autism-spectrum quotient (AQ): a study in dutch population and patient groups. J. Autism Dev. Disord. 38(8), 1555–1566 (2008)CrossRefGoogle Scholar
  56. 56.
    Zhang, L., et al.: Psychometric properties of the autism-spectrum quotient in both clinical and non-clinical samples: Chinese version for mainland China. BMC Psychiatry 16(1), 213 (2016)CrossRefGoogle Scholar
  57. 57.
    Ashwood, K., et al.: Predicting the diagnosis of autism in adults using the autism-spectrum quotient (AQ) questionnaire. Psychol. Med. 46(12), 2595–2604 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hanna Drimalla
    • 1
    • 2
    • 3
    Email author
  • Niels Landwehr
    • 1
    • 5
  • Irina Baskow
    • 2
  • Behnoush Behnia
    • 4
  • Stefan Roepke
    • 4
  • Isabel Dziobek
    • 2
    • 3
  • Tobias Scheffer
    • 1
  1. 1.Department of Computer ScienceUniversity of PotsdamPotsdamGermany
  2. 2.Department of PsychologyHumboldt-Universität zu BerlinBerlinGermany
  3. 3.Berlin School of Mind and BrainHumboldt-Universität zu BerlinBerlinGermany
  4. 4.Department of Psychiatry and PsychotherapyCampus Benjamin Franklin, Charité-Universitätsmedizin BerlinBerlinGermany
  5. 5.Leibniz Institute for Agricultural Engineering and BioeconomyPotsdamGermany

Personalised recommendations