Vision-Based Classification of Developmental Disorders Using Eye-Movements

  • Guido PusiolEmail author
  • Andre Esteva
  • Scott S. Hall
  • Michael Frank
  • Arnold Milstein
  • Li Fei-Fei
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9901)


This paper proposes a system for fine-grained classification of developmental disorders via measurements of individuals’ eye-movements using multi-modal visual data. While the system is engineered to solve a psychiatric problem, we believe the underlying principles and general methodology will be of interest not only to psychiatrists but to researchers and engineers in medical machine vision. The idea is to build features from different visual sources that capture information not contained in either modality. Using an eye-tracker and a camera in a setup involving two individuals speaking, we build temporal attention features that describe the semantic location that one person is focused on relative to the other person’s face. In our clinical context, these temporal attention features describe a patient’s gaze on finely discretized regions of an interviewing clinician’s face, and are used to classify their particular developmental disorder.


Autism Spectrum Disorder Autism Spectrum Disorder Developmental Disorder Recurrent Neural Network Convolutional Neural Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Boraston, Z., Blakemore, S.J.: The application of eye-tracking technology in the study of autism. J. Physiol. 581(3), 893–898 (2007)CrossRefGoogle Scholar
  2. 2.
    Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches (2014). arXiv:1409.1259
  3. 3.
    Csibra, G., Gergely, G.: Social learning and social cognition: The case for pedagogy. In: Process of Change in Brain and Cognitive Development, Attention and Performance 9, pp. 152–158 (2006)Google Scholar
  4. 4.
    Golarai, G., Grill-Spector, K., Reiss, A.: Autism and the development of face processing. Clin. Neurosci. Res. 6, 145–160 (2006)CrossRefGoogle Scholar
  5. 5.
    Hagerman, P.J.: The fragile x prevalence paradox. J. Med. Genet. 45, 498–499 (2008)CrossRefGoogle Scholar
  6. 6.
    Hall, S.S., Frank, M.C., Pusiol, G.T., Farzin, F., Lightbody, A.A., Reiss, A.L.: Quantifying naturalistic social gaze in fragile x syndrome using a novel eye tracking paradigm. Am. J. Med. Genet. Part B Neuropsychiatric Genet. 168, 564–572 (2015)CrossRefGoogle Scholar
  7. 7.
    Hashemi, J., Spina, T.V., Tepper, M., Esler, A., Morellas, V., Papanikolopoulos, N., Sapiro, G.: A computer vision approach for the assessment of autism-related behavioral markers. In: 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1–7. IEEE (2012)Google Scholar
  8. 8.
    Jones, W., Klin, A.: Attention to eyes is present but in decline in 2-6-month-old infants later diagnosed with autism. Nature (2013)Google Scholar
  9. 9.
    Jzefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: Bach, F.R., Blei, D.M. (eds.) ICML, vol. 37, pp. 2342–2350 (2015)Google Scholar
  10. 10.
    Klin, A., Jones, W., Schultz, R., Volkmar, F., Cohen, D.: Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Arch. Gen. Psychiatry 59(9), 809–816 (2002)CrossRefGoogle Scholar
  11. 11.
    Kumar, Y., Dewal, M.L., Anand, R.S.: Epileptic seizure detection using DWT based fuzzy approximate entropy and support vector machine. Neurocomputing 133, 271–279 (2014)CrossRefGoogle Scholar
  12. 12.
    Rehg, J.M., Rozga, A., Abowd, G.D., Goodwin, M.S.: Behavioral imaging and autism. IEEE Pervasive Comput. 13(2), 84–87 (2014)CrossRefGoogle Scholar
  13. 13.
    Restrepo, J.F., Schlotthauer, G., Torres, M.E.: Maximum approximate entropy and r threshold: A new approach for regularity changes detection (2014). arXiv:1405.7637 (nlin.CD)
  14. 14.
    Sullivan, K., Hatton, D.D., Hammer, J., Sideris, J., Hooper, S., Ornstein, P.A., Bailey, D.B.: Sustained attention and response inhibition in boys with fragile X syndrome: measures of continuous performance. Am. J. Med. Gen. Part B Neuropsychiatric Gen. 144B(4), 517–532 (2007)CrossRefGoogle Scholar
  15. 15.
    Vinyals, O., Kaiser, L.u., Koo, T., Petrov, S., Sutskever, I., Hinton, G.: Grammar as a foreign language. In: Advances in Neural Information Processing Systems 28, pp. 2773–2781. Curran Associates, Inc. (2015).
  16. 16.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: CVPR, pp. 2879–2886. IEEE (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Guido Pusiol
    • 1
    Email author
  • Andre Esteva
    • 2
  • Scott S. Hall
    • 3
  • Michael Frank
    • 4
  • Arnold Milstein
    • 5
  • Li Fei-Fei
    • 1
  1. 1.Department of Computer ScienceStanford UniversityStanfordUSA
  2. 2.Department of Electrical EngineeringStanford UniversityStanfordUSA
  3. 3.Department of PsychiatryStanford UniversityStanfordUSA
  4. 4.Department of PsychologyStanford UniversityStanfordUSA
  5. 5.Department of MedicineStanford UniversityStanfordUSA

Personalised recommendations