Journal of Autism and Developmental Disorders

, Volume 45, Issue 5, pp 1121–1136 | Cite as

Applying Machine Learning to Facilitate Autism Diagnostics: Pitfalls and Promises

  • Daniel Bone
  • Matthew S. Goodwin
  • Matthew P. Black
  • Chi-Chun Lee
  • Kartik Audhkhasi
  • Shrikanth Narayanan
Original Paper

Abstract

Machine learning has immense potential to enhance diagnostic and intervention research in the behavioral sciences, and may be especially useful in investigations involving the highly prevalent and heterogeneous syndrome of autism spectrum disorder. However, use of machine learning in the absence of clinical domain expertise can be tenuous and lead to misinformed conclusions. To illustrate this concern, the current paper critically evaluates and attempts to reproduce results from two studies (Wall et al. in Transl Psychiatry 2(4):e100, 2012a; PloS One 7(8), 2012b) that claim to drastically reduce time to diagnose autism using machine learning. Our failure to generate comparable findings to those reported by Wall and colleagues using larger and more balanced data underscores several conceptual and methodological problems associated with these studies. We conclude with proposed best-practices when using machine learning in autism research, and highlight some especially promising areas for collaborative work at the intersection of computational and behavioral science.

Keywords

Autism diagnostic observation schedule Autism diagnostic interview Machine learning Signal processing Autism Diagnosis 

Notes

Acknowledgments

This work was supported by funds from NSF Award 1029035, “Computational Behavioral Science: Modeling, Analysis, and Visualization of Social and Communicative Behavior”, NIH grants P50 DC013027 and R01 DC012774, and the Alfred E. Mann Innovation in Engineering Fellowship. The authors are grateful to Shanping Qiu for her efforts in acquiring and preparing the BID data for analysis.

References

  1. Abrahams, B. S., & Geschwind, D. H. (2010). Connecting genes to brain in the autism spectrum disorders. Archives of Neurology, 67(4), 395.CrossRefPubMedCentralPubMedGoogle Scholar
  2. AGRE Pedigree Algorithms. (2013). AGRE Pedigree Algorithms. N.p., n.d. Web. http://www.research.agre.org/agrecatalog/algorithm.cfm.
  3. Amaral, D., Dawson, G., & Geschwind, D. (Eds.). (2011). Autism spectrum disorders. Oxford: Oxford University Press.Google Scholar
  4. American Psychiatric Association (Ed.). (2013). Diagnostic and statistical manual of mental disorders: DSM-5. American Psychiatric Association.Google Scholar
  5. Audhkhasi, K., & Narayanan, S. (2013). A globally-variant locally-constant model for fusion of labels from multiple diverse experts without using reference labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4), 769–783.CrossRefPubMedGoogle Scholar
  6. Baldi, P. (2001). Bioinformatics: The machine learning approach. Cambridge, MA: The MIT Press.Google Scholar
  7. Black, M. P., Katsamanis, A., Baucom, B. R., Lee, C. C., Lammert, A. C., Christensen, A., et al. (2013). Toward automating a human behavioral coding system for married couples’ interactions using speech acoustic features. Speech Communication, 55(1), 1–21.CrossRefGoogle Scholar
  8. Bone, D., Black, M. P., Lee, C. C., Williams, M. E., Levitt, P., Lee, S., & Narayanan, S. (2012). Spontaneous-speech acoustic–prosodic features of children with autism and the interacting psychologist. In INTERSPEECH (pp. 1043–1046).Google Scholar
  9. Bone, D., Black, M. P., Lee, C. C., Williams, M. E., Levitt, P., Lee, S., & Narayanan, S. (2014, in press). The Psychologist as an Interlocutor in Autism Spectrum Disorder Assessment: Insights from a Study of Spontaneous Prosody. Journal of Speech, Language, and Hearing Research.Google Scholar
  10. Bone, D., Lee, C. C., Chaspari, T., Black, M. P., Williams, M. E., Lee, S., Levitt, P. & Narayanan, S. (2013). Acoustic–prosodic, turn-taking, and language cues in child–psychologist interactions for varying social demand. In INTERSPEECH (pp. 2400–2404).Google Scholar
  11. Chaspari, T., Bone, D., Gibson, J., Lee, C. C., & Narayanan, S. (2013). Using physiology and language cues for modeling verbal response latencies of children with ASD. In 2013 IEEE International Conference on acoustics, speech and signal processing (ICASSP) (pp. 3702–3706).Google Scholar
  12. Constantino, J. N., LaVesser, P. D., Zhang, Y., Abbacchi, A. M., Gray, T., & Todd, R. D. (2007). Rapid quantitative assessment of autistic social impairment by classroom teachers. Journal of the American Academy of Child and Adolescent Psychiatry, 46(12), 1668–1676.CrossRefPubMedGoogle Scholar
  13. Dawson, G., Webb, S., Schellenberg, G. D., Dager, S., Friedman, S., Aylward, E., et al. (2002). Defining the broader phenotype of autism: Genetic, brain, and behavioral perspectives. Development and Psychopathology, 14(3), 581–611.CrossRefPubMedGoogle Scholar
  14. Duda, M., Kosmicki, J. A., & Wall, D. P. (2014). Testing the accuracy of an observation-based classifier for rapid detection of autism risk. Translational Psychiatry, 4(8), e424.CrossRefPubMedCentralPubMedGoogle Scholar
  15. Freund, Y., & Mason, L. (1999). The alternating decision tree learning algorithm. In ICML (Vol. 99, pp. 124–133).Google Scholar
  16. Geschwind, D. H., Sowinski, J., Lord, C., Iversen, P., Shestack, J., Jones, P., et al. (2001). The autism genetic resource exchange: A resource for the study of autism and related neuropsychiatric conditions. American Journal of Human Genetics, 69(2), 463.CrossRefPubMedCentralPubMedGoogle Scholar
  17. Gotham, K., Risi, S., Pickles, A., & Lord, C. (2007). The autism diagnostic observation schedule: Revised algorithms for improved diagnostic validity. Journal of Autism and Developmental Disorders, 37(4), 613–627.CrossRefPubMedGoogle Scholar
  18. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1), 10–18.CrossRefGoogle Scholar
  19. Hu, V. W., & Steinberg, M. E. (2009). Novel clustering of items from the Autism Diagnostic Interview-Revised to define phenotypes within autism spectrum disorders. Autism Research, 2(2), 67–77.CrossRefPubMedCentralPubMedGoogle Scholar
  20. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In IJCAI (Vol. 14, No. 2, pp. 1137–1145).Google Scholar
  21. Lai, M. C., Lombardo, M. V., Chakrabarti, B., & Baron-Cohen, S. (2013). Subgrouping the autism “spectrum”: Reflections on DSM-5. PLoS Biology, 11(4).Google Scholar
  22. Lee, H., Marvin, A. R., Watson, T., Piggot, J., Law, J. K., Law, P. A., et al. (2010). Accuracy of phenotyping of autistic children based on internet implemented parent report. American Journal of Medical Genetics Part B: Neuropsychiatric Genetics, 153(6), 1119–1126.Google Scholar
  23. Levitt, P., & Campbell, D. B. (2009). The genetic and neurobiologic compass points toward common signaling dysfunctions in autism spectrum disorders. The Journal of Clinical Investigation, 119(4), 747.CrossRefPubMedCentralPubMedGoogle Scholar
  24. Lord, C., & Jones, R. M. (2012). Annual Research Review: Re-thinking the classification of autism spectrum disorders. Journal of Child Psychology and Psychiatry, 53(5), 490–509.CrossRefPubMedCentralPubMedGoogle Scholar
  25. Lord, C., Risi, S., Lambrecht, L., Cook, E. H, Jr, Leventhal, B. L., DiLavore, P. C., et al. (2000). The Autism Diagnostic Observation Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.CrossRefPubMedGoogle Scholar
  26. Lord, C., Rutter, M., & Le Couteur, A. (1994). Autism diagnostic interview-revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24(5), 659–685.CrossRefPubMedGoogle Scholar
  27. Narayanan, S., & Georgiou, P. G. (2013). Behavioral signal processing: Deriving human behavioral informatics from speech and language. Proceedings of the IEEE, 101(5), 1203–1233.CrossRefPubMedCentralPubMedGoogle Scholar
  28. Picard, R. W. (2000). Affective computing. Cambridge, MA: MIT press.Google Scholar
  29. Rehg, J. M., Abowd, G. D., Rozga, A., Romero, M., Clements, M. A., Sclaroff, S., & Ye, Z. (2013). Decoding children’s social behavior. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 3414–3421). IEEE.Google Scholar
  30. Rehg, J. M., Rozga, A., Abowd, G. D., & Goodwin, M. S. (2014). Behavioral imaging and autism. Pervasive Computing, IEEE, 13(2), 84–87.CrossRefGoogle Scholar
  31. Rosenberg, A. (2012). Classifying skewed data: Importance weighting to optimize average recall. In INTERSPEECH (pp. 2242–2245).Google Scholar
  32. Schuller, B., Steidl, S., & Batliner, A. (2009, September). The INTERSPEECH 2009 emotion challenge. In INTERSPEECH (pp. 312–315).Google Scholar
  33. Schuller, B., Steidl, S., Batliner, A., Schiel, F., & Krajewski, J. (2011, August). The INTERSPEECH 2011 Speaker State Challenge. In INTERSPEECH (pp. 3201–3204).Google Scholar
  34. Wall, D. P., Dally, R., Luyster, R., Jung, J. Y., & DeLuca, T. F. (2012b). Use of artificial intelligence to shorten the behavioral diagnosis of autism. PloS One, 7(8).Google Scholar
  35. Wall, D. P., Kosmicki, J. A., DeLuca, T., Harstad, E. B., & Fusaro, V. A. (2012a). Use of machine learning to shorten observation-based screening and diagnosis of autism. Translational Psychiatry, 2(4), e100.CrossRefPubMedCentralPubMedGoogle Scholar
  36. Wei, L., Yang, Y., Nishikawa, R. M., & Jiang, Y. (2005). A study on several machine learning methods for classification of malignant and benign clustered microcalcifications. IEEE Transactions on Medical Imaging, 24(3), 371–380.CrossRefPubMedGoogle Scholar
  37. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G. D., & Rehg, J. M. (2012). Detecting eye contact using wearable eye-tracking glasses. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (pp. 699–704). ACM.Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Daniel Bone
    • 1
  • Matthew S. Goodwin
    • 2
  • Matthew P. Black
    • 3
  • Chi-Chun Lee
    • 4
  • Kartik Audhkhasi
    • 1
  • Shrikanth Narayanan
    • 1
  1. 1.Signal Analysis & Interpretation Laboratory (SAIL)University of Southern CaliforniaLos AngelesUSA
  2. 2.Department of Health SciencesNortheastern UniversityBostonUSA
  3. 3.Information Sciences InstituteUniversity of Southern CaliforniaMarina del ReyUSA
  4. 4.Department of Electrical EngineeringNational Tsing Hua UniversityHsinchuTaiwan

Personalised recommendations