A Clinician’s Guide to Machine Learning in Neuropsychological Research and Practice

Abstract

Machine learning (ML) techniques can help harness insights from data that complement and extend those that can be attained by traditional statistical methods. The current article introduces clinicians to concepts underlying ML and explores how it can be applied within the domain of neuropsychology. Specifically, we illustrate an application of ML to a dataset that includes a battery of standardized measures designed to provide diagnostic support for concussions, including standardized neurocognitive (CPT 3) and neurobehavioral (BESS, NIH 4 meter gait) measures, gait sensor data, and a CDC concussion symptom checklist. These variables were used to predict the decision-making of a pediatric neurologist evaluating a group of child/adolescent patients. With a sample of 111 cases, ML (using a general linear model and deep learning as illustrations) achieved accuracies of 91% and 84.8% and AUCs of 1.0 and .947, respectively, when predicting the neurologist’s binomial decision-making (safe/remove). In presenting the data and various considerations for interpretation, we attempt to balance both the promise and perils of ML.

This is a preview of subscription content, log in to check access.

Notes

  1. 1.

    In the context of artificial neural networks, a rectifier is an activation function defined as the positive part of its argument:

    f(x) = x+ = max(0, x),

    where x is the input to a neuron. Wikipedia

  2. 2.

    Softmax is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. Wikipedia

References

  1. Battista, P., Salvatore, C., & Castiglioni, I. (2017, 2017). Optimizing neuropsychological assessments for cognitive, behavioral, and functional impairment classification: a machine learning study. Behavioural Neurology, 1–19. https://doi.org/10.1155/2017/1850909.

    Article  Google Scholar 

  2. Bleiberg, J., Cernich, A. N., Cameron, K., et al. (2004). Duration of cognitive impairment after sports concussion. Neurosurgery, 54, 1073–1080. https://doi.org/10.1227/01.NEU.0000118820.33396.6A.

    Article  PubMed  Google Scholar 

  3. Broglio, S.P., Ferrara, M.S., Piland, S.G., Anderson, R.B. (2006). Concussion history is not a predictor of computerised neurocognitive performance. British Journal of Sports Medicine, 40(9), 802-805.

    Article  Google Scholar 

  4. Broglio, S. P., & Puetz, T. W. (2008). The effect of sport concussion on neurocognitive function, self-report symptoms, and postural control: a meta-analysis. Sports Medicine, 38, 53–67.

    Article  Google Scholar 

  5. Broglio, S. P., Ferrara, M. S., Macciocchi, S. N., Baumgartner, T. A., & Elliott, R. (2007). Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42(4), 509–514.

    PubMed  PubMed Central  Google Scholar 

  6. Bruce, J., Echemendia, R., Meeuwisse, W., Comper, P., & Sisco, A. (2014). 1-year test–retest reliability of ImPACT in professional ice hockey players. The Clinical Neuropsychologist, 28(1), 14–25.

    Article  Google Scholar 

  7. Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668–1674.

    Article  Google Scholar 

  8. Frank, G. (1984). The Boulder Model: History, rationale, and critique. Professional Psychology: Research and Practice, 15(3), 417–435. https://doi.org/10.1037/0735-7028.15.3.417.

    Article  Google Scholar 

  9. Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: a meta-analysis. Psychological Assessment, 12, 19–30. https://doi.org/10.1037//1040-3590.12.1.19.

    Article  PubMed  Google Scholar 

  10. Henry, L. C., Elbin, R. J., Collins, M. W., et al. (2016). Examining recovery trajectories after sport-related concussion with a multimodal clinical assessment approach. Neurosurgery, 78, 232–241. https://doi.org/10.1227/NEU.0000000000001041.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Hinton, G. (2018). Deep Learning—a technology with the potential to transform health care. Journal of the American Medical Association, 320(11), 1101–1102. https://doi.org/10.1001/jama.2018.11100.

    Article  PubMed  Google Scholar 

  12. Iverson, G. L., Gardner, A. J., Terry, D. P., et al. (2017). Predictors of clinical recovery from concussion: a systematic review. British Journal of Sports Medicine, 51, 941–948.

    Article  Google Scholar 

  13. Kaye, A. J., Gallagher, R., Callahan, J. M., & Nance, M. L. (2010). Mild traumatic brain injury in the pediatric population: the role of the pediatrician in routine follow-up. Journal of Trauma, 68, 1396–1400.

    Article  Google Scholar 

  14. Kirelik, S. B., & McAvoy, K. (2016). Acute concussion management with remove-reduce/educate/adjust-accommodate/pace (REAP). The Journal of emergency medicine, 50(2), 320–324.

    Article  Google Scholar 

  15. Lau, B., Lovell, M. R., Collins, M. W., & Pardini, J. (2009). Neurocognitive and symptom predictors of recovery in high school athletes. Clinical Journal of Sport Medicine, 19(3), 216–221.

    Article  Google Scholar 

  16. Lecci, L., Wiiliams, M., Taravath, S., Frank, H.G., Dugan, K., Page, G.R, & Keith, J.R. (in press). Validation of a concussion screening battery for use in medical settings. Archives of Clinical Neuropsychology.

  17. LeCun, Y., Bengio, Y., & Hinton, G. H. (2015). Deep learning. Nature, 521, 436–444. https://doi.org/10.1038/nature14539.

    Article  PubMed  Google Scholar 

  18. Macciocchi, S. N., Barth, J. T., Alves, W., et al. (1996). Neuropsychological functioning and recovery after mild head injury in collegiate Athletes. Neurosurgery, 39, 510–514.

    Article  Google Scholar 

  19. McCrea, M., Guskiewicz, K. M., Marshall, S. W., et al. (2003). Acute effects and recovery time following concussion in collegiate football players: The NCAA Concussion Study. JAMA, 290, 2556–2563. https://doi.org/10.1001/jama.290.19.2556.

    Article  PubMed  Google Scholar 

  20. McCrory, P., Meeuwisse, W. H., Aubry, M., Cantu, R. C., Dvorak, J., Echemendia, R. J., et al. (2013). Consensus statement on concussion in sport—the 4th International Conference on Concussion in Sport held in Zurich, November 2012. PM&R, 5(4), 255–279.

    Article  Google Scholar 

  21. McCrory, P., Meeuwisse, W., Dvorak, J., Aubry, M., Bailes, J., Broglio, S., et al. (2017). Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016. British Journal of Sports Medicine, 51(11), 838–847.

    PubMed  Google Scholar 

  22. Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence (Vol. x 149 pp.). Minneapolis: University of Minnesota Press. https://doi.org/10.1037/11281-000.

    Google Scholar 

  23. Parker, T. M., Osternig, L. R., van Donkelaar, P., & Chou, L. S. (2006). Gait stability after a concussion. Medicine and Science in Sports & Exercise, 38, 1031–1040.

    Article  Google Scholar 

  24. Resch, J. E., Macciocchi, S., & Ferrara, M. S. (2013). Preliminary evidence of equivalence of alternate forms of the ImPACT. The Clinical Neuropsychologist, 27(8), 1265–1280.

    Article  Google Scholar 

  25. Rosenblatt, F. (1957). The perceptron – a perceiving and recognizing automation. Report 85-460-1, Cornell Aeronautical Laboratory.

  26. Sahl, S. M. (2015). Estimating R 2 shrinkage in regression. International Journal of Technical Research and Applications, 3, 01–06.

    Google Scholar 

  27. Schatz, P., & Putz, B. O. (2006). Cross-validation of measures used for computer-based assessment of concussion. Applied Neuropsychology, 13(3), 151–159.

    Article  Google Scholar 

  28. Shatte, A., Hutchinson, D., & Teague, S. (2019). Machine learning in mental health: a scoping review of methods and applications. Psychological Medicine, 49(9), 1426–1448. https://doi.org/10.1017/S0033291719000151.

    Article  PubMed  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Len Lecci.

Ethics declarations

Conflict of Interest

Julian Keith is a paid consultant for SportGait, which is the concussion battery utilized in the current research. Mark Williams is one of the founders and stockholder in SportGait. Len Lecci is a paid consultant and stockholder in SportGait. Sasiharan Taravath has no conflicts to report and he provided the clinical data herein reported.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Keith, J., Williams, M., Taravath, S. et al. A Clinician’s Guide to Machine Learning in Neuropsychological Research and Practice. J Pediatr Neuropsychol 5, 177–187 (2019). https://doi.org/10.1007/s40817-019-00075-1

Download citation

Keywords

  • Machine learning
  • concussion screening
  • assessment
  • decision support tool