Sports Medicine

, Volume 48, Issue 7, pp 1739–1749 | Cite as

Quantifying the Value of Multidimensional Assessment Models for Acute Concussion: An Analysis of Data from the NCAA-DoD Care Consortium

  • Gian-Gabriel P. GarciaEmail author
  • Steven P. Broglio
  • Mariel S. Lavieri
  • Michael McCrea
  • Thomas McAllister
  • CARE Consortium Investigators
Original Research Article
Part of the following topical collections:
  1. The NCAA-DoD Concussion Assessment, Research and Education (CARE) Consortium



Many concussion assessment methods exist, but few studies quantify the performance of these methods to determine which can best assess acute concussion alone or in combination.


The objectives of this study were to evaluate: (1) selected concussion assessments for acute concussion assessment; (2) the utility of change scores for acute concussion assessment; and (3) concussion assessment capabilities when constrained to limited clinical data or objective clinical measures.


The ‘acute concussion’ group contained assessments from < 6 h post-injury (n = 560) and 24–48 h post-injury (n = 733). The ‘normal performance’ group contained assessments from baseline testing (n = 842) and unrestricted return to play (n = 707) timepoints. Univariate and multivariate logistic regression models were created separately for < 6- and 24- to 48-h timepoints. Models were evaluated on sensitivity, specificity, and area under the receiver operating characteristic curve.


Within the univariate analysis, Sport Concussion Assessment Tool symptom assessments had the highest combination of sensitivity, specificity, and area under the receiver operating characteristic curve, with values up to 0.93, 0.97, and 0.98, respectively. Full models had a sensitivity, specificity, and area under the receiver operating characteristic curve up to 0.94, 0.97, and 0.99, respectively, and outperformed all univariate models, raw score models, and objective models. Objective models were outperformed by all multivariate models and the univariate models containing only Sport Concussion Assessment Tool symptom assessments.


Results support the use of multidimensional assessment batteries over single instruments and suggest the importance of self-reported symptoms in acute concussion assessment. Balance assessments, however, may not provide additional benefit when symptom and neurocognitive assessments are available. Additionally, change scores provide some clinical utility over raw scores, but the difference may not be clinically meaningful.



This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE 1256260. This publication was made possible, in part, with support from the Grand Alliance Concussion Assessment, Research, and Education (CARE) Consortium, funded, in part, by the National Collegiate Athletic Association and the Department of Defense. The US Army Medical Research Acquisition Activity, Fort Detrick, MD, USA is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under Award No. W81XWH-14-2-0151. Opinions, interpretations, conclusions, and recommendations are those of the author(s) and are not necessarily endorsed by the Department of Defense (Defense Health Program funds).

The CARE Consortium Investigators are listed alphabetically by institution: April Marie (Reed) Hoy, MS, ATC (Azusa Pacific University); Louise A. Kelly, PhD (California Lutheran University); Justus D. Ortega, PhD (Humboldt State University); Nicholas Port, PhD (Indiana University); Margot Putukian MD (Princeton University); T. Dianne Langford, PhD (Temple University); Scott Anderson, ATC and Gerald McGinty, DPT (US Air Force Academy); Patrick O’Donnell, MHA (US Coast Guard Academy); Steven J. Svoboda, MD (US Military Academy); John P. DiFiori (University of California-Los Angeles); Holly J. Benjamin MD (University of Chicago); Thomas Buckley, EdD, ATC, and Thomas W. Kaminski, PhD, ATC (University of Delaware); James R. Clugston, MD, MS (University of Florida); Julianne D. Schmidt, PhD, ATC (University of Georgia); Jason P. Mihalik, PhD, CAT(C), ATC (University of North Carolina at Chapel Hill); Christina L. Master, MD (University of Pennsylvania); Micky Collins, PhD, and Anthony P. Kontos, PhD (University of Pittsburgh Medical Center); Sara P.D. Chrisman, MD, MPH (University of Washington); Christopher M. Miles, MD (Wake Forest University); Brian H. Dykhuizen, MS, ATC (Wilmington College); Alison Brooks MD, MPH (University of Wisconsin).

Compliance with Ethical Standards

Conflict of interest

Gian-Gabriel Garcia, Steven Broglio, Mariel Lavieri, Thomas McAllister, and Michael McCrea have no conflicts of interest directly relevant to the content of this study.

Ethics approval

This study was approved by each local institutional review board and the Army Human Research Protection Office.

Supplementary material

40279_2018_880_MOESM1_ESM.docx (20 kb)
Supplementary material 1 (DOCX 19 kb)
40279_2018_880_MOESM2_ESM.docx (47 kb)
Supplementary material 2 (DOCX 46 kb)
40279_2018_880_MOESM3_ESM.docx (16 kb)
Supplementary material 3 (DOCX 16 kb)


  1. 1.
    McCrory P, Meeuwisse W, Dvorak J, Aubry M, Bailes J, Broglio S, et al. Consensus statement on concussion in sport: the 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017;51:838–47.PubMedGoogle Scholar
  2. 2.
    Zuckerman SL, Kerr ZY, Yengo-Kahn A, Wasserman E, Covassin T, Solomon GS. Epidemiology of sports-related concussion in NCAA athletes from 2009-2010 to 2013-2014: incidence, recurrence, and mechanisms. Am J Sports Med. 2015;43:2654–62.CrossRefPubMedGoogle Scholar
  3. 3.
    Giza CC, Kutcher JS, Ashwal S, Barth J, Getchius TSD, Gioia GA, et al. Summary of evidence-based guideline update: evaluation and management of concussion in sports: report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology. 2013;80:2250–7.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Broglio SP, Cantu RC, Gioia GA, Guskiewicz KM, Kutcher J, Palm M, et al. National Athletic Trainers’ Association position statement: management of sport concussion. J Athl Train. 2014;49:245–65.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Harmon KG, Drezner JA, Gammons M, Guskiewicz KM, Halstead M, Herring SA, et al. American Medical Society for Sports Medicine position statement: concussion in sport. Br J Sports Med. 2013;47:15–26.CrossRefPubMedGoogle Scholar
  6. 6.
    McCrory P, Meeuwisse WH, Kutcher JS, Jordan BD, Gardner A. What is the evidence for chronic concussion-related changes in retired athletes: behavioural, pathological and clinical outcomes? Br J Sports Med. 2013;47:327–30.CrossRefPubMedGoogle Scholar
  7. 7.
    Randolph C, Karantzoulis S, Guskiewicz K. Prevalence and characterization of mild cognitive impairment in retired national football league players. J Int Neuropsychol Soc. 2013;19:873–80.CrossRefPubMedGoogle Scholar
  8. 8.
    Guskiewicz KM, Marshall SW, Bailes J, McCrea M, Cantu RC, Randolph C, et al. Association between recurrent concussion and late-life cognitive impairment in retired professional football players. Neurosurgery. 2005;57:719–26.CrossRefPubMedGoogle Scholar
  9. 9.
    Daneshvar DH, Riley DO, Nowinski CJ, McKee AC, Stern RA, Cantu RC. Long-term consequences: effects on normal development profile after concussion. Phys Med Rehabil Clin N Am. 2011;22:683–700.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Kerr ZY, Marshall SW, Harding HP, Guskiewicz KM. Nine-year risk of depression diagnosis increases with increasing self-reported concussions in retired professional football players. Am J Sports Med. 2012;40:2206–12.CrossRefPubMedGoogle Scholar
  11. 11.
    Guskiewicz KM, Marshall SW, Bailes J, Mccrea M, Harding HP, Matthews A, et al. Recurrent concussion and risk of depression in retired professional football players. Med Sci Sports Exerc. 2007;39:903–9.CrossRefPubMedGoogle Scholar
  12. 12.
    De Beaumont L, Lassonde M, Leclerc S, Théoret H. Long-term and cumulative effects of sports concussion on motor cortex inhibition. Neurosurgery. 2007;61:329–36.CrossRefPubMedGoogle Scholar
  13. 13.
    Barr WB, McCrea M. Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. J Int Neuropsychol Soc. 2001;7:693–702.CrossRefPubMedGoogle Scholar
  14. 14.
    Broglio SP, Macciocchi SN, Ferrara MS. Sensitivity of the concussion assessment battery. Neurosurgery. 2007;60:1050–7.CrossRefPubMedGoogle Scholar
  15. 15.
    Broglio SP, Ferrara MS, Sopiarz K, Kelly MS. Reliable change of the sensory organization test. Clin J Sport Med. 2008;18:148–54.CrossRefPubMedGoogle Scholar
  16. 16.
    Lovell MR, Iverson GL, Collins MW, Podell K, Johnston KM, Pardini D, et al. Measurement of symptoms following sports-related concussion: reliability and normative data for the post-concussion scale. Appl Neuropsychol. 2006;13:166–74.CrossRefPubMedGoogle Scholar
  17. 17.
    McCrea M, Barr WB, Guskiewicz K, Randolph C, Marshall SW, Cantu R, et al. Standard regression-based methods for measuring recovery after sport-related concussion. J Int Neuropsychol Soc. 2005;11:58–69.CrossRefPubMedGoogle Scholar
  18. 18.
    Echemendia RJ, Meeuwisse W, McCrory P, Davis GA, Putukian M, Leddy J, et al. The sport concussion assessment tool 5th Edition (SCAT5). Br J Sports Med. 2017;5:1–3.Google Scholar
  19. 19.
    Randolph C. Baseline neuropsychological testing in managing sport-related concussion. Curr Sports Med Rep. 2011;10:21–6.CrossRefPubMedGoogle Scholar
  20. 20.
    Schmidt JD, Register-Mihalik JK, Mihalik JP, Kerr ZY, Guskiewicz KM. Identifying impairments after concussion: normative data versus individualized baselines. Med Sci Sports Exerc. 2012;44:1621–8.CrossRefPubMedGoogle Scholar
  21. 21.
    Echemendia RJ, Bruce JM, Bailey CM, Sanders JF, Arnett P, Vargas G. The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data. Clin Neuropsychol. 2012;26:1077–91.CrossRefPubMedGoogle Scholar
  22. 22.
    Valovich McLeod TC, Bay RC, Lam KC, Chhabra A. Representative baseline values on the sport concussion assessment tool 2 (SCAT2) in adolescent athletes vary by gender, grade, and concussion history. Am J Sports Med. 2012;40:927–33.CrossRefPubMedGoogle Scholar
  23. 23.
    Putukian M, Echemendia R, Dettwiler-Danspeckgruber A, Duliba T, Bruce J, Furtado JL, et al. Prospective clinical assessment using sideline concussion assessment tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin J Sport Med. 2015;25:36–42.CrossRefPubMedGoogle Scholar
  24. 24.
    Covassin T, Elbin RJ, Harris W, Parker T, Kontos A. The role of age and sex in symptoms, neurocognitive performance, and postural stability in athletes after concussion. Am J Sports Med. 2012;40:1303–12.CrossRefPubMedGoogle Scholar
  25. 25.
    Covassin T, Buz Swanik C, Sachs ML. Sex differences and the incidence of concussions among collegiate athletes. J Athl Train. 2003;38:238–44.PubMedPubMedCentralGoogle Scholar
  26. 26.
    Covassin T, Schatz P, Swanik CB. Sex differences in neuropsychological function and post-concussion symptoms of concussed collegiate athletes. Neurosurgery. 2007;61:345–50.CrossRefPubMedGoogle Scholar
  27. 27.
    Shehata N, Wiley JP, Richea S, Benson BW, Duits L, Meeuwisse WH. Sport concussion assessment tool: baseline values for varsity collision sport athletes. Br J Sports Med. 2009;43:730–4.CrossRefPubMedGoogle Scholar
  28. 28.
    Bruce JM, Echemendia RJ. Concussion history predicts self-reported symptoms before and following a concussive event. Neurology. 2004;63:1516–8.CrossRefPubMedGoogle Scholar
  29. 29.
    Covassin T, Stearne D, Elbin R. Concussion history and postconcussion neurocognitive performance and symptoms in collegiate athletes. J Athl Train. 2008;43:119–24.CrossRefPubMedPubMedCentralGoogle Scholar
  30. 30.
    Guskiewicz KM, Mccrea M, Marshall SW, Cantu RC, Randolph C, Barr W, et al. Cumulative effects associated with recurrent concussion in football players: the NCAA Concussion Study. J Am Med Assoc. 2003;290:2549–55.CrossRefGoogle Scholar
  31. 31.
    Broglio SP, McCrea M, McAllister T, Harezlak J, Katz B, Hack D, et al. A national study on the effects of concussion in collegiate athletes and US Military Service Academy members: the NCAA–DoD concussion assessment, research and education (CARE) consortium structure and methods. Sport Med. 2017;47:1437–51.CrossRefGoogle Scholar
  32. 32.
    Carney N, Ghajar J, Jagoda A, Bedrick S, Davis-O’Reilly C, Du Coudray H, et al. Concussion guidelines step 1: systematic review of prevalent indicators. Neurosurgery. 2014;75:3–15.CrossRefGoogle Scholar
  33. 33.
    McCrory P, Meeuwisse WH, Aubry M, Cantu RC, Dvorák J, Echemendia RJ, et al. Consensus statement on concussion in sport. In: the 4th international conference on concussion in sport held in Zurich, November 2012. Br J Sports Med. 2013;47:255–79.Google Scholar
  34. 34.
    Asken BM, McCrea MA, Clugston JR, Snyder AR, Houck ZM, Bauer RM. “Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery. J Athl Train. 2016;51:329–35.CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Elbin RJ, Sufrinko AM, Schatz P, French J, Henry L, Burkhart S, et al. Removal from play after concussion and recovery time. Pediatrics. 2016;138:e20160910.CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    McCrea M, Kelly JP, Randolph C, Kluge J, Bartolic E, Finn G, et al. Standardized assessment of concussion (SAC): on-site mental status evaluation of the athlete. J Head Trauma Rehabil. 1998;13:27–35.CrossRefPubMedGoogle Scholar
  37. 37.
    Concussion in Sport Group. Sport concussion assessment tool: 3rd edition. Br J Sports Med. 2013;47(5):259.Google Scholar
  38. 38.
    Bell DR, Guskiewicz KM, Clark MA, Padua DA. Systematic review of the balance error scoring system. Sports Health. 2011;3:287–95.CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Van Buuren S, Boshuizen H, Knook D. Multiple imputation of missing blood pressure covariates in survival analysis. Stat Med. 1999;18:681–94.CrossRefPubMedGoogle Scholar
  40. 40.
    McCrea M, Guskiewicz KM, Marshall SW, Barr W, Randolph C, Cantu RC, et al. Acute effects and recovery time following concussions in collegiate football players. JAMA. 2004;290:2556–63.CrossRefGoogle Scholar
  41. 41.
    McCrea M, Guskiewicz K, Randolph C, Barr WB, Hammeke TA, Marshall SW, et al. Incidence, clinical course, and predictors of prolonged recovery time following sport-related concussion in high school and college athletes. J Int Neuropsychol Soc. 2013;19:22–33.CrossRefPubMedGoogle Scholar
  42. 42.
    Bozdogan H. Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika. 1987;52:345–70.CrossRefGoogle Scholar
  43. 43.
    Williamson IJS, Goodman D. Converging evidence for the under-reporting of concussions in youth ice hockey. Br J Sports Med. 2006;40:128–32.CrossRefPubMedPubMedCentralGoogle Scholar
  44. 44.
    Pepe MS, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata J. 2009;9:1–16.PubMedPubMedCentralGoogle Scholar
  45. 45.
    Meehan WP, Mannix RC, Stracciolini A, Elbin RJ, Collins MW. Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163:721–5.CrossRefPubMedPubMedCentralGoogle Scholar
  46. 46.
    Sufrinko A, McAllister-Deitrick J, Womble M, Kontos A. Do sideline concussion assessments predict subsequent neurocognitive impairment after sport-related concussion? J Athl Train. 2017;52:676–81.CrossRefPubMedPubMedCentralGoogle Scholar
  47. 47.
    Sufrinko AM, Marchetti GF, Cohen PE, Elbin RJ, Re V, Kontos AP. Using acute performance on a comprehensive neurocognitive, vestibular, and ocular motor assessment battery to predict recovery duration after sport-related concussions. Am J Sports Med. 2017;45:1187–94.CrossRefPubMedGoogle Scholar
  48. 48.
    Register-Mihalik JK, Guskiewicz KM, Mihalik JP, Schmidt JD, Kerr ZY, McCrea MA. Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery. J Head Trauma Rehabil. 2013;28:274–83.CrossRefPubMedGoogle Scholar
  49. 49.
    Resch JE, Brown CN, Schmidt J, Macciocchi SN, Blueitt D, Cullum CM, et al. The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. BMJ Open Sport Exerc Med. 2016;2:e000012.CrossRefPubMedPubMedCentralGoogle Scholar
  50. 50.
    Chin EY, Nelson LD, Barr WB, McCrory P, McCrea MA. Reliability and validity of the sport concussion assessment tool-3 (SCAT3) in high school and collegiate athletes. Am J Sports Med. 2016;44:2276–85.CrossRefPubMedGoogle Scholar
  51. 51.
    Gessel LM, Fields SK, Collins CL, Dick RW, Comstock RD. Concussions among United States high school and collegiate athletes. J Athl Train. 2007;42:495–503.PubMedPubMedCentralGoogle Scholar
  52. 52.
    Lincoln AE, Caswell SV, Almquist JL, Dunn RE, Norris JB, Hinton RY. Trends in concussion incidence in high school sports. Am J Sports Med. 2011;39:958–63.CrossRefPubMedGoogle Scholar
  53. 53.
    Dick RW. Is there a gender difference in concussion incidence and outcomes? Br J Sports Med. 2009;43:i46–50.CrossRefPubMedGoogle Scholar
  54. 54.
    Broshek DK, Kaushik T, Freeman JR, Erlanger D, Webbe F, Barth JT. Sex differences in outcome following sports-related concussion. J Neurosurg. 2005;102:856–63.CrossRefPubMedGoogle Scholar
  55. 55.
    Valovich McLeod TC, Perrin DH, Guskiewicz KM, Shultz SJ, Diamond R, Gansneder BM. Serial administration of clinical concussion assessments and learning effects in healthy young athletes. Clin J Sport Med. 2004;14:287–95.CrossRefPubMedGoogle Scholar
  56. 56.
    Moreau MS, Langdon J, Buckley TA. The lived experience of an in-season concussion amongst NCAA Division I student-athletes. Int J Exerc Sci. 2014;7:62–74.Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Gian-Gabriel P. Garcia
    • 1
    Email author
  • Steven P. Broglio
    • 2
  • Mariel S. Lavieri
    • 1
  • Michael McCrea
    • 3
  • Thomas McAllister
    • 4
  • CARE Consortium Investigators
  1. 1.Department of Industrial and Operations EngineeringUniversity of MichiganAnn ArborUSA
  2. 2.School of KinesiologyUniversity of MichiganAnn ArborUSA
  3. 3.Departments of Neurosurgery and NeurologyMedical College of WisconsinMilwaukeeUSA
  4. 4.Department of PsychiatryIndiana University School of MedicineIndianapolisUSA

Personalised recommendations