Skip to main content

Advertisement

SpringerLink
Log in

Validation of a deep learning model for traumatic brain injury detection and NIRIS grading on non-contrast CT: a multi-reader study with promising results and opportunities for improvement

  • Diagnostic Neuroradiology
  • Published:
Neuroradiology Aims and scope Submit manuscript

Abstract

Purpose

This study aimed to assess and externally validate the performance of a deep learning (DL) model for the interpretation of non-contrast computed tomography (NCCT) scans of patients with suspicion of traumatic brain injury (TBI).

Methods

This retrospective and multi-reader study included patients with TBI suspicion who were transported to the emergency department and underwent NCCT scans. Eight reviewers, with varying levels of training and experience (two neuroradiology attendings, two neuroradiology fellows, two neuroradiology residents, one neurosurgery attending, and one neurosurgery resident), independently evaluated NCCT head scans. The same scans were evaluated using the version 5.0 of the DL model icobrain tbi. The establishment of the ground truth involved a thorough assessment of all accessible clinical and laboratory data, as well as follow-up imaging studies, including NCCT and magnetic resonance imaging, as a consensus amongst the study reviewers. The outcomes of interest included neuroimaging radiological interpretation system (NIRIS) scores, the presence of midline shift, mass effect, hemorrhagic lesions, hydrocephalus, and severe hydrocephalus, as well as measurements of midline shift and volumes of hemorrhagic lesions. Comparisons using weighted Cohen’s kappa coefficient were made. The McNemar test was used to compare the diagnostic performance. Bland–Altman plots were used to compare measurements.

Results

One hundred patients were included, with the DL model successfully categorizing 77 scans. The median age for the total group was 48, with the omitted group having a median age of 44.5 and the included group having a median age of 48. The DL model demonstrated moderate agreement with the ground truth, trainees, and attendings. With the DL model’s assistance, trainees’ agreement with the ground truth improved. The DL model showed high specificity (0.88) and positive predictive value (0.96) in classifying NIRIS scores as 0–2 or 3–4. Trainees and attendings had the highest accuracy (0.95). The DL model’s performance in classifying various TBI CT imaging common data elements was comparable to that of trainees and attendings. The average difference for the DL model in quantifying the volume of hemorrhagic lesions was 6.0 mL with a wide 95% confidence interval (CI) of − 68.32 to 80.22, and for midline shift, the average difference was 1.4 mm with a 95% CI of − 3.4 to 6.2.

Conclusion

While the DL model outperformed trainees in some aspects, attendings’ assessments remained superior in most instances. Using the DL model as an assistive tool benefited trainees, improving their NIRIS score agreement with the ground truth. Although the DL model showed high potential in classifying some TBI CT imaging common data elements, further refinement and optimization are necessary to enhance its clinical utility.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. GBD (2016) Traumatic Brain Injury and Spinal Cord Injury Collaborators (2019) Global, regional, and national burden of traumatic brain injury and spinal cord injury, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol 18:56–87. https://doi.org/10.1016/S1474-4422(18)30415-0

    Article  Google Scholar 

  2. Dewan MC, Rattani A, Gupta S, et al (2019) Estimating the global incidence of traumatic brain injury. J Neurosurg 130:1080–1097. https://doi.org/10.3171/2017.10.JNS17352

  3. Multiple cause of death data on CDC WONDER. https://wonder.cdc.gov/mcd.html. Accessed 10 Jan 2023

  4. Zaloshnja E, Miller T, Langlois JA, Selassie AW (2008) Prevalence of long-term disability from traumatic brain injury in the civilian population of the United States, 2005. J Head Trauma Rehabil 23:394–400. https://doi.org/10.1097/01.HTR.0000341435.52004.ac

    Article  PubMed  Google Scholar 

  5. Schweitzer AD, Niogi SN, Whitlow CT, Tsiouris AJ (2019) Traumatic brain injury: imaging patterns and complications. Radiographics 39:1571–1595. https://doi.org/10.1148/rg.2019190076

    Article  PubMed  Google Scholar 

  6. Wintermark M, Li Y, Ding VY et al (2018) Neuroimaging radiological interpretation system for acute traumatic brain injury. J Neurotrauma 35:2665–2672. https://doi.org/10.1089/neu.2017.5311

    Article  PubMed  Google Scholar 

  7. Wintermark M, Sanelli PC, Anzai Y et al (2015) Imaging evidence and recommendations for traumatic brain injury: conventional neuroimaging techniques. J Am Coll Radiol 12:e1-14. https://doi.org/10.1016/j.jacr.2014.10.014

    Article  PubMed  Google Scholar 

  8. Raj R, Siironen J, Skrifvars MB, et al (2014) Predicting outcome in traumatic brain injury: development of a novel computerized tomography classification system (Helsinki computerized tomography score). Neurosurgery 75:632–646; discussion 646–647. https://doi.org/10.1227/NEU.0000000000000533

  9. Marshall LF, Marshall SB, Klauber MR et al (1991) A new classification of head injury based on computerized tomography. J Neurosurg 75:S14–S20. https://doi.org/10.3171/sup.1991.75.1s.0s14

    Article  Google Scholar 

  10. Maas AIR, Hukkelhoven CWPM, Marshall LF, Steyerberg EW (2005) Prediction of outcome in traumatic brain injury with computed tomographic characteristics: a comparison between the computed tomographic classification and combinations of computed tomographic predictors. Neurosurgery 57:1173–1182; discussion 1173–1182. https://doi.org/10.1227/01.neu.0000186013.63046.6b

  11. Grewal M, Srivastava MM, Kumar P, Varadarajan S (2017) RADNET: Radiologist level accuracy using deep learning for HEMORRHAGE detection in CT Scans. https://doi.org/10.48550/ARXIV.1710.04934

  12. Lin E, Yuh EL (2022) Computational approaches for acute traumatic brain injury image recognition. Front Neurol 13:791816. https://doi.org/10.3389/fneur.2022.791816

    Article  PubMed  PubMed Central  Google Scholar 

  13. Vande Vyvere T, Wilms G, Claes L et al (2019) Central versus local radiological reading of acute computed tomography characteristics in multi-center traumatic brain injury research. J Neurotrauma 36:1080–1092. https://doi.org/10.1089/neu.2018.6061

    Article  PubMed  Google Scholar 

  14. Lin X, Yan Z, Kuang Z et al (2022) Fracture R-CNN: An anchor-efficient anti-interference framework for skull fracture detection in CT images. Med Phys 49:7179–7192. https://doi.org/10.1002/mp.15809

    Article  PubMed  Google Scholar 

  15. Yao H, Williamson C, Gryak J, Najarian K (2020) Automated hematoma segmentation and outcome prediction for patients with traumatic brain injury. Artif Intell Med 107:101910. https://doi.org/10.1016/j.artmed.2020.101910

    Article  PubMed  Google Scholar 

  16. Radiology facing a global shortage. https://www.rsna.org/news/2022/may/Global-Radiologist-Shortage. Accessed 18 Nov 2022

  17. Inkeaw P, Angkurawaranon S, Khumrin P et al (2022) Automatic hemorrhage segmentation on head CT scan for traumatic brain injury using 3D deep learning model. Comput Biol Med 146:105530. https://doi.org/10.1016/j.compbiomed.2022.105530

    Article  PubMed  Google Scholar 

  18. Monteiro M, Newcombe VFJ, Mathieu F et al (2020) Multiclass semantic segmentation and quantification of traumatic brain injury lesions on head CT using deep learning: an algorithm development and multicentre validation study. Lancet Digit Health 2:e314–e322. https://doi.org/10.1016/S2589-7500(20)30085-6

    Article  PubMed  Google Scholar 

  19. Pease M, Arefan D, Barber J et al (2022) Outcome prediction in patients with severe traumatic brain injury using deep learning from head CT scans. Radiology 304:385–394. https://doi.org/10.1148/radiol.212181

    Article  PubMed  Google Scholar 

  20. Remedios S, Roy S, Blaber J et al (2019) Distributed deep learning for robust multi-site segmentation of CT imaging after traumatic brain injury. Proc SPIE Int Soc Opt Eng 10949:109490A. https://doi.org/10.1117/12.2511997

    Article  PubMed  PubMed Central  Google Scholar 

  21. Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1:206–215. https://doi.org/10.1038/s42256-019-0048-x

    Article  PubMed  PubMed Central  Google Scholar 

  22. Karabacak M, Ozkara BB, Mordag S, Bisdas S (2022) Deep learning for prediction of isocitrate dehydrogenase mutation in gliomas: a critical approach, systematic review and meta-analysis of the diagnostic test performance using a Bayesian approach. Quant Imaging Med Surg 12:4033–4046. https://doi.org/10.21037/qims-22-34

    Article  PubMed  PubMed Central  Google Scholar 

  23. Zhou B, Ding VY, Li Y et al (2019) Validation of the neuroimaging radiological interpretation system for acute traumatic brain injury. J Comput Assist Tomogr 43:690–696. https://doi.org/10.1097/RCT.0000000000000913

    Article  PubMed  Google Scholar 

  24. Creeden S, Ding VY, Parker JJ et al (2020) Interobserver agreement for the computed tomography severity grading scales for acute traumatic brain injury. J Neurotrauma 37:1445–1451. https://doi.org/10.1089/neu.2019.6871

    Article  PubMed  Google Scholar 

  25. Haacke EM, Duhaime AC, Gean AD et al (2010) Common data elements in radiologic imaging of traumatic brain injury. J Magn Reson Imaging 32:516–543. https://doi.org/10.1002/jmri.22259

    Article  PubMed  Google Scholar 

  26. Kwak R, Kadoya S, Suzuki T (1983) Factors affecting the prognosis in thalamic hemorrhage. Stroke 14:493–500. https://doi.org/10.1161/01.str.14.4.493

    Article  CAS  PubMed  Google Scholar 

  27. Jain S, Vyvere TV, Terzopoulos V et al (2019) Automatic quantification of computed tomography features in acute traumatic brain injury. J Neurotrauma 36:1794–1803. https://doi.org/10.1089/neu.2018.6183

    Article  PubMed  PubMed Central  Google Scholar 

  28. Zeiler MD (2012) ADADELTA: An adaptive learning rate method. https://doi.org/10.48550/ARXIV.1212.5701

  29. Modat M, Ridgway GR, Taylor ZA et al (2010) Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed 98:278–284. https://doi.org/10.1016/j.cmpb.2009.09.002

    Article  PubMed  Google Scholar 

  30. Çiçek Ö, Abdulkadir A, Lienkamp SS, et al (2016) 3D U-Net: Learning dense volumetric segmentation from sparse annotation. https://doi.org/10.48550/ARXIV.1606.06650

  31. McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 22:276–282

    Article  PubMed  Google Scholar 

  32. Parizel PM, Kremling J, Janssen C, et al (2015) Radiologic imaging of sports-induced brain injuries. In: Glaudemans AWJM, Dierckx RAJO, Gielen JLMA, Zwerver J (Hans) (eds) Nuclear medicine and radiologic imaging in sports injuries. Springer, Berlin, Heidelberg, pp 147–170

  33. Yuh EL, Mukherjee P, Lingsma HF et al (2013) Magnetic resonance imaging improves 3-month outcome prediction in mild traumatic brain injury. Ann Neurol 73:224–235. https://doi.org/10.1002/ana.23783

    Article  PubMed  Google Scholar 

  34. Quon JL, Han M, Kim LH et al (2020) Artificial intelligence for automatic cerebral ventricle segmentation and volume calculation: a clinical tool for the evaluation of pediatric hydrocephalus. J Neurosurg Pediatr 27:131–138. https://doi.org/10.3171/2020.6.PEDS20251

    Article  PubMed  PubMed Central  Google Scholar 

  35. Klimont M, Flieger M, Rzeszutek J et al (2019) Automated ventricular system segmentation in paediatric patients treated for hydrocephalus using deep learning methods. Biomed Res Int 2019:3059170. https://doi.org/10.1155/2019/3059170

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kok YE, Pszczolkowski S, Law ZK et al (2022) Semantic segmentation of spontaneous intracerebral hemorrhage, intraventricular hemorrhage, and associated edema on CT images using deep learning. Radiol Artif Intell 4:e220096. https://doi.org/10.1148/ryai.220096

    Article  PubMed  PubMed Central  Google Scholar 

  37. Peng Q, Chen X, Zhang C et al (2022) Deep learning-based computed tomography image segmentation and volume measurement of intracerebral hemorrhage. Front Neurosci 16:965680. https://doi.org/10.3389/fnins.2022.965680

    Article  PubMed  PubMed Central  Google Scholar 

  38. Kim CH, Hahm MH, Lee DE et al (2021) Clinical usefulness of deep learning-based automated segmentation in intracranial hemorrhage. Technol Health Care 29:881–895. https://doi.org/10.3233/THC-202533

    Article  PubMed  Google Scholar 

  39. Farzaneh N, Williamson CA, Jiang C et al (2020) Automated segmentation and severity analysis of subdural hematoma for patients with traumatic brain injuries. Diagnostics (Basel) 10:773. https://doi.org/10.3390/diagnostics10100773

    Article  CAS  PubMed  Google Scholar 

  40. L L, M W, B L, et al (2021) Deep learning for hemorrhagic lesion detection and segmentation on brain CT images. IEEE J Biomed Health Informa 25. https://doi.org/10.1109/JBHI.2020.3028243

  41. Dhar R, Falcone GJ, Chen Y et al (2020) Deep learning for automated measurement of hemorrhage and perihematomal edema in supratentorial intracerebral hemorrhage. Stroke 51:648–651. https://doi.org/10.1161/STROKEAHA.119.027657

    Article  PubMed  Google Scholar 

  42. Chilamkurthy S, Ghosh R, Tanamala S et al (2018) Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet 392:2388–2396. https://doi.org/10.1016/S0140-6736(18)31645-3

    Article  PubMed  Google Scholar 

  43. Phaphuangwittayakul A, Guo Y, Ying F et al (2022) An optimal deep learning framework for multi-type hemorrhagic lesions detection and quantification in head CT images for traumatic brain injury. Appl Intell (Dordr) 52:7320–7338. https://doi.org/10.1007/s10489-021-02782-9

    Article  PubMed  Google Scholar 

Download references

Author information

Author notes

  1. Bin Jiang and Burak Berksu Ozkara are equally contributing authors

Authors and Affiliations

  1. Department of Radiology, Neuroradiology Division, Stanford University, Stanford, CA, USA

    Bin Jiang, Bryan Lanzman, Sara Shams & Ying Li

  2. Department of Neuroradiology, MD Anderson Cancer Center, Houston, TX, USA

    Burak Berksu Ozkara, Hui Chen & Max Wintermark

  3. Deparment of Neuroradiology, University of Illinois College of Medicine Peoria, Peoria, IL, USA

    Sean Creeden

  4. Department of Neurology, The University of Arizona, Tucson, AZ, USA

    Guangming Zhu

  5. Department of Medicine, Stanford University, Stanford, CA, USA

    Victoria Y. Ding

  6. Department of Neuroimaging and Neurointervention, Stanford University, Stanford, CA, USA

    Dylan Wolman, Austin Trinh & Alexander Khalaf

  7. Department of Radiology, Karolinska University Hospital, Stockholm, Sweden

    Sara Shams

  8. Institution for Clinical Neuroscience, Karolinska Institute, Stockholm, Sweden

    Sara Shams

  9. Device-Based Neuroelectronics Laboratory, Mayo Clinic, Phoenix, AZ, USA

    Jonathon J. Parker

  10. Department of Neurological Surgery, Mayo Clinic, Phoenix, AZ, USA

    Jonathon J. Parker

  11. Department of Neurosurgery, University of Pennsylvania School of Medicine, Philadelphia, PA, USA

    Casey H. Halpern

  12. Department of Surgery, Corporal Michael J. Crescenz Veterans Affairs Medical Center, Philadelphia, PA, USA

    Casey H. Halpern

Authors
  1. Bin Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Burak Berksu Ozkara
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sean Creeden
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangming Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Victoria Y. Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bryan Lanzman
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dylan Wolman
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sara Shams
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Austin Trinh
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Alexander Khalaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Jonathon J. Parker
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Casey H. Halpern
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Max Wintermark
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Max Wintermark.

Ethics declarations

Competing interests

Dr. Max Wintermark is a consultant for Icometrix. The other authors have nothing to disclose in relation to this article.

Ethical approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Institutional Review Board at Stanford University.

Informed consent

The requirement to obtain written informed consent was waived due to the study’s retrospective nature.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, B., Ozkara, B.B., Creeden, S. et al. Validation of a deep learning model for traumatic brain injury detection and NIRIS grading on non-contrast CT: a multi-reader study with promising results and opportunities for improvement. Neuroradiology 65, 1605–1617 (2023). https://doi.org/10.1007/s00234-023-03170-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00234-023-03170-5

Keywords

Search

Navigation