Skip to main content

Advertisement

SpringerLink
Log in

Validation of an artificial intelligence solution for acute triage and rule-out normal of non-contrast CT head scans

  • Diagnostic Neuroradiology
  • Published:
Neuroradiology Aims and scope Submit manuscript

Abstract

Purpose

Non-contrast CT head scans provide rapid and accurate diagnosis of acute head injury; however, increased utilisation of CT head scans makes it difficult to prioritise acutely unwell patients and places pressure on busy emergency departments (EDs). This study validates an AI algorithm to triage patients presenting with Intracranial Haemorrhage (ICH) or Acute Infarct whilst also identifying a subset of patients as Normal, with the potential to function as a rule-out test.

Methods

In total, 390 CT head scans were collected from 3 institutions in the UK, US and India. Ground-truth labels were assigned by 3 FRCR consultant radiologists. AI performance, as well as the performance of 3 independent radiologists, was measured against ground-truth labels.

Results

The algorithm showed AUC values of 0.988 (0.978–0.994), 0.933 (0.901–0.961) and 0.939 (0.919–0.958) for ICH, Acute Infarct and Normal, respectively. Sensitivity/specificity for ICH and Acute Infarct were 0.988/0.925 and 0.833/0.927, respectively, compared to 0.907/0.991 and 0.618/0.977 for radiologists. AI rule-out of Normal scans achieved 0.93% negative predictive value (NPV) for the removal of 54.3% of Normal cases, compared to 86.8% NPV for radiologists.

Conclusion

We show our algorithm can provide effective triage of ICH and Acute Infarct to prioritise acutely unwell patients. AI can also benefit clinical accuracy, with the algorithm identifying 91.3% of radiologist false negatives for ICH and 69.1% for Acute Infarct. Rule-out of Normal scans has huge potential for workload management in busy EDs, in this case removing 27.4% of all scans with no acute findings missed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

Due to the mixture of data sources, data used in the study is not publicly available. All image data from public datasets is available; however, data labels are proprietary.

Code availability

All code is proprietary.

References

  1. Klang E et al (2017) Overuse of head CT examinations for the investigation of minor head trauma: analysis of contributing factors. J Am Coll Radiol 14(2):171–176. https://doi.org/10.1016/j.jacr.2016.08.032

    Article  PubMed  Google Scholar 

  2. Coles JP (2007) Imaging after brain injury. Br J Anaesth 99(1):49–60. https://doi.org/10.1093/bja/aem141

    Article  CAS  PubMed  Google Scholar 

  3. Saver JL (2006) Time is brain - quantified. Stroke 37(1):263–266. https://doi.org/10.1161/01.STR.0000196957.55928.ab

    Article  PubMed  Google Scholar 

  4. Jovin TG et al (2015) Thrombectomy within 8 hours after symptom onset in ischemic stroke. N Engl J Med 372(24):2296–2306. https://doi.org/10.1056/nejmoa1503780

    Article  CAS  PubMed  Google Scholar 

  5. Sauser K, Levine DA, Nickles AV, Reeves MJ (2014) Hospital variation in thrombolysis times among patients with acute ischemic stroke: the contributions of door-to-imaging time and imaging-to-needle time. JAMA Neurol 71(9):1155–1161. https://doi.org/10.1001/jamaneurol.2014.1528

    Article  PubMed  Google Scholar 

  6. Kapur N (2020) The NHS long term plan SUSHRUTA. J. Heal. Policy Opin 12(1):10–11. https://doi.org/10.38192/12.1.4

    Article  Google Scholar 

  7. A. Patel, V. Berdunov, D. King, Z. Quayyum, R. Wittenberg, and M. Knapp, “Current, future and avoidable costs of stroke in the UK Part 2: Societal costs of stroke in the next 20 years and potential returns from increased spending on research,” Stroke Assoc., p. 12, 2017, [Online]. Available: https://www.stroke.org.uk/sites/default/files/costs_of_stroke_in_the_uk_report_-executive_summary_part_2.pdf.

  8. Kuo W, Hӓne C, Mukherjee P, Malik J, Yuh EL (2019) Expert-level detection of acute intracranial hemorrhage on head computed tomography using deep learning. Proc Natl Acad Sci U S A 116(45):22737–22745. https://doi.org/10.1073/pnas.1908021116

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Lee H et al (2019) An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 3(3):173–182. https://doi.org/10.1038/s41551-018-0324-9

    Article  PubMed  Google Scholar 

  10. McLouth J et al (2021) Validation of a deep learning tool in the detection of intracranial hemorrhage and large vessel occlusion. Front Neurol 12(April):1–10. https://doi.org/10.3389/fneur.2021.656112

    Article  Google Scholar 

  11. O’Neill TJ et al (2020) Active reprioritization of the reading worklist using artificial intelligence has a beneficial effect on the turnaround time for interpretation of head CTs with intracranial hemorrhage. Radiol. Artif. Intell. 7:e200024. https://doi.org/10.1148/ryai.2020200024

    Article  Google Scholar 

  12. Goebel J et al (2018) Automated ASPECT rating: comparison between the Frontier ASPECT Score software and the Brainomix software. Neuroradiol. https://doi.org/10.1007/s00234-018-2098-x

    Article  Google Scholar 

  13. Chilamkurthy S et al (2018) Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet 392(10162):2388–2396. https://doi.org/10.1016/S0140-6736(18)31645-3

    Article  PubMed  Google Scholar 

  14. N. I. for H. and C. E. NICE, “Head injury triage, assessment, investigation and early management of head injury in children, young people and adults,” NICE Clin. Guidel., no. January, 2014.

  15. T. Dyer et al., “Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm,” Clin. Radiol., no. xxxx, 2021, https://doi.org/10.1016/j.crad.2021.01.015.

  16. Flanders AE et al (2020) Construction of a machine learning dataset through collaboration: the RSNA 2019 brain CT hemorrhage challenge. Radiol Artif Intell 2(4):e209002. https://doi.org/10.1148/ryai.2020209002

    Article  PubMed  PubMed Central  Google Scholar 

  17. M. Tan and Q. V. Le, “EfficientNet: rethinking model scaling for convolutional neural networks,” 2019.

  18. B. Zhou, Y. Li, and J. Wang, “A weakly supervised adaptive DenseNet for classifying thoracic diseases and identifying abnormalities,” 2018, [Online]. Available: http://arxiv.org/abs/1807.01257.

  19. Hssayeni MD, Croock MS, Salman AD, Al-Khafaji HF, Yahya ZA, Ghoraani B (2020) Intracranial hemorrhage segmentation using a deep convolutional model. Data 5(1):1–18. https://doi.org/10.3390/data5010014

    Article  Google Scholar 

  20. Noma H, Matsushima Y, Ishii R (2021) Confidence interval for the AUC of SROC curve and some related methods using bootstrap for meta-analysis of diagnostic accuracy studies. Commun Stat Case Stud Data Anal Appl. https://doi.org/10.1080/23737484.2021.1894408

    Article  Google Scholar 

  21. M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochem. Medica, 2012, https://doi.org/10.11613/bm.2012.031.

  22. H. Yoo et al., “AI-based improvement in lung cancer detection on chest radiographs : results of a multi-reader study in NLST dataset,” 2021.

  23. M. Bernhardt et al., “Active label cleaning: improving dataset quality under resource constraints,” 2021, [Online]. Available: http://arxiv.org/abs/2109.00574.

  24. Hwang EJ et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiol 293(3):573–580. https://doi.org/10.1148/radiol.2019191225

    Article  Google Scholar 

  25. Nael K et al (2021) Automated detection of critical findings in multi-parametric brain MRI using a system of 3D neural networks. Sci Rep 11(1):1–10. https://doi.org/10.1038/s41598-021-86022-7

    Article  CAS  Google Scholar 

  26. Royal College of Radiologists, “Clinical radiology UK workforce census 2020 report,” R. Coll. Radiol., no. April, p. 72, 2021, [Online]. Available: https://www.rcr.ac.uk/system/files/publication/field_publication_files/clinical-radiology-uk-workforce-census-2020-report.pdf.

  27. Nabulsi Z et al (2021) Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci Rep 11(1):1–15. https://doi.org/10.1038/s41598-021-93967-2

    Article  CAS  Google Scholar 

  28. E. Chen and A. Y. Ng, “CheXbreak : misclassification Identification for deep learning models interpreting chest X-rays.”

Download references

Funding

Behold.ai was the sole funder of this study.

Author information

Authors and Affiliations

  1. Behold.ai, 180 Borough high St, London, SE1 1LB, UK

    Tom Dyer, Tom Naunton Morgan, Mike Hughes & Simon Rasalingham

  2. Department of Radiology, Barking, Havering and Redbridge University Hospitals NHS Trust, Romford, RM7 0AG, UK

    Sanjiv Chawda & Raed Alkilani

Authors
  1. Tom Dyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanjiv Chawda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raed Alkilani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tom Naunton Morgan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mike Hughes
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Simon Rasalingham
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Wrote the paper — TD.

Conceived design and analysis — TD, SC, SR.

Algorithm development — TD, MH, TNM, SC, RA.

Corresponding author

Correspondence to Tom Dyer.

Ethics declarations

Ethics approval

Ethics approval was not required either because data was obtained from public datasets or the requirement was waived due to the retrospective nature of the study.

Consent to participate

Not applicable.

Consent for publication

All authors give consent to publish.

Conflict of interest

TD, SR and TNM are employees of Behold.ai. SC, RA, TNM, MH and SR are stock/share-holders in Behold.ai.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dyer, T., Chawda, S., Alkilani, R. et al. Validation of an artificial intelligence solution for acute triage and rule-out normal of non-contrast CT head scans. Neuroradiology 64, 735–743 (2022). https://doi.org/10.1007/s00234-021-02826-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00234-021-02826-4

Keywords

Search

Navigation