Skip to main content

Advertisement

SpringerLink
Log in

The R-AI-DIOLOGY checklist: a practical checklist for evaluation of artificial intelligence tools in clinical neuroradiology

  • Review
  • Published:
Neuroradiology Aims and scope Submit manuscript

Abstract

Artificial intelligence (AI)-based tools are gradually blending into the clinical neuroradiology practice. Due to increasing complexity and diversity of such AI tools, it is not always obvious for the clinical neuroradiologist to capture the technical specifications of these applications, notably as commercial tools very rarely provide full details. The clinical neuroradiologist is thus confronted with the increasing dilemma to base clinical decisions on the output of AI tools without knowing in detail what is happening inside the “black box” of those AI applications. This dilemma is aggravated by the fact that currently, no established and generally accepted rules exist concerning best clinical practice and scientific and clinical validation nor for the medico-legal consequences in cases of wrong diagnoses. The current review article provides a practical checklist of essential points, intended to aid the user to identify and double-check necessary aspects, although we are aware that not all this information may be readily available at this stage, even for certified and commercially available AI tools. Furthermore, we therefore suggest that the developers of AI applications provide this information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

Not applicable (review article).

Code availability

Not applicable (review article).

Abbreviations

AD:

Alzheimer dementia

ADNI:

Alzheimer Disease Neuroimaging Initiative

AI:

Artificial intelligence

CADASIL:

Cerebral autosomal dominant arteriopathy with subcortical infarcts and leukoencephalopathy

CNN:

Convolutional neural networks

CT:

Computed tomography

DICOM:

Digital Imaging and Communications in Medicine

DL:

Deep learning

GDPR:

General Data Protection Regulation

MPRAGE:

Magnetization prepared rapid gradient-echo

MR:

Magnetic resonance

MS:

Multiple sclerosis

PHI:

Personal health information

RANO:

Response assessment in neurooncology

SaMD:

Software as a medical device

SL:

Supervised learning

SLE:

Systemic lupus erythematosus

SVD:

Small vessel disease

USL:

Unsupervised learning

References

  1. Topol EJ (2019) High-performance medicine: the convergence of human and artificial intelligence. Nat Med 25(1):44–56

    Article  CAS  Google Scholar 

  2. Muehlematter UJ, Daniore P, Vokinger KN (2021) Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. Lancet Digit Health 3:e195–e203

    Article  Google Scholar 

  3. Pesapane F, Codari M, Sardanelli F (2018) Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine. Eur Radiol Exp 2(1):35

    Article  Google Scholar 

  4. https://grand-challenge.org/aiforradiology/

  5. Hedderich DM, Keicher M, Wiestler B, & Gruber… MJ (2021) AI for doctors—a course to educate medical professionals in artificial intelligence for medical imaging. Healthcare

  6. Huisman M, Ranschaert E, Parker W, Mastrodicasa D, Koci M, Pinto de Santos D, Coppola F, Morozov S, Zins M, Bohyn C, Koç U, Wu J, Veean S, Fleischmann D, Leiner T, Willemink MJ (2021) An international survey on AI in radiology in 1,041 radiologists and radiology residents part 1: fear of replacement, knowledge, and attitude. Eur Radiol 31(9):7058–7066

    Article  Google Scholar 

  7. Lui YW, Chang PD, Zaharchuk G, Barboriak DP, Flanders AE, Wintermark M, Hess CP, Filippi CG (2020) Artificial intelligence in neuroradiology: current status and future directions. AJNR Am J Neuroradiol 41(8):E52–E59

    CAS  PubMed  PubMed Central  Google Scholar 

  8. Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ, Kadoury S, Tang A (2017) Deep learning: a primer for radiologists. Radiographics 37(7):2113–2131

    Article  Google Scholar 

  9. Cheng PM, Montagnon E, Yamashita R, Pan I, Cadrin-Chênevert A, Perdigón Romero F, Chartrand G, Kadoury S, Tang A (2021) Deep learning: an update for radiologists. Radiographics 41(5):1427–1445

    Article  Google Scholar 

  10. https://phil415.pbworks.com/f/TuringComputing.pdf

  11. Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221–248

    Article  CAS  Google Scholar 

  12. Yamashita R, Nishio M, Do RKG, Togashi K (2018) Convolutional neural networks: an overview and application in radiology. Insights Imaging 9(4):611–629

    Article  Google Scholar 

  13. Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286(3):800–809

    Article  Google Scholar 

  14. Sica GT (2006) Bias in research studies. Radiology 238(3):780–789

    Article  Google Scholar 

  15. Molina D, Pérez-Beteta J, Martínez-González A, Martino J, Velasquez C, Arana E, Pérez-García VM (2017) Lack of robustness of textural measures obtained from 3D brain tumor MRIs impose a need for standardization. PLoS One 12(6):e0178843

    Article  Google Scholar 

  16. Valverde JM, Imani V, Abdollahzadeh A, De Feo R, Prakash M, Ciszek R, Tohka J (2021) Transfer learning in magnetic resonance brain imaging: a systematic review. J Imaging 7(4):66

    Article  Google Scholar 

  17. Haller S, Falkovskiy P, Meuli R, Thiran JP, Krueger G, Lovblad KO, Kober T, Roche A, Marechal B (2016) Basic MR sequence parameters systematically bias automated brain volume estimation. Neuroradiology 58(11):1153–1160

    Article  Google Scholar 

  18. Biberacher V, Schmidt P, Keshavan A, Boucard CC, Righart R, Sämann P, Preibisch C, Fröbel D, Aly L, Hemmer B, Zimmer C, Henry RG, Mühlau M (2016) Intra- and interscanner variability of magnetic resonance imaging based volumetry in multiple sclerosis. Neuroimage 14:2188–197

    Google Scholar 

  19. Chaddad A, Kucharczyk MJ, Daniel P, Sabri S, Jean-Claude BJ, Niazi T, Abdulkarim B (2019) Radiomics in glioblastoma: current status and challenges facing clinical implementation. Front Oncol 9:374

    Article  Google Scholar 

  20. Larson DB, Boland GW (2019) Imaging quality control in the era of artificial intelligence. J Am Coll Radiol 16(9 Pt B):1259–1266

    Article  Google Scholar 

  21. Copelan AZ, Smith ER, Drocton GT, Narsinh KH, Murph D, Khangura RS, Hartley ZJ, Abla AA, Dillon WP, Dowd CF, Higashida RT, Halbach VV, Hetts SW, Cooke DL, Keenan K, Nelson J, Mccoy D, Ciano M, Amans MR (2020) Recent administration of iodinated contrast renders core infarct estimation inaccurate using RAPID software. AJNR Am J Neuroradiol 41(12):2235–2242

    Article  CAS  Google Scholar 

  22. Aryanto KY, Oudkerk M, van Ooijen PM (2015) Free DICOM de-identification tools in clinical research: functioning and safety of patient privacy. Eur Radiol 25(12):3685–3695

    Article  CAS  Google Scholar 

  23. Schwarz CG, Kremers WK, Therneau TM, Sharp RR, Gunter JL, Vemuri P, Arani A, Spychalla AJ, Kantarci K, Knopman DS, Petersen RC, Jack CR (2019) Identification of anonymous MRI research participants with face-recognition software. N Engl J Med 381(17):1684–1686

    Article  Google Scholar 

  24. Schwarz CG, Kremers WK, Wiste HJ, Gunter JL, Vemuri P, Spychalla AJ, Kantarci K, Schultz AP, Sperling RA, Knopman DS, Petersen RC, Jack CR, Alzheimer’s DNI, (2021) Changing the face of neuroimaging research: comparing a new MRI de-facing technique with popular alternatives. Neuroimage 231:117845

    Article  Google Scholar 

  25. Neubauer T, Heurix J (2011) A methodology for the pseudonymization of medical data. Int J Med Inform 80(3):190–204

    Article  Google Scholar 

  26. Ngiam KY, Khor IW (2019) Big data and machine learning algorithms for health-care delivery. Lancet Oncol 20(5):e262–e273

    Article  Google Scholar 

  27. Willemink MJ, Koszek WA, Hardell C, Wu J, Fleischmann D, Harvey H, Folio LR, Summers RM, Rubin DL, Lungren MP (2020) Preparing medical imaging data for machine learning. Radiology 295(1):4–15

    Article  Google Scholar 

  28. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017R0745&from=EN; Annex VIII

  29. https://ec.europa.eu/health/sites/default/files/md_sector/docs/mdcg_2021-24_en.pdf AOD

  30. https://www.fda.gov/medical-devices/overview-device-regulation/classify-your-medical-device AOD

  31. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices AOD

  32. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device AOD

  33. van Leeuwen KG, Schalekamp S, Rutten MJCM, van Ginneken B, de Rooij M (2021) Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol 31(6):3797–3804

    Article  Google Scholar 

  34. Pemberton HG, Zaki LAM, Goodkin O, Das RK, Steketee RME, Barkhof F, Vernooij MW (2021) Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis-a systematic review. Neuroradiology 63(11):1773–1789

    Article  Google Scholar 

Download references

Funding

None.

Author information

Authors and Affiliations

  1. CIMC-Centre d’Imagerie Médicale de Cornavin, Place de Cornavin 18, 1201, Geneva, Switzerland

    Sven Haller

  2. Department of Surgical Sciences, Radiology, Uppsala University, Uppsala, Sweden

    Sven Haller

  3. Faculty of Medicine, University of Geneva, Geneva, Switzerland

    Sven Haller

  4. Department of Radiology, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, People’s Republic of China

    Sven Haller

  5. Department of Medical Imaging Ziekenhuis, Oost-Limburg Genk, Schiepse Bos 6, 3600, Genk, Belgium

    Sofie Van Cauter

  6. Department of Radiology, University Hospitals Leuven, Herestraat 49, 3000, Leuven, Belgium

    Sofie Van Cauter

  7. Division of Medicine and Life Sciences, Department Neurosciences, Hasselt University, Campus Diepenbeek, Agoralaan Building D, 3590, Diepenbeek, Belgium

    Sofie Van Cauter

  8. AI Medical AG, Goldhaldenstr 22a, Zollikon, CH-8702, Switzerland

    Christian Federau

  9. Faculty of Medicine, University of Zürich, Pestalozzistrasse 3, CH-8032, Zurich, Switzerland

    Christian Federau

  10. Department of Neuroradiology, Klinikum rechts der Isar, School of Medicine, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany

    Dennis M. Hedderich

  11. Department of Radiology, APHP, Hôpitaux Raymond-Poincaré & Ambroise Paré, DMU Smart Imaging, GH Université Paris-Saclay, U 1179 UVSQ/Paris-Saclay, Paris, France

    Myriam Edjlali

  12. Laboratoire d’imagerie Biomédicale Multimodale (BioMaps), Université Paris-Saclay, CEA, CNRS, Inserm, Service Hopsitalier Frédéric Joliot, Orsay, France

    Myriam Edjlali

Authors
  1. Sven Haller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sofie Van Cauter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Federau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dennis M. Hedderich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Myriam Edjlali
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Sven HALLER

Literature review, manuscript editing, manuscript approval, figures.

Sofie Van Cauter

Literature review, manuscript editing, manuscript approval, figures.

Christian Federau

Literature review, manuscript editing, manuscript approval.

Dennis M. Hedderich

Literature review, manuscript editing, manuscript approval.

Myriam Edjlalis

Literature review, manuscript editing, manuscript approval.

Corresponding author

Correspondence to Sven Haller.

Ethics declarations

Conflict of interest

Christian FEDERAU is working for AI Medical, yet there is no reference to this company in the current review article.

The other authors declare no conflict of interest related to the content of this review article.

Ethics approval

Not applicable (review article).

Consent to participate

Not applicable (review article).

Consent for publication

Not applicable (review article).

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haller, S., Van Cauter, S., Federau, C. et al. The R-AI-DIOLOGY checklist: a practical checklist for evaluation of artificial intelligence tools in clinical neuroradiology. Neuroradiology 64, 851–864 (2022). https://doi.org/10.1007/s00234-021-02890-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00234-021-02890-w

Keywords

Search

Navigation