Abstract
Purpose
Artificial intelligence (AI) has emerged as a transformative force in medical research and is garnering increased attention in the public consciousness. This represents a critical time period in which medical researchers, healthcare providers, insurers, regulatory agencies, and patients are all developing and shaping their beliefs and policies regarding the use of AI in the healthcare sector. The successful deployment of AI will require support from all these groups. This commentary proposes that widespread support for medical AI must be driven by clear and transparent scientific reporting, beginning at the earliest stages of scientific research.
Methods
A review of relevant guidelines and literature describing how scientific reporting plays a central role at key stages in the life cycle of an AI software product was conducted. To contextualize this principle within a specific medical domain, we discuss the current state of predictive tissue outcome modeling in acute ischemic stroke and the unique challenges presented therein.
Results and Conclusion
Translating AI methods from the research to the clinical domain is complicated by challenges related to model design and validation studies, medical product regulations, and healthcare providers’ reservations regarding AI’s efficacy and affordability. However, each of these limitations is also an opportunity for high-impact research that will help to accelerate the clinical adoption of state-of-the-art medical AI. In all cases, establishing and adhering to appropriate reporting standards is an important responsibility that is shared by all of the parties involved in the life cycle of a prospective AI software product.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Research Nester. Artificial Intelligence (AI) in Healthcare Market Analysis by Technology; by Offering; by Application and by End-User—Global Supply & Demand Analysis & Opportunity Outlook 2023–2033. 2023. https://www.researchnester.com/reports/artificial-intelligence-in-healthcare-market/1429. Accessed 01 Aug 2023.
GBD 2019 Collaborators. Global, regional, and national burden of diseases and injuries for adults 70 years and older: systematic analysis for the Global Burden of Disease 2019 Study. BMJ. 2022;376:e068208.
Mather M, Jacobsen LA, Pollard KM. Aging in the United States, vol. 2. In: Population Bulletin, no. 70, vol. 2. Population Reference Bureau. 2015. https://www.prb.org/wp-content/uploads/2019/07/population-bulletin-2015-70-2-aging-us.pdf. Accessed 01 Aug 2023.
Boniol M, Kunjumen T, Nair TS, Siyam A, Campbell J, Diallo K. The global health workforce stock and distribution in 2020 and 2030: a threat to equity and ‘universal’ health coverage? Bmj Glob Health. 2022;7(6):e9316.
Hartman M, Martin AB, Washington B, Catlin A, National Health Expenditure Accounts Team T. National Health Care Spending In 2020: Growth Driven By Federal Spending In Response To The COVID-19 Pandemic: National Health Expenditures study examines US health care spending in 2020. Health Aff. 2022;41(1):13–25.
Wolff J, Pauling J, Keck A, Baumbach J. Systematic Review of Economic Impact Studies of Artificial Intelligence in Health Care. J Med Internet Res. 2020;22(2):e16866.
Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. In: Artif Intell Heal Elsevier. 2020;pp:25–60.
Ebrahimian S, et al. FDA-regulated AI Algorithms: Trends, Strengths, and Gaps of Validation Studies. Academic Radiology. 2022;29(4):559–566.
Van Der Heijden AA, Abramoff MD, Verbraak F, Van Hecke MV, Liem A, Nijpels G. Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmol. 2018;96(1):63–8.
Huang X‑M, et al. Cost-effectiveness of artificial intelligence screening for diabetic retinopathy in rural China. BMC Health Serv Res. 2022;22(1):260.
Bellemo V, et al. Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study. Lancet Digit Health. 2019;1(1):e35–e44.
Straka M, Albers GW, Bammer R. Real-time diffusion-perfusion mismatch analysis in acute stroke. J Magn Reson Imaging. 2010;32(5):1024–37.
FDA approves stroke-detecting AI software. Nat Biotechnol. 2018;36(4):290.
Van Leeuwen KG, et al. Cost-effectiveness of artificial intelligence aided vessel occlusion detection in acute stroke: an early health technology assessment. Insights Imaging. 2021;12(1):133.
Fuller SD, et al. Five-Year Cost-Effectiveness Modeling of Primary Care-Based, Nonmydriatic Automated Retinal Image Analysis Screening Among Low-Income Patients With Diabetes. J Diabetes Sci Technol. 2022;16(2):415–27.
Schwendicke F, et al. Cost-effectiveness of Artificial Intelligence for Proximal Caries Detection. J Dent Res. 2021;100(4):369–76.
AI/ML-Enabled Medical Devices. FDA. 2023. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices. Accessed 01 Aug 2023.
Souza R, et al. Image-encoded biological and non-biological variables may be used as shortcuts in deep learning models trained on multisite neuroimaging data. J Am Med Inform Assoc. 2023;30(12):1925–33.
Stanley EAM, Wilms M, Forkert ND. Disproportionate Subgroup Impacts and Other Challenges of Fairness in Artificial Intelligence for Medical Image Analysis. In: Baxter JSH et al. editors. Lecture Notes in Computer Science, vol 13755. Springer:Cham; 2022. Pp. 14–25.
Stanley EAM, et al. Towards objective and systematic evaluation of bias in medical imaging AI. Arxiv. 2023;2311:2115.
Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol Artif Intell. 2020;2(2):e200029.
Norgeot B, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med. 2020;26(9):1320–4.
Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med. 2015;162(1):55–63.
Wolff RF, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019;170(1):51.
Collins GS, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. Bmj Open. 2021;11(7):e48008.
Naverro CLA, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques. Syst Rev.
Roberts M, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat Mach Intell. 2021;3(3):199–217.
Wen J, Zhang Z, Lan Y, Cui Z, Cai J, Zhang W. A survey on federated learning: challenges and applications. Int J Mach Learn Cybern. 2023;14:513–35.
Artificial Intelligence / Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. FDA. 2021. https://www.fda.gov/media/145022/download. Accessed 01 Aug 2023.
The Software Precertification (Pre-Cert) Pilot Program: Tailored Total Product Lifecycle Approaches and Key Findings. FDA. 2022. https://www.fda.gov/media/161815/download. Accessed 01 Aug 2023.
Good Machine Learning Practice for Medical Device Development: Guiding Principles. FDA, Health Canada, Medicines and Healthcare Products Regulatory Agency. 2021. https://www.fda.gov/media/153486/download. Accessed 01 Aug 2023.
Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447–53.
Straw I, Wu H. Investigating for bias in healthcare algorithms: a sex-stratified analysis of supervised machine learning models in liver disease prediction. Bmj Health Care Inform. 2022;29(1):e100457.
Beede E, et al. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM;. 2020;pp:1–12.
Reis L, Maier C, Mattke J, et al. Addressing User Resistance Would Have Prevented a Healthcare AI Project Failure. MSQE. 2020; https://doi.org/10.17705/2msqe.00038.
Canon CL, Chick JFB, DeQuesada I, Gunderman RB, Hoven N, Prosper AE. Physician Burnout in Radiology: Perspectives From the Field. Am J Roentgenol. 2022;218(2):370–4.
Yates SW. Physician Stress and Burnout. Am J Med. 2020;133(2):160–4.
Fred HL, Physician Burnout SMS. Causes, Consequences, and (?) Cures. Tex Heart Inst J. 2018;45(4):198–202.
Li C, Parpia C, Sriharan A, Keefe DT. Electronic medical record-related burnout in healthcare providers: a scoping review of outcomes and interventions. Bmj Open. 2022;12(8):e60865.
Stern AD, Goldfarb A, Minssen T, Price WNII. AI Insurance: How Liability Insurance Can Drive the Responsible Adoption of Artificial Intelligence in Health Care. Nejm Catal. 2022;3(4).
Chen Y, Stavropoulou C, Narasinkan R, Baker A, Scarbrough H. Professionals’ responses to the introduction of AI innovations in radiology and their implications for future adoption: a qualitative study. BMC Health Serv Res. 2021;21(1):813.
Kurowecki D, Lee SY, Monteiro S, Finlay K. Resident Physicians’ Perceptions of Diagnostic Radiology and the Declining Interest in the Specialty. Acad Radiol. 2021;28(2):261–70.
Chen MM, Golding LP, Nicola GN. Who Will Pay for AI?. Radiology: Artificial Intelligence. 2021;3(3):e210030.
Varoquaux G, Cheplygina V. Machine learning for medical imaging: methodological failures and recommendations for the future. Npj Digit Med. 2022;5(1):48.
Fernández-Delgado M, Cernadas E, Barro S, Amorim D. Do we need hundreds of classifiers to solve real world classification problems? J Mach Learn Res. 2014;15(1):3133–81.
NSERC Alliance—Alberta Innovates (AI) Advance grants. Natural Sciences and Engineering Research Council of Canada. 2023. https://www.nserc-crsng.gc.ca/Innovate-Innover/AI-AI_eng.asp. Accessed 01 Aug 2023.
Equitable AI activating opportunities. Canadian Institutes of Health Research. 2023. https://cihr-irsc.gc.ca/e/51204.html. Accessed 01 Aug 2023.
Yu W, Jiang W‑J. A Simple Imaging Guide for Endovascular Thrombectomy in Acute Ischemic Stroke: From Time Window to Perfusion Mismatch and Beyond. Front Neurol. 2019;10:502.
Heran M, et al. Canadian Stroke Best Practice Recommendations: Acute Stroke Management, 7th Edition Practice Guidelines Update. Can J Neurol Sci. 2022; https://doi.org/10.1017/cjn.2022.344.
Goyal M, et al. Challenging the Ischemic Core Concept in Acute Ischemic Stroke Imaging. Stroke. 2020;51(10):3147–55.
Winder AJ, Siemonsen S, Flottmann F, Thomalla G, Fiehler J, Forkert ND. Technical considerations of multi-parametric tissue outcome prediction methods in acute ischemic stroke patients. Sci Rep. 2019;9(1):13208.
Winder AJ, Wilms M, Amador K, Flottmann F, Fiehler J, Forkert ND. Predicting the tissue outcome of acute ischemic stroke from acute 4D computed tomography perfusion imaging using temporal features and deep learning. Front Neurosci. 2022;16:1009654.
Amador K, Wilms M, Winder A, Fiehler J, Forkert ND. Predicting treatment-specific lesion outcomes in acute ischemic stroke from 4D CT perfusion imaging using spatio-temporal convolutional neural networks. Med Image Anal. 2022;82:102610.
Winzeck S, et al. ISLES 2016 and 2017-Benchmarking Ischemic Stroke Lesion Outcome Prediction Based on Multispectral MRI. Front Neurol. 2018;9:679.
Wang X, Fan Y, Zhang N, Li J, Duan Y, Yang B. Performance of Machine Learning for Tissue Outcome Prediction in Acute Ischemic Stroke: A Systematic Review and Meta-Analysis. Front Neurol. 2022;13:910259.
Fiehler J, et al. ERASER: A Thrombectomy Study With Predictive Analytics End Point. Stroke. 2019;50(5):1275–8.
Nielsen A, Hansen MB, Tietze A, Mouridsen K. Prediction of Tissue Outcome and Assessment of Treatment Effect in Acute Ischemic Stroke Using Deep Learning. Stroke. 2018;49:1394–401.
Winder A, Wilms M, Fiehler J, Forkert ND. Treatment Efficacy Analysis in Acute Ischemic Stroke Patients Using In Silico Modeling Based on Machine Learning: A Proof-of-Principle. Biomedicines. 2021;9(10):1357.
Bücke P, et al. What You Always Wanted to Know about Endovascular Therapy in Acute Ischemic Stroke but Never Dared to Ask: A Comprehensive Review. Rev Cardiovasc Med. 2022;23(10):340.
Ovbiagele B, Kidwell CS, Starkman S, Saver JL. Neuroprotective agents for the treatment of acute ischemic stroke. Curr Neurol Neurosci Rep. 2003;3(1):9–20.
Litvinenko IV, et al. The algorithm of reperfusion treatment of the ischemic stroke: focus on DAWN and DEFUSE-3 trials. Arter Gipertenz. 2021;27(1):29–40.
Geris L, Rousseau CF, Noailly J, Afshari P, Auffret M, Chu W‑Y. The Role of Artificial Intelligence Within In Silico Medicine. Avicenna Alliance and VPH Institute. 2022. https://www.avicenna-alliance.com/upload/ai-in-health-white-paper-aug-2022_64bfda2bddc52.pdf. Accessed 01. Sep. 2023.
Konduri PR, Marquering HA, Van Bavel EE, Hoekstra A, Majoie CBLM, The INSIST Investigators. In-Silico Trials for Treatment of Acute Ischemic Stroke. Front Neurol.. 11:558125. 2020.
Miller C, et al. In silico trials for treatment of acute ischemic stroke: Design and implementation. Comput Biol Med. 2021;137:104802.
Benzakoun J, et al. Tissue outcome prediction in hyperacute ischemic stroke: Comparison of machine learning models. J Cereb Blood Flow Metab. 2021;41(11):3085–96.
Stier N, Vincent N, Liebeskind D, Scalzo F. Deep learning of tissue fate features in acute ischemic stroke. In, Vol. 2015. International Conference on Bioinformatics and Biomedicine (BIBM). IEEE: IEEE; 2015. pp. 1316–21.
Rajput D, Wang W‑J, Chen C‑C. Evaluation of a decided sample size in machine learning applications. Bmc Bioinformatics. 2023;24:48.
Forkert ND, Fiehler J. Effect of sample size on multi-parametric prediction of tissue outcome in acute ischemic stroke using a random forest classifier.
Carter RE, Attia ZI, Lopez-Jimenez F, Friedman PA. Pragmatic considerations for fostering reproducible research in artificial intelligence. Npj Digit Med. 2019;2(1):42.
Ni H, et al. Asymmetry Disentanglement Network for Interpretable Acute Ischemic Stroke Infarct Segmentation in Non-contrast CT Scans. In: Wang L, Dou Q, Fletcher PT, Speidel S, Li S, editors. Image Computing and Computer Assisted Intervention—MICCAI. Switzerland, Vol. 2022. Cham: Springer Nature: Medical; 2022. pp. 416–26.
Kao P‑Y, Chen JW, Manjunath BS. Predicting Clinical Outcome of Stroke Patients with Tractographic Feature. In: Crimi A, Bakas S, editors. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Cham: Springer; 2020. pp. 32–43.
Kemmling A, et al. Multivariate Dynamic Prediction of Ischemic Infarction and Tissue Salvage as a Function of Time and Degree of Recanalization. J Cereb Blood Flow Metab. 2015;35(9):1397–405.
Yu Y, et al. Use of Deep Learning to Predict Final Ischemic Stroke Lesions From Initial Magnetic Resonance Imaging. Jama Netw Open. 2020;3(3):e200772.
Debs N, et al. Impact of the reperfusion status for predicting the final stroke infarct using deep learning. Neuroimage: Clin. 2021;29:102548.
Livne M, Boldsen JK, Mikkelsen IK, Fiebach JB, Sobesky J, Mouridsen K. Boosted Tree Model Reforms Multimodal Magnetic Resonance Imaging Infarct Prediction in Acute Stroke. Stroke. 2018;49(4):912–8.
Dubin JR, Simon SD, Norrell K, Perera J, Gowen J, Cil A. Risk of Recall Among Medical Devices Undergoing US Food and Drug Administration 510(k) Clearance and Premarket Approval, 2008-2017. JAMA Netw Open. 2021;4(5):e217274.
Ardaugh BM, Graves SE, Redberg RF. The 510(k) Ancestry of a Metal-on-Metal Hip Implant. N Engl J Med. 2013;368(2):97–100.
Developing a Software Precertification Program: A Working Model. FDA. 2019. https://www.fda.gov/media/119722/download. Accessed 01 Sep 2023.
Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device: Discussion Paper and Request for Feedback. FDA. 2019. https://www.fda.gov/media/122535/download. Accessed 01 Sep 2023.
Furlan NE, et al. The Impact of Age on Mortality and Disability in Patients With Ischemic Stroke Who Underwent Cerebral Reperfusion Therapy: A Brazilian Cohort Study. Front Aging Neurosci. 2021;13:649902.
Zhang Y, Liu S, Li C, Wang J. Application of Deep Learning Method on Ischemic Stroke Lesion. Segmentation J Shanghai Jiaotong Univ (sci). 2022;27(1):99–111.
Nielsen C, Tuladhar A, Forkert ND. Investigating the Vulnerability of Federated Learning-Based Diabetic Retinopathy Grade Classification to Gradient Inversion Attacks. In: Antony B, Fu H, Lee CS, MacGillivray T, Xu Y, Zheng Y, editors. Ophthalmic Medical Image Analysis. Cham: Springer; 2022. pp. 183–92.
Stanley EAM, Wilms M, Mouches P, Forkert ND. Fairness-related performance and explainability effects in deep learning models for brain image analysis. J Med Imag. 2022;9:6.
Benjamens S, Dhunnoo P, Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. Npj Digit Med. 2020;3(1):118.
AI Central. ACR Data Science Institute. 2023. https://aicentral.acrdsi.org/. Accessed 01 Aug 2023.
FDA-approved A.I.-based algorithms. The Medical Futurist. 2021. https://medicalfuturist.com/fda-approved-ai-based-algorithms/. Accessed 01 Aug 2023.
Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021;27(4):582–4.
Lehman CD, Wellman RD, Buist DSM, Kerlikowske K, Tosteson ANA, Miglioretti DL. Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection. JAMA Intern Med. 2015;175(11):1828.
Salim M, et al. External Evaluation of 3 Commercial Artificial Intelligence Algorithms for Independent Assessment of Screening Mammograms. JAMA Oncol. 2020;6(10):1581.
Lee JH, et al. Improving the Performance of Radiologists Using Artificial Intelligence-Based Detection Support Software for Mammography: A Multi-Reader Study. Korean J Radiol. 2022;23(5):505.
Funding
This work was supported by the Canada Research Chairs program as well as by the River Fund at Calgary Foundation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
J. Fiehler is, by appointment of the university hospital, the CEO of Eppdata GmbH and has received fees as consultant or lecturer from Acandis, Bayer, Boehringer-Ingelheim, Codman, Covidien, MicroVention, Penumbra, Philips, Sequent, Siemens and Stryker. Consultant for Codman, MicroVention. Lectures for Boehringer-Ingelheim, Covidien, and Penumbra. Funding to institution: MicroVention. Unrelated: consultant for Acandis, Sequent, Stryker. A.J. Winder, E.A. Stanley and N.D. Forkert declare that they have no competing interests.
Ethical standards
For this article no studies with human participants or animals were performed by any of the authors. All studies mentioned were in accordance with the ethical standards indicated in each case.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Winder, A.J., Stanley, E.A., Fiehler, J. et al. Challenges and Potential of Artificial Intelligence in Neuroradiology. Clin Neuroradiol 34, 293–305 (2024). https://doi.org/10.1007/s00062-024-01382-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00062-024-01382-7