Skip to main content
SpringerLink
Log in

Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI

  • Chapter
  • First Online:
Clinical Applications of Artificial Intelligence in Real-World Data

Abstract

Research in medical artificial intelligence (AI) is experiencing an explosive growth. This growth highlights the potential of AI to significantly improve healthcare across a wide spectrum of applications such as risk stratification, diagnosis, therapeutics, and resource management among others. However, despite the great promises of medical AI and recent technological advancements, a gap persists in translating and deploying AI solutions within clinical settings. This gap is attributed to the risks and challenges that these promising technologies entail. To bring AI one step closer to the real-word clinical practice, we identify and outline the principal clinical, ethical and socio-ethical risks associated with AI in healthcare, unravelling their potential sources. These risks include potential errors leading to patient harm, risk of bias causing exacerbated health disparities, lack of transparency and trust, as well as susceptibilities to hacking and data privacy breaches. Furthermore, we discuss approaches towards minimizing risks and developing tools that can be safely deployed and routinely used in the clinic. Moreover, we introduce a set of concrete recommendations aimed at mitigating risks and maximizing the advantages presented by medical AI. These recommendations include fostering multi-stakeholder engagement throughout the AI production lifecycle, increased transparency and traceability, exhaustive clinical validation of AI tools, and comprehensive AI training and education for both medical practitioners and the general public. The adoption of such policies stands to significantly influence the trajectory and deployment of AI within clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi E, Segars WP, Tsui BMW, Kinahan PE, Bottenus N, Frangi AF, Maidment A, Lo J, Samei E. Virtual clinical trials in medical imaging: a review. J Med Imaging (Bellingham, Wash.). 2020;7(4):042805. https://doi.org/10.1117/1.JMI.7.4.042805

  2. Adamson AS, Smith A. Machine learning and health care disparities in dermatology. JAMA Dermatol. 2018;154(11):1247–1248. http://dx.doi.org/10.1001/jamadermatol.2018.2348.

  3. Alami H, Rivard L, Lehoux P, Hoffman SJ, Cadeddu SBM, Savoldelli M, Samri MA, Ag Ahmed MA, Fleet R, Fortin J-P. Artificial intelligence in health care: laying the Foundation for responsible, sustainable, and inclusive innovation in low- and middle-income countries. Glob Health. 2020;16(1):52. https://doi.org/10.1186/s12992-020-00584-1.

    Article  Google Scholar 

  4. Allen M, Pearn K, Monks T, Bray BD, Everson R, Salmon A, James M, Stein K. Can clinical audits be enhanced by pathway simulation and machine learning? An example from the acute stroke pathway. BMJ Open. 2019;9(9): e028296. https://doi.org/10.1136/bmjopen-2018-028296.

    Article  Google Scholar 

  5. Aminololama-Shakeri S, López JE. The doctor-patient relationship with artificial intelligence. AJR Am J Roentgenol. 2019;212(2):308–10. https://doi.org/10.2214/AJR.18.20509.

    Article  Google Scholar 

  6. Arora, A. Conceptualising artificial intelligence as a digital healthcare innovation: an introductory review. Med Devices (Auckland, N.Z.). 2020;13:223–30. https://doi.org/10.2147/mder.s262590

  7. Barda AJ, Horvat CM, Hochheiser H. A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare. BMC Med Inform Decis Mak. 2020;20(1):257. https://doi.org/10.1186/s12911-020-01276-x.

    Article  Google Scholar 

  8. Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nat Mach Intell. 2021;3(1):25–7. https://doi.org/10.1038/s42256-020-00254-2.

    Article  Google Scholar 

  9. Barocas S, Hardt M, Narayanan A. Fairness in machine learning’ Nips tutorial, vol. 1. 2017. p. 2. https://fairmlbook.org

  10. Bates DW, Levine D, Syrowatka A, Kuznetsova M, Craig KJT, Rui A, Jackson GP, Rhee K. The potential of artificial intelligence to improve patient safety: a scoping review. Npj Digit Med. 2021;4(1):54. https://doi.org/10.1038/s41746-021-00423-6.

    Article  Google Scholar 

  11. Bellamy RKE, Dey K, Hind M, Hoffman SC, Houde S, Kannan K, Lohia P, Martino J, Mehta S, Mojsilovic A, Nagar S, Ramamurthy KN, Richards J, Saha D, Sattigeri P, Singh, M, Varshney, KR, Zhang Y. AI fairness 360: an extensible toolkit for detecting and mitigating algorithmic bias. IBM J Res Dev. 2019;63(4/5):4:1–4:15. https://doi.org/10.1147/jrd.2019.2942287

  12. Bird S, Dudík M, Edgar R, Horn B, Lutz R, Milan V, Sameki M, Wallach H, Walker K, Design A. Fairlearn: a toolkit for assessing and improving fairness in AI. Microsoft.com. 2020. https://www.microsoft.com/en-us/research/uploads/prod/2020/05/Fairlearn_WhitePaper-2020-09-22.pdf

  13. Brooke J. SUS: A “quick and dirty” usability scale. In: Usability evaluation in industry 1st ed. CRC Press; 1996. p. 207–12.

    Google Scholar 

  14. Campello VM, Gkontra P, Izquierdo C, Martin-Isla C, Sojoudi A, Full PM, Maier-Hein K, Zhang Y, He Z, Ma J, Parreno M, Albiol A, Kong F, Shadden SC, Acero JC, Sundaresan V, Saber M, Elattar M, Li H, Lekadir K. Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M&Ms challenge. IEEE Trans Med Imaging. 2021;40(12):3543–54. https://doi.org/10.1109/TMI.2021.3090082.

    Article  Google Scholar 

  15. Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Atanasova K. Artificial intelligence, bias and clinical safety. BMJ Qual Saf. 2019;28(3):231–7. https://doi.org/10.1136/bmjqs-2018-008370.

    Article  Google Scholar 

  16. Ching T, Himmelstein DS, Beaulieu-Jones BK, Kalinin AA, Do BT, Way GP, Ferrero E, Agapow P-M, Zietz M, Hoffman MM, Xie W, Rosen GL, Lengerich BJ, Israeli J, Lanchantin J, Woloszynek S, Carpenter AE, Shrikumar A, Xu J, Greene CS. Opportunities and obstacles for deep learning in biology and medicine. J R Soc Interface. 2018;15(141):20170387. https://doi.org/10.1098/rsif.2017.0387.

    Article  Google Scholar 

  17. Chinzei K, Shimizu A, Mori K, Harada K, Takeda H, Hashizume M, Ishizuka M, Kato, N, Kawamori R, Kyo S, Nagata K, Yamane T, Sakuma I, Ohe K, Mitsuishi M. Regulatory science on AI-based medical devices and systems. Adv Biomed Eng 2018;7(0):118–23. https://doi.org/10.14326/abe.7.118

  18. Cohen IG. Informed consent and medical artificial intelligence: What to tell the patient? SSRN Electron J. 2020. https://doi.org/10.2139/ssrn.3529576.

    Article  Google Scholar 

  19. Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, Logullo P, Beam AL, Peng L, Van Calster B, van Smeden M, Riley RD, Moons KG. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021;11(7): e048008. https://doi.org/10.1136/bmjopen-2020-048008.

    Article  Google Scholar 

  20. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 2015;13(1):1. https://doi.org/10.1186/s12916-014-0241-z.

    Article  Google Scholar 

  21. Din NU, Ukoumunne OC, Rubin G, Hamilton W, Carter B, Stapley S, Neal RD. Age and gender variations in cancer diagnostic intervals in 15 cancers: Analysis of data from the UK clinical practice research datalink. PLOS One. 2015;10(5):e0127717. http://dx.doi.org/10.1371/journal.pone.0127717.

  22. Domalpally A, Channa R. Real-world validation of artificial intelligence algorithms for ophthalmic imaging. Lancet Digit Health. 2021;3(8):e463–4. https://doi.org/10.1016/S2589-7500(21)00140-0.

    Article  Google Scholar 

  23. Doyen S, Dadario NB. 12 plagues of AI in healthcare: a practical guide to current issues with using machine learning in a medical context. Frontiers in Digital Health. 2022;4: 765406. https://doi.org/10.3389/fdgth.2022.765406.

    Article  Google Scholar 

  24. Ellahham S, Ellahham N, Simsekler MCE. Application of artificial intelligence in the health care safety context: opportunities and challenges. Am J Med Qual: Off J Am Coll Med Qual. 2020;35(4):341–8. https://doi.org/10.1177/1062860619878515.

    Article  Google Scholar 

  25. European Commission, Directorate-General for Communications Networks, Content and Technology. The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office. 2020. https://doi.org/10.2759/002360

  26. European Commission. Proposal for a Regulation laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union legislative acts. (n.d.). Europeansources.Info. 2021. Available 4 Aug 2022 from https://www.europeansources.info/record/proposal-for-a-regulation-laying-down-harmonised-rules-on-artificial-intelligence-artificial-intelligence-act-and-amending-certain-union-legislative-acts/

  27. European Health Data Space. Public health. n.d. Available 4 Aug 2022, from https://health.ec.europa.eu/ehealth-digital-health-and-care/european-health-data-space_en

  28. De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, Askham H, Glorot X, O’Donoghue B, Visentin D, van den Driessche G, Lakshminarayanan B, Meyer C, Mackinder F, Bouton S, Ayoub K, Chopra R, King D, Karthikesalingam A, Ronneberger O. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24(9):1342–50. https://doi.org/10.1038/s41591-018-0107-6.

    Article  Google Scholar 

  29. Ferryman K, Pitcan M. Fairness in precision medicine. 2018. https://datasociety.net/library/fairness-in-precision-medicine/

  30. Fihn SD, Saria S, Mendonça E, Hain S, Matheny M, Shah N, Liu H, Auerbach A. Deploying AI in clinical settings. In artificial intelligence in health care: the hope, the hype, the promise, the peril. In: Matheny M, Israni ST, Ahmed M, Whicher D, editors. Washington, DC: National Academy of Medicine; 2019.

    Google Scholar 

  31. Filice RW, Ratwani RM. The case for user-centered artificial intelligence in radiology. Radiology Artificial Intelligence. 2020;2(3): e190095. https://doi.org/10.1148/ryai.2020190095.

    Article  Google Scholar 

  32. Finlayson SG, Bowers JD, Ito J, Zittrain JL, Beam AL, Kohane IS. Adversarial attacks on medical machine learning. Science (New York, N.Y.). 2019;363(6433):1287–89. https://doi.org/10.1126/science.aaw4399

  33. Floyd BJ. Problems in accurate medical diagnosis of depression in female patients. Social Sci Medicine. 1997;44(3):403–412. http://dx.doi.org/10.1016/s0277-9536(96)00159-1.

  34. Freeman K, Dinnes J, Chuchu N, Takwoingi Y, Bayliss SE, Matin RN, Jain A, Walter FM, Williams HC, Deeks JJ. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. BMJ (Clin Res Ed). 2020;368: m127. https://doi.org/10.1136/bmj.m127.

    Article  Google Scholar 

  35. Gerke S, Minssen T, Cohen G. Ethical and legal challenges of artificial intelligence-driven healthcare. In: Artificial intelligence in healthcare. Elsevier; 2020. p. 295–36.

    Google Scholar 

  36. German Data Ethics Commission, Opinion of the Data Ethics Commission. 2019. https://www.bmi.bund.de/SharedDocs/downloads/EN/themen/it-digital-policy/datenethikkommission-abschlussgutachten-lang.pdf?__blob=publicationFile&v=4

  37. Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3(11):e745–50. https://doi.org/10.1016/S2589-7500(21)00208-9.

    Article  Google Scholar 

  38. Gillespie N, Lockey S, Curtis C. Trust in artificial Intelligence: a five country study. The University of Queensland and KPMG; 2021. https://doi.org/10.14264/e34bfa3

  39. Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology. 2016;278(2):563–77. https://doi.org/10.1148/radiol.2015151169.

    Article  Google Scholar 

  40. Gomez Rossi J, Rojas-Perilla N, Krois J, Schwendicke F. Cost-effectiveness of artificial intelligence as a decision-support system applied to the detection and grading of melanoma, dental caries, and diabetic retinopathy. JAMA Netw Open. 2022;5(3): e220269. https://doi.org/10.1001/jamanetworkopen.2022.0269.

    Article  Google Scholar 

  41. Guo J, Li B. The application of medical artificial intelligence technology in rural areas of developing countries. Health Equity. 2018;2(1):174–81. https://doi.org/10.1089/heq.2018.0037.

    Article  MathSciNet  Google Scholar 

  42. Guo J, Li B The application of medical artificial intelligence technology in rural areas of developing countries. In: Health equity, vol. 2, Issue 1. Mary Ann Liebert Inc.; 2018. p. 174–81. https://doi.org/10.1089/heq.2018.0037

  43. Haibe-Kains B, Adam GA, Hosny A, Khodakarami F, Massive Analysis Quality Control (MAQC) Society Board of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38 Mason Christopher E. 39 Jones Wendell 40 Dopazo Joaquin 41 Furlanello Cesare 42, Waldron L, Wang B, McIntosh C, Goldenberg A, Kundaje A, Greene CS, Broderick T, Hoffman M. M, Leek JT, Korthauer K, Huber W, Brazma A, Pineau J, Tibshirani R, Hastie T, Ioannidis JPA, Quackenbush J, Aerts HJWL. Transparency and reproducibility in artificial intelligence. Nature. 2020;586(7829):E14–E16. https://doi.org/10.1038/s41586-020-2766-y.

  44. Harned Z, Lungren MP, Rajpurkar P. Machine vision, medical AI, and malpractice. Compar Polit Econ: Regul eJ. 2019. https://jolt.law.harvard.edu/digest/machine-vision-medical-ai-and-malpractice

  45. Harvey HB, Gowda V. How the FDA regulates AI. Acad Radiology. 2020;27(1):58–61. http://dx.doi.org/10.1016/j.acra.2019.09.017.

  46. He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nat Med. 2019;25(1):30–6. https://doi.org/10.1038/s41591-018-0307-0.

    Article  Google Scholar 

  47. Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Inf Assoc: JAMIA. 2020;27(12):2011–5. https://doi.org/10.1093/jamia/ocaa088.

    Article  Google Scholar 

  48. Hill NR, Sandler B, Mokgokong R, Lister S, Ward T, Boyce R, Farooqui U, Gordon J. Cost-effectiveness of targeted screening for the identification of patients with atrial fibrillation: evaluation of a machine learning risk prediction algorithm. J Med Econ. 2020;23(4):386–93. https://doi.org/10.1080/13696998.2019.1706543.

    Article  Google Scholar 

  49. Hocking L, Parks S, Altenhofer M, Gunashekar S. Reuse of health data by the European pharmaceutical industry: current practice and implications for the future. RAND Corporation. 2019. https://doi.org/10.7249/RR3247.

    Article  Google Scholar 

  50. Hoffman KM, Trawalter S, Axt JR, Oliver MN. Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites. Proc Natl Acad Sci USA. 2016;113(16):4296–301. https://doi.org/10.1073/pnas.1516047113.

    Article  Google Scholar 

  51. Johnson B, Bartola J, Angell R, Keith K, Witty S, Giguere SJ, Brun Y. Fairkit, fairkit, on the wall, who’s the fairest of them all? Supporting data scientists in training fair models. 2020. https://doi.org/10.48550/ARXIV.2012.09951

  52. Karimi D, Dou H, Warfield SK, Gholipour A. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Med Image Anal. 2020;65(101759): 101759. https://doi.org/10.1016/j.media.2020.101759.

    Article  Google Scholar 

  53. Kaushal A, Altman R, Langlotz C. Geographic distribution of US cohorts used to train deep learning algorithms. JAMA: J Am Med Assoc 2020;324(12):1212–13. https://doi.org/10.1001/jama.2020.12067

  54. Kiener M. ‘“You may be hacked” and other things doctors should tell you’. The Conversation. 3 November 2020. https://theconversation.com/you-may-be-hacked-and-other-things-doctors-should-tell-you-148946

  55. Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: Results from recently published papers. Korean J Radioly: Off J Korean Radiol Soc. 2019;20(3):405–10. https://doi.org/10.3348/kjr.2019.0025.

    Article  Google Scholar 

  56. Klonoff DC. Cybersecurity for connected diabetes devices. J Diabetes Sci Technol. 2015;9(5):1143–7. https://doi.org/10.1177/1932296815583334.

    Article  Google Scholar 

  57. Koene A, Clifton C, Hatada Y, Webb H, Richardson R. A governance framework for algorithmic accountability and transparency. EPRS, European Parliament; 2019. https://www.europarl.europa.eu/thinktank/en/document/EPRS_STU(2019)624262

  58. Kompa B, Snoek J, Beam AL. Second opinion needed: communicating uncertainty in medical machine learning. Npj Digit Med. 2021;4(1):4. https://doi.org/10.1038/s41746-020-00367-3.

    Article  Google Scholar 

  59. Koops B-J. The concept of function creep. Law Innov Technol. 2021;13(1):29–56. https://doi.org/10.1080/17579961.2021.1898299.

    Article  Google Scholar 

  60. Langlotz CP, Allen B, Erickson BJ, Kalpathy-Cramer J, Bigelow K, Cook TS, Flanders AE, Lungren MP, Mendelson DS, Rudie JD, Wang G, Kandarpa K. A roadmap for foundational research on artificial intelligence in medical imaging: From the 2018 NIH/RSNA/ACR/the academy workshop. Radiology. 2019;291(3):781–91. https://doi.org/10.1148/radiol.2019190613.

    Article  Google Scholar 

  61. Larson DB, Harvey H, Rubin DL, Irani N, Tse JR, Langlotz CP. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: summary and recommendations. J Am Coll Radiol: JACR. 2021;18(3 Pt A):413–24. https://doi.org/10.1016/j.jacr.2020.09.060

  62. Lehne M, Sass J, Essenwanger A, Schepers J, Thun S. Why digital medicine depends on interoperability. Npj Digit Med. 2019;2(1):79. https://doi.org/10.1038/s41746-019-0158-1.

    Article  Google Scholar 

  63. Lekadir K et al. ‘FUTURE-AI: best practices for trustworthy AI in medicine’. 2022. www.future-ai.org

  64. Lekadir K, Quaglio G, Tselioudis Garmendia A, Gallin C. Artificial intelligence in healthcare: applications, risks, and ethical and societal impacts. (n.d.). Europa.Eu.; 2022. https://www.europarl.europa.eu/thinktank/en/document/EPRS_STU(2022)729512

  65. Leone D, Schiavone F, Appio FP, Chiao B. How does artificial intelligence enable and enhance value co-creation in industrial markets? An exploratory case study in the healthcare ecosystem. J Bus Res. 2021;129:849–59. https://doi.org/10.1016/j.jbusres.2020.11.008.

    Article  Google Scholar 

  66. Leslie D, Mazumder A, Peppin A, Wolters MK, Hagerty A. Does “AI” stand for augmenting inequality in the era of covid-19 healthcare? BMJ (Clinical Research Ed). 2021;372: n304. https://doi.org/10.1136/bmj.n304.

    Article  Google Scholar 

  67. Lewis JR. The system usability scale: Past, present, and future. Int J Hum-Comput Interact. 2018;34(7):577–90. https://doi.org/10.1080/10447318.2018.1455307.

    Article  Google Scholar 

  68. Li Y, Vasconcelos N. REPAIR: Removing representation bias by dataset resampling. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR); 2019. https://doi.org/10.1109/CVPR.2019.00980

  69. Linardatos P, Papastefanopoulos V, Kotsiantis S. Explainable AI: a review of machine learning interpretability methods. Entropy (Basel, Switzerland). 2020;23(1):18. https://doi.org/10.3390/e23010018.

    Article  Google Scholar 

  70. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med 2020;26(9):1364–1374. https://doi.org/10.1038/s41591-020-1034-x

  71. Lyratzopoulos G, Abel GA, McPhail S, Neal RD, Rubin GP. Gender inequalities in the promptness of diagnosis of bladder and renal cancer after symptomatic presentation: evidence from secondary analysis of an English primary care audit survey. BMJ Open. 2013;3(6):e002861. http://dx.doi.org/10.1136/bmjopen-2013-002861.

  72. Mackey TK, Nayyar G. Digital danger: a review of the global public health, patient safety and cybersecurity threats posed by illicit online pharmacies. Br Med Bull. 2016;118(1):110–26. https://doi.org/10.1093/bmb/ldw016.

    Article  Google Scholar 

  73. Maliha G, Gerke S, Cohen IG, Parikh RB. Artificial intelligence and liability in medicine: balancing safety and innovation. Milbank Q. 2021;99(3):629–47. https://doi.org/10.1111/1468-0009.12504.

    Article  Google Scholar 

  74. Manne R, Kantheti SC. Application of artificial intelligence in healthcare: chances and challenges. Curr J Appl Sci Technol. 2021;40(6):78–89. https://doi.org/10.9734/cjast/2021/v40i631320.

    Article  Google Scholar 

  75. Matheny ME, Whicher D, Thadaney Israni S. Artificial intelligence in health care: A report from the national academy of medicine: a report from the national academy of medicine. JAMA: J Am Med Assoc 2020;323(6):509–10. https://doi.org/10.1001/jama.2019.21579

  76. McCarthy J, Hayes PJ. Some philosophical problems from the standpoint of artificial intelligence. In: Readings in artificial intelligence. Elsevier; 1981. p. 431–50.

    Google Scholar 

  77. McKeown A, Mourby M, Harrison P, Walker S, Sheehan M, Singh I. Ethical issues in consent for the reuse of data in health data platforms. Sci Eng Ethics. 2021;27(1):9. https://doi.org/10.1007/s11948-021-00282-0.

    Article  Google Scholar 

  78. McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, Back T, Chesus M, Corrado GS, Darzi A, Etemadi M, Garcia-Vicente F, Gilbert FJ, Halling-Brown M, Hassabis D, Jansen S, Karthikesalingam A, Kelly CJ, King D, Shetty S. International evaluation of an AI system for breast cancer screening. Nature. 2020;577(7788):89–94. https://doi.org/10.1038/s41586-019-1799-6.

    Article  Google Scholar 

  79. Meskó B, Görög M. A short guide for medical professionals in the era of artificial intelligence. Npj Digital Medicine. 2020;3(1):126. https://doi.org/10.1038/s41746-020-00333-z.

    Article  Google Scholar 

  80. Mikolajczyk A, Grochowski M. Data augmentation for improving deep learning in image classification problem. In: 2018 International interdisciplinary Ph.D. workshop (IIPhDW). IEEE; 2018. https://doi.org/10.1109/iiphdw.2018.8388338

  81. Mikolajczyk A, Grochowski M. Data augmentation for improving deep learning in image classification problem. In: 2018 International interdisciplinary Ph.D. workshop (IIPhDW); 2018. https://doi.org/10.1109/iiphdw.2018.8388338

  82. Mongan J, Moy L, Kahn CE Jr. Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers. Radiol Artif Intell. 2020;2(2): e200029. https://doi.org/10.1148/ryai.2020200029.

    Article  Google Scholar 

  83. Mora-Cantallops M, Sánchez-Alonso S, García-Barriocanal E, Sicilia M-A. Traceability for trustworthy AI: a review of models and tools. Big Data Cogn Comput. 2021;5(2):20. https://doi.org/10.3390/bdcc5020020.

    Article  Google Scholar 

  84. Morley J, Floridi L. An ethically mindful approach to AI for health care. Lancet. 2020;395(10220):254–5. https://doi.org/10.1016/S0140-6736(19)32975-7.

    Article  Google Scholar 

  85. Muehlematter UJ, Daniore P, Vokinger KN. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. Lancet Digit Health. 2021;3(3):e195–203. https://doi.org/10.1016/S2589-7500(20)30292-2.

    Article  Google Scholar 

  86. Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, Topol EJ, Ioannidis JPA, Collins GS, Maruthappu M. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ (Clin Res Ed). 2020;368: m689. https://doi.org/10.1136/bmj.m689.

    Article  Google Scholar 

  87. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science (New York, N.Y.). 2019;366(6464):447–453. https://doi.org/10.1126/science.aax2342

  88. Ota N, Tachibana K, Kusakabe T, Sanada S, Kondoh M. A concept for a Japanese regulatory framework for emerging medical devices with frequently modified behavior: a regulatory concept for innovation. Clin Transl Sci. 2020;13(5):877–9. https://doi.org/10.1111/cts.12784.

    Article  Google Scholar 

  89. Panch T, Mattie H, Atun R. Artificial intelligence and algorithmic bias: implications for health systems. J Glob Health. 2019;9(2): 010318. https://doi.org/10.7189/jogh.09.020318.

    Article  Google Scholar 

  90. Paranjape K, Schinkel M, Nannan Panday R, Car J, Nanayakkara P. Introducing artificial intelligence training in medical education. JMIR Med Educ. 2019;5(2): e16048. https://doi.org/10.2196/16048.

    Article  Google Scholar 

  91. Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018;286(3):800–9. https://doi.org/10.1148/radiol.2017171920.

    Article  MathSciNet  Google Scholar 

  92. Park Y, Jackson GP, Foreman MA, Gruen D, Hu J, Das AK. Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open. 2020;3(3):326–31. https://doi.org/10.1093/jamiaopen/ooaa033.

    Article  Google Scholar 

  93. Pinto A, Pinto F, Faggian A, Rubini G, Caranci F, Macarini L, Genovese EA, Brunese L (2013) Sources of error in emergency ultrasonography. Crit Ultrasound J 2013;5 Suppl 1(S1):S1. https://doi.org/10.1186/2036-7902-5-S1-S1

  94. Quaglio G, Pirona A, Esposito G, Karapiperis T, Brand H, Dom G, Bertinato L, Montanari L, Kiefer F, Carrà G. Knowledge and utilization of technology-based interventions for substance use disorders: an exploratory study among health professionals in the European Union. Drugs (Abingdon, England). 2018;26(5):437–46. https://doi.org/10.1080/09687637.2018.1475549.

    Article  Google Scholar 

  95. Raji ID, Smart A, White RN, Mitchell M, Gebru T, Hutchinson B, Smith-Loud J, Theron D, Barnes P. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In: Proceedings of the 2020 conference on fairness, accountability, and transparency; 2020. https://doi.org/10.1145/3351095.3372873

  96. Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med. 2018;169(12):866–72. https://doi.org/10.7326/M18-1990.

    Article  Google Scholar 

  97. Reddy S, Rogers W, Makinen V-P, Coiera E, Brown P, Wenzel M, Weicken E, Ansari S, Mathur P, Casey A, Kelly B. Evaluation framework to guide implementation of AI systems into healthcare settings. BMJ Health Care Inf 2021; 28(1). https://doi.org/10.1136/bmjhci-2021-100444

  98. Reisman D, Schultz J, Crawford K, Whittaker M. A practical framework for public agency accountability. Ainowinstitute.org. n.d. Available 3 Aug 2022, from https://ainowinstitute.org/aiareport2018.pdf

  99. Richardson B, Gilbert JE. A framework for fairness: A systematic review of existing fair AI solutions. 2021. arXiv [cs.AI]. http://arxiv.org/abs/2112.05700.

  100. Roberts H, Cowls J, Morley J, Taddeo M, Wang V, Floridi L. The Chinese approach to artificial intelligence: an analysis of policy, ethics, and regulation. AI Soc. 2021;36(1):59–77. https://doi.org/10.1007/s00146-020-00992-2.

    Article  Google Scholar 

  101. Samulowitz A, Gremyr I, Eriksson E, Hensing G. “brave men” and “emotional women”: A theory-guided literature review on gender bias in health care and gendered norms towards patients with chronic pain. Journal de La Societe Canadienne Pour Le Traitement de La Douleur (Pain Res Manag). 2018;2018:1–14. https://doi.org/10.1155/2018/6358624.

    Article  Google Scholar 

  102. Scheetz J, Rothschild P, McGuinness M, Hadoux X, Soyer HP, Janda M, Condon JJJ, Oakden-Rayner L, Palmer LJ, Keel S, van Wijngaarden P. A survey of clinicians on the use of artificial intelligence in ophthalmology, dermatology, radiology and radiation oncology. Sci Rep. 2021;11(1):5193. https://doi.org/10.1038/s41598-021-84698-5.

    Article  Google Scholar 

  103. Scott I, Carter S, Coiera E. Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Health Care Inf. 2021;28(1): e100251. https://doi.org/10.1136/bmjhci-2020-100251.

    Article  Google Scholar 

  104. Seyyed-Kalantari L, Liu G, McDermott M, Chen IY, Ghassemi M. CheXclusion: Fairness gaps in deep chest X-ray classifiers. Biocomputing 2021. 2020. https://doi.org/10.1142/9789811232701_0022

  105. Shin EK, Mahajan R, Akbilgic O, Shaban-Nejad A. Sociomarkers and biomarkers: predictive modeling in identifying pediatric asthma patients at risk of hospital revisits. Npj Digit Med. 2018;1(1):50. https://doi.org/10.1038/s41746-018-0056-y.

    Article  Google Scholar 

  106. Shortliffe EH, Sepúlveda MJ (2018) Clinical decision support in the era of artificial intelligence. JAMA: J Am Med Assoc 2018;320(21):2199–200. https://doi.org/10.1001/jama.2018.17163

  107. Sipola T, Kokkonen T. One-pixel attacks against medical imaging: a conceptual framework. In: Advances in intelligent systems and computing. Springer; 2021. p. 197–03. https://doi.org/10.1007/978-3-030-72657-7_19

  108. Sit C, Srinivasan R, Amlani A, Muthuswamy K, Azam A, Monzon L, Poon DS. Attitudes and perceptions of UK medical students towards artificial intelligence and radiology: a multicentre survey. Insights into Imaging. 2020;11(1). https://doi.org/10.1186/s13244-019-0830-7.

  109. Smith H. Clinical AI: opacity, accountability, responsibility and liability. AI Society. 2021;36(2):535–545. http://dx.doi.org/10.1007/s00146-020-01019-6.

  110. Srinivasan H. ML-fairness-gym: A tool for exploring long-term impacts of machine learning systems. Googleblog.com. 2020. https://ai.googleblog.com/2020/02/ml-fairness-gym-tool-for-exploring-long.html.

  111. Stylianou N, Fackrell R, Vasilakis C. Are medical outliers associated with worse patient outcomes? A retrospective study within a regional NHS hospital using routine data. BMJ Open. 2017;7(5): e015676. https://doi.org/10.1136/bmjopen-2016-015676.

    Article  Google Scholar 

  112. Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics (Oxford, England). 2020;21(2):345–52. https://doi.org/10.1093/biostatistics/kxz041.

    Article  MathSciNet  Google Scholar 

  113. Tanguay-Sela M, Benrimoh D, Perlman K, Israel S, Mehltretter J, Armstrong C, Fratila R, Parikh S, Karp J, Heller K, Vahia I, Blumberger D, Karama S, Vigod S, Myhr G, Martins R, Rollins C, Popescu C, Lundrigan E, Margolese H. Evaluating the usability and impact of an artificial intelligence-powered clinical decision support system for depression treatment. Biol Psychiat. 2020;87(9):S171. https://doi.org/10.1016/j.biopsych.2020.02.451.

    Article  Google Scholar 

  114. Tulk Jesso S, Kelliher A, Sanghavi H, Martin T, Henrickson Parker S. Inclusion of clinicians in the development and evaluation of clinical artificial intelligence tools: A systematic literature review. Front Psychol. 2022;13: 830345. https://doi.org/10.3389/fpsyg.2022.830345.

    Article  Google Scholar 

  115. Tutt A. An FDA for algorithms. SSRN Electron J. 2016. https://doi.org/10.2139/ssrn.2747994.

    Article  Google Scholar 

  116. U.S. Food and Drug Administration (FDA). Artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD) action plan; 2021.

    Google Scholar 

  117. Viceconti M, Pappalardo F, Rodriguez B, Horner M, Bischoff J, Musuamba Tshinanu F. In silico trials: verification, validation and uncertainty quantification of predictive models used in the regulatory evaluation of biomedical products. Methods (San Diego, Calif.). 2021;185:120–27. https://doi.org/10.1016/j.ymeth.2020.01.011

  118. Vokinger KN, Feuerriegel S, Kesselheim AS. Continual learning in medical devices: FDA’s action plan and beyond. Lancet Digit Health. 2021;3(6):e337–8. https://doi.org/10.1016/S2589-7500(21)00076-5.

    Article  Google Scholar 

  119. Vokinger KN, Feuerriegel S, Kesselheim AS. Mitigating bias in machine learning for medicine. Commun Med. 2021;1(1):25. https://doi.org/10.1038/s43856-021-00028-w.

    Article  Google Scholar 

  120. Wager TD, Woo C-W. Imaging biomarkers and biotypes for depression. Nat Med. 2017;23(1):16–7. https://doi.org/10.1038/nm.4264.

    Article  Google Scholar 

  121. Westergaard D, Moseley P, Sørup FKH, Baldi P, Brunak S. Population-wide analysis of differences in disease progression patterns in men and women. Nat Commun. 2019;10(1):666. https://doi.org/10.1038/s41467-019-08475-9.

    Article  Google Scholar 

  122. Whitby B. Automating medicine the ethical way. In: Machine medical ethics. Springer; 2015. p. 223–32.

    Google Scholar 

  123. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S, PROBAST Group†. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51–58. https://doi.org/10.7326/M18-1376

  124. World Health Organization (WHO). Ethics and governance of artificial intelligence for health: WHO guidance. 2021. https://www.who.int/publications/i/item/9789240029200

  125. Xivuri K, Twinomurinzi H. A systematic review of fairness in artificial intelligence algorithms. In: Responsible AI and analytics for an ethical and inclusive digitized society. Springer; 2021. p. 271–284. https://doi.org/10.1007/978-3-030-85447-8_24

  126. Xu W. Toward human-centered AI: a perspective from human-computer interaction. Interactions. 2019;26(4):42–6. https://doi.org/10.1145/3328485.

    Article  Google Scholar 

  127. Xu H, Ma Y, Liu H-C, Deb D, Liu H, Tang J-L, Jain AK. Adversarial attacks and defenses in images, graphs and text: a review. Int J Autom Comput. 2020;17(2):151–78. https://doi.org/10.1007/s11633-019-1211-x.

    Article  Google Scholar 

  128. Yan L, Zhang H-T, Goncalves J, Xiao Y, Wang M, Guo Y, Sun C, Tang X, Jing L, Zhang M, Huang X, Xiao Y, Cao H, Chen Y, Ren T, Wang F, Xiao Y, Huang S, Tan X, Yuan Y. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell. 2020;2(5):283–8. https://doi.org/10.1038/s42256-020-0180-7.

    Article  Google Scholar 

  129. Yu K-H, Kohane IS. Framing the challenges of artificial intelligence in medicine. BMJ Qual Saf. 2019;28(3):238–41. https://doi.org/10.1136/bmjqs-2018-008551.

    Article  Google Scholar 

  130. Yu AC, Mohajer B, Eng J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol Artif Intell. 2022;4(3): e210064. https://doi.org/10.1148/ryai.210064.

    Article  Google Scholar 

  131. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11): e1002683. https://doi.org/10.1371/journal.pmed.1002683.

    Article  Google Scholar 

  132. Zhang BH, Lemoine B, Mitchell M. Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM conference on AI, ethics, and society; 2018.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Artificial Intelligence in Medicine Lab (BCN-AIM), Barcelona, Spain

    Polyxeni Gkontra & Karim Lekadir

  2. Panel for the Future of Science and Technology (STOA), European Parliament, Brussels, Belgium

    Gianluca Quaglio

  3. Faculty of Medicine, School of Public Health, Imperial College, London, UK

    Anna Tselioudis Garmendia

Authors
  1. Polyxeni Gkontra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gianluca Quaglio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anna Tselioudis Garmendia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Karim Lekadir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Polyxeni Gkontra .

Editor information

Editors and Affiliations

  1. Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands

    Folkert W. Asselbergs

  2. Institute of Health Informatics, University College London, London, UK

    Spiros Denaxas

  3. Department of Data Science and Biostatistics, University Medical Center Utrecht (UMCU), Utrecht, The Netherlands

    Daniel L. Oberski

  4. Cedars-Sinai Medical Center, Los Angeles, CA, USA

    Jason H. Moore

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gkontra, P., Quaglio, G., Garmendia, A.T., Lekadir, K. (2023). Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI. In: Asselbergs, F.W., Denaxas, S., Oberski, D.L., Moore, J.H. (eds) Clinical Applications of Artificial Intelligence in Real-World Data. Springer, Cham. https://doi.org/10.1007/978-3-031-36678-9_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36678-9_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36677-2

  • Online ISBN: 978-3-031-36678-9

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation