Skip to main content
SpringerLink
Log in

Machine Learning in Medical Imaging – Clinical Applications and Challenges in Computer Vision

  • Chapter
  • First Online:
Artificial Intelligence in Medicine

Abstract

Applications for Machine Learning in Healthcare have rapidly increased in recent years. In particular, the analysis of images using machine learning and computer vision is one of the most important domains in the area. The idea that machines can outperform the human eye in recognizing subtle patterns is not new, but it is now gaining momentum with large financial investments and the arrival of many startups with a focus in this area. Several examples illustrate that machine learning has enabled us to detect more diffuse patterns that are difficult to detect by non-experts. This chapter provides a state-of-the-art review of machine learning and computer vision in medical image analysis. We start with a brief introduction to computer vision and an overview of deep learning architectures. We proceed to highlight relevant progress in clinical development and translation across various medical specialties of dermatology, pathology, ophthalmology, radiology, and cardiology, focusing on the domains of computer vision and machine learning. Furthermore, we introduce some of the challenges that the disciplines of computer vision and machine learning face within a traditional regulatory environment. This chapter highlights the developments of computer vision and machine learning in medicine by displaying a breadth of powerful examples that give the reader an understanding of the potential impact and challenges that computer vision and machine learning can play in the clinical environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bercovich E, Javitt MC (2018) Medical imaging: from roentgen to the digital revolution, and beyond. Rambam Maimonides Med J 9. https://doi.org/10.5041/RMMJ.10355

  2. Ivakhnenko AG (1971) Polynomial theory of complex systems. IEEE Trans Syst Man Cybern SMC-1:364–378. https://doi.org/10.1109/TSMC.1971.4308320

    Article  Google Scholar 

  3. Fukushima K (1980) Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36:193–202. https://doi.org/10.1007/BF00344251

    Article  Google Scholar 

  4. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  5. LeCun Y et al (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1:541–551. https://doi.org/10.1162/neco.1989.1.4.541

    Article  Google Scholar 

  6. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60:84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  7. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  Google Scholar 

  8. Nair V, Hinton G (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning, pp 807–814

    Google Scholar 

  9. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the Inception architecture for computer vision. In: Proceedings of 2016 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp 2818–2826. https://doi.org/10.1109/CVPR.2016.308

    Chapter  Google Scholar 

  10. Szegedy C et al (2015) Going deeper with convolutions. In: Proceedings of 2015 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

    Chapter  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp 770, 2016–778. https://doi.org/10.1109/CVPR.2016.90

  12. Net I (2014) Large scale visual recognition challenge 2014 (ILSVRC2014). http://www.image-net.org/challenges/LSVRC/2014/

  13. Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2016) Inception-v4, inception-ResNet and the impact of residual connections on learning. arXiv arXiv:1602.07261

    Google Scholar 

  14. COCO (2015) Common objects in context. https://cocodataset.org/#home

  15. Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. arXiv arXiv:1804.02767

    Google Scholar 

  16. Liu W et al (2016) SSD: single shot MultiBox detector. arXiv arXiv:1512.02325

    Google Scholar 

  17. Lin T, Goyal P, Girshick R, Kaiming H, Dollar P (2018) Focal loss for dense object detection. arXiv arXiv:1708.02002

    Google Scholar 

  18. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv arXiv:1311.2524

    Google Scholar 

  19. Girshick R (2015) Fast R-CNN. arXiv arXiv:1504.08083

    Google Scholar 

  20. Ren S, He K, Girschick R, Sun J (2012) Faster R-CNN: towards real-time object detection with region proposal networks. arXiv arXiv:1504.08083

    Google Scholar 

  21. Long J, Shelhamer E, Darrell T (2014) Fully convolutional networks for semantic segmentation. arXiv arXiv:1411.4038

    Google Scholar 

  22. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv arXiv:1505.04597

    Google Scholar 

  23. Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2016) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv arXiv:1606.00915

    Google Scholar 

  24. He K, Gkioxari G, Girshick R (2017) Mask R-CNN. arXiv arXiv:1703.06870

    Google Scholar 

  25. Krahenbuhl P, Koltun V (2012) Efficient inference in fully connected CRFs with Gaussian edge potentials. arXiv arXiv:1210.5644

    Google Scholar 

  26. Esteva A et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118. https://doi.org/10.1038/nature21056

    Article  Google Scholar 

  27. Brinker TJ et al (2019) Deep neural networks are superior to dermatologists in melanoma image classification. Eur J Cancer 119:11–17. https://doi.org/10.1016/j.ejca.2019.05.023

    Article  Google Scholar 

  28. Brinker TJ et al (2019) Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur J Cancer 113:47–54. https://doi.org/10.1016/j.ejca.2019.04.001

    Article  Google Scholar 

  29. Han SS et al (2018) Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. J Invest Dermatol 138:1529–1538. https://doi.org/10.1016/j.jid.2018.01.028

    Article  Google Scholar 

  30. Udrea A et al (2020) Accuracy of a smartphone application for triage of skin lesions based on machine learning algorithms. J Eur Acad Dermatol Venereol 34:648–655. https://doi.org/10.1111/jdv.15935

    Article  Google Scholar 

  31. Goulding JM, Levine S, Blizard RA, Deroide F, Swale VJ (2009) Dermatological surgery: a comparison of activity and outcomes in primary and secondary care. Br J Dermatol 161:110–114. https://doi.org/10.1111/j.1365-2133.2009.09228.x

    Article  Google Scholar 

  32. Koelink CJ et al (2014) Diagnostic accuracy and cost-effectiveness of dermoscopy in primary care: a cluster randomized clinical trial. J Eur Acad Dermatol Venereol 28:1442–1449. https://doi.org/10.1111/jdv.12306

    Article  Google Scholar 

  33. Aeffner F et al (2018) Digital microscopy, image analysis, and virtual slide repository. ILAR J 59:66–79. https://doi.org/10.1093/ilar/ily007

    Article  Google Scholar 

  34. Zarella MD et al (2019) A practical guide to whole slide imaging: a white paper from the digital pathology association. Arch Pathol Lab Med 143:222–234. https://doi.org/10.5858/arpa.2018-0343-RA

    Article  Google Scholar 

  35. Ehteshami Bejnordi B et al (2017) Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318:2199–2210. https://doi.org/10.1001/jama.2017.14585

    Article  Google Scholar 

  36. Coudray N et al (2018) Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 24:1559–1567. https://doi.org/10.1038/s41591-018-0177-5

    Article  Google Scholar 

  37. Schneiderman H (1990) Clinical methods: the history, physical, and laboratory examinations, 3rd edn. LexisNexis

    Google Scholar 

  38. Gulshan V et al (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316:2402–2410. https://doi.org/10.1001/jama.2016.17216

    Article  Google Scholar 

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

    Google Scholar 

  40. Abràmoff MD et al (2016) Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci 57:5200–5206. https://doi.org/10.1167/iovs.16-19964

    Article  Google Scholar 

  41. Abramoff MD, Lavin PT, Birch M, Shah N, Folk JC (2018) Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med 1:39. https://doi.org/10.1038/s41746-018-0040-6

    Article  Google Scholar 

  42. Abramoff MD et al (2013) Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol 131:351–357. https://doi.org/10.1001/jamaophthalmol.2013.1743

    Article  Google Scholar 

  43. U.S. Food and Drug Administration (2018, April 11) FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems [Press release]. https://www.fda.gov/news-events/press-announcements/fda-permits-marketing-artificial-intelligence-based-device-detect-certain-diabetes-related-eye

  44. Rajpurkar P et al (2017) CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv arXiv:1711.05225

    Google Scholar 

  45. Halabi SS et al (2019) The RSNA pediatric bone age machine learning challenge. Radiology 290:498–503. https://doi.org/10.1148/radiol.2018180736

    Article  Google Scholar 

  46. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  47. Thian YL et al (2019) Convolutional neural networks for automated fracture detection and localization on wrist radiographs. Radiol Artif Intell 1:e180001. https://doi.org/10.1148/ryai.2019180001

    Article  Google Scholar 

  48. Lundervold AS, Lundervold A (2019) An overview of deep learning in medical imaging focusing on MRI. Z Med Phys 29:102–127. https://doi.org/10.1016/j.zemedi.2018.11.002

    Article  Google Scholar 

  49. Bangalore Yogananda CG et al (2020) A fully automated deep learning network for brain tumor segmentation. Tomography 6:186–193. https://doi.org/10.18383/j.tom.2019.00026

    Article  Google Scholar 

  50. Narayana PA et al (2020) Deep learning for predicting enhancing lesions in multiple sclerosis from noncontrast MRI. Radiology 294:398–404. https://doi.org/10.1148/radiol.2019191061

    Article  Google Scholar 

  51. Islam J, Zhang Y (2018) Brain MRI analysis for Alzheimer’s disease diagnosis using an ensemble system of deep convolutional neural networks. Brain Inform 5:2. https://doi.org/10.1186/s40708-018-0080-3

    Article  Google Scholar 

  52. Struyfs H et al (2020) Automated MRI volumetry as a diagnostic tool for Alzheimer’s disease: validation of icobrain dm. Neuroimage Clin 26:102243. https://doi.org/10.1016/j.nicl.2020.102243

    Article  Google Scholar 

  53. Gong E, Pauly JM, Wintermark M, Zaharchuk G (2018) Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI. J Magn Reson Imaging 48:330–340. https://doi.org/10.1002/jmri.25970

    Article  Google Scholar 

  54. Schieda N et al (2018) Gadolinium-based contrast agents in kidney disease: a comprehensive review and clinical practice guideline issued by the Canadian Association of Radiologists. Can J Kidney Health Dis 5:2054358118778573. https://doi.org/10.1177/2054358118778573

    Article  Google Scholar 

  55. Arbabshirani MR et al (2018) Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. NPJ Digit Med 1:9. https://doi.org/10.1038/s41746-017-0015-z

    Article  Google Scholar 

  56. Hassan AE et al (2020) Early experience utilizing artificial intelligence shows significant reduction in transfer times and length of stay in a hub and spoke model. Interv Neuroradiol 26:615–622. https://doi.org/10.1177/1591019920953055

    Article  Google Scholar 

  57. Morey J, Fiano E, Yaeger K, Zhang X, Fifi J (2020) Impact of Viz LVO on time-to-treatment and clinical outcomes in large vessel occlusion stroke patients presenting to primary stroke centers. https://doi.org/10.1101/2020.07.02.20143834

  58. Kamnitsas K et al (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78. https://doi.org/10.1016/j.media.2016.10.004

    Article  Google Scholar 

  59. Kamnitsas, K. et al. (2016). DeepMedic for Brain Tumor Segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Winzeck, S., Handels, H. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2016. Lecture Notes in Computer Science, vol 10154. Springer, Cham. https://doi.org/10.1007/978-3-319-55524-9_14

  60. Wang B et al (2019) Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation. Med Phys 46:1707–1718. https://doi.org/10.1002/mp.13416

    Article  Google Scholar 

  61. Almeida G, Tavares J (2020) Deep learning in radiation oncology treatment planning for prostate cancer: a systematic review. J Med Syst 44:179. https://doi.org/10.1007/s10916-020-01641-3

    Article  Google Scholar 

  62. Men K et al (2018) Fully automatic and robust segmentation of the clinical target volume for radiotherapy of breast cancer using big data and deep learning. Phys Med 50:13–19. https://doi.org/10.1016/j.ejmp.2018.05.006

    Article  Google Scholar 

  63. Sahiner B et al (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46:e1–e36. https://doi.org/10.1002/mp.13264

    Article  Google Scholar 

  64. Hannun AY et al (2019) Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med 25:65–69. https://doi.org/10.1038/s41591-018-0268-3

    Article  Google Scholar 

  65. Ultromics (2019). Ultromics receives FDA clearance for EchoGo Pro; a first-of-kind solution to help diagnose CAD [Press release]. https://www.ultromics.com/press-releases/ultromics-receives-fda-clearance-for-a-first-of-kind-solution-in-echocardiography-to-help-clinicians-diagnose-disease-1

  66. U.S. Food and Drug Administration (2020, February 7). FDA Authorizes Marketing of First Cardiac Ultrasound Software That Uses Artificial Intelligence to Guide User [Press release]. https://www.fda.gov/news-events/press-announcements/fda-authorizes-marketing-first-cardiac-ultrasound-software-uses-artificial-intelligence-guide-user

  67. Nazir MS, Nicol E (2019) The SCOT-HEART trial: cardiac CT to guide patient management and improve outcomes. Cardiovasc Res 115:e88–e90. https://doi.org/10.1093/cvr/cvz173

    Article  Google Scholar 

  68. Budoff MJ et al (2009) Coronary calcium predicts events better with absolute calcium scores than age-sex-race/ethnicity percentiles: MESA (Multi-Ethnic Study of Atherosclerosis). J Am Coll Cardiol 53:345–352. https://doi.org/10.1016/j.jacc.2008.07.072

    Article  Google Scholar 

  69. Shadmi R, Mazo V, Bregman-Amitai O, Elnekave E (2018) 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, pp 24–28

    Book  Google Scholar 

  70. Stemmer A et al (2020) Using machine learning algorithms to review computed tomography scans and assess risk for cardiovascular disease: retrospective analysis from the National Lung Screening Trial (NLST). PLoS One 15:e0236021. https://doi.org/10.1371/journal.pone.0236021

    Article  Google Scholar 

  71. Siegersma KR et al (2019) Artificial intelligence in cardiovascular imaging: state of the art and implications for the imaging cardiologist. Neth Heart J 27:403–413. https://doi.org/10.1007/s12471-019-01311-1

    Article  Google Scholar 

  72. Weikert T et al (2020) Machine learning in cardiovascular radiology: ESCR position statement on design requirements, quality assessment, current applications, opportunities, and challenges. Eur Radiol. https://doi.org/10.1007/s00330-020-07417-0

  73. Wang F, Preininger A (2019) AI in health: state of the art, challenges, and future directions. Yearb Med Inform 28:16–26. https://doi.org/10.1055/s-0039-1677908

    Article  Google Scholar 

  74. Bien N et al (2018) Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med 15:e1002699. https://doi.org/10.1371/journal.pmed.1002699

    Article  Google Scholar 

  75. Finlayson SG et al (2019) Adversarial attacks on medical machine learning. Science 363:1287–1289. https://doi.org/10.1126/science.aaw4399

    Article  Google Scholar 

  76. U.S. Food and Drug Administration (2020) Premarket notification 510(k). https://www.fda.gov/medical-devices/premarket-submissions/premarket-notification-510k

  77. U.S. Food and Drug Administration (2019) Premarket Approval (PMA). https://www.fda.gov/medical-devices/premarketsubmissions/premarket-approval-pma

  78. U.S. Food and Drug Administration (2019) De Novo classification request. https://www.fda.gov/medical-devices/premarket-submissions/de-novo-classification-request

  79. Kohli A, Mahajan V, Seals K, Kohli A, Jha S (2019) Concepts in U.S. Food and Drug Administration regulation of artificial intelligence for medical imaging. AJR Am J Roentgenol 213:886–888. https://doi.org/10.2214/AJR.18.20410

    Article  Google Scholar 

  80. Benjamens S, Dhunnoo P, Mesko B (2020) The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digit Med 3:118. https://doi.org/10.1038/s41746-020-00324-0

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia

    Ojas Mehta, Zhibin Liao, Mark Jenkinson, Gustavo Carneiro & Johan Verjans

  2. Oxford Centre for Functional MRI of the Brain (FMRIB), Wellcome Centre for Integrative Neuroimaging, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK

    Mark Jenkinson

  3. South Australian Health and Medical Research Institute, Adelaide, Australia

    Johan Verjans

  4. Royal Adelaide Hospital, Adelaide, Australia

    Johan Verjans

Authors
  1. Ojas Mehta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhibin Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Jenkinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gustavo Carneiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johan Verjans
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Johan Verjans .

Editor information

Editors and Affiliations

  1. Department of Emergency Medicine, Monash Health, Clayton, VIC, Australia

    Manda Raz

  2. St Vincent’s Hospital Melbourne, Fitzroy, VIC, Australia

    Tam C. Nguyen

  3. St. Vincent’s Health Australia, Sydney, NSW, Australia

    Erwin Loh

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Mehta, O., Liao, Z., Jenkinson, M., Carneiro, G., Verjans, J. (2022). Machine Learning in Medical Imaging – Clinical Applications and Challenges in Computer Vision. In: Raz, M., Nguyen, T.C., Loh, E. (eds) Artificial Intelligence in Medicine. Springer, Singapore. https://doi.org/10.1007/978-981-19-1223-8_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-1223-8_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-1222-1

  • Online ISBN: 978-981-19-1223-8

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation