The use of machine learning (ML) has been increasing rapidly in the medical imaging field, including computer-aided diagnosis (CAD), radiomics, and medical image analysis. Recently, an ML area called deep learning emerged in the computer vision field and became very popular in many fields. It started from an event in late 2012, when a deep-learning approach based on a convolutional neural network (CNN) won an overwhelming victory in the best-known worldwide computer vision competition, ImageNet Classification. Since then, researchers in virtually all fields, including medical imaging, have started actively participating in the explosively growing field of deep learning. In this paper, the area of deep learning in medical imaging is overviewed, including (1) what was changed in machine learning before and after the introduction of deep learning, (2) what is the source of the power of deep learning, (3) two major deep-learning models: a massive-training artificial neural network (MTANN) and a convolutional neural network (CNN), (4) similarities and differences between the two models, and (5) their applications to medical imaging. This review shows that ML with feature input (or feature-based ML) was dominant before the introduction of deep learning, and that the major and essential difference between ML before and after deep learning is the learning of image data directly without object segmentation or feature extraction; thus, it is the source of the power of deep learning, although the depth of the model is an important attribute. The class of ML with image input (or image-based ML) including deep learning has a long history, but recently gained popularity due to the use of the new terminology, deep learning. There are two major models in this class of ML in medical imaging, MTANN and CNN, which have similarities as well as several differences. In our experience, MTANNs were substantially more efficient in their development, had a higher performance, and required a lesser number of training cases than did CNNs. “Deep learning”, or ML with image input, in medical imaging is an explosively growing, promising field. It is expected that ML with image input will be the mainstream area in the field of medical imaging in the next few decades.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Suzuki K. Machine learning for medical imaging, vol. 3. Algorithms. 2010. http://www.mdpi.com/journal/algorithms/special_issues/machine-learning-for-medical-imaging. Accessed 5 July 2017.
Wang F, Yan P, Suzuki K, Shen D, eds. Machine learning in medical imaging (MLMI), vol. 6357. Lecture notes in computer science. Berlin: Springer; 2010.
Suzuki K, Wang F, Shen D, Yan P, eds. Machine learning in medical imaging (MLMI), vol. 7009. Lecture notes in computer science. Berlin: Springer; 2011.
Suzuki K. Machine learning for medical imaging 2012, vol. 5. Algorithms. 2012. http://www.mdpi.com/journal/algorithms/special_issues/medical_imaging_2012. Accessed 5 July 2017.
Suzuki K, Yan P, Wang F, Shen D. Machine learning in medical imaging. Int J Biomed Imaging. 2012;2012:123727.
Suzuki K. Machine learning in computer-aided diagnosis: medical imaging intelligence and analysis. Hershey: IGI Global; 2012.
Wang F, Shen D, Yan P, Suzuki K, editors. Machine learning in medical imaging (MLMI), vol. 7588. Lecture notes in computer science. Berlin: Springer; 2012.
Suzuki K. Machine learning in computer-aided diagnosis of the thorax and colon in CT: a survey. IEICE Trans Inf Syst. 2013;E96-D(4):772–83.
Wu G, Zhang D, Shen D, Yan P, Suzuki K, Wang F, editors. Machine learning in medical imaging (MLMI), vol. 8184. Lecture notes in computer science. Berlin: Springer; 2013.
Yan P, Suzuki K, Wang F, Shen D. Machine learning in medical imaging. Mach Vision Appl. 2013;24(7):1327–9.
Shen D, Wu G, Zhang D, Suzuki K, Wang F, Yan P. Machine learning in medical imaging. Comput Med Imaging Graph. 2015;41:1–2.
Suzuki K, Zhou L, Wang Q. Machine learning in medical imaging. Pattern Recognit. 2017;63:465–7.
El-Baz A, Gimel’farb G, Suzuki K. Machine learning applications in medical image analysis. Comput Math Methods Med. 2017;2017:2.
Doi K. Overview on research and development of computer-aided diagnostic schemes. Semin Ultrasound CT MRI. 2004;25(5):404–10.
Doi K. Current status and future potential of computer-aided diagnosis in medical imaging. Br J Radiol. 2005;78(Spec No 1):S3–19.
Doi K. Diagnostic imaging over the last 50 years: research and development in medical imaging science and technology. Phys Med Biol. 2006;51(13):R5–27.
Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph. 2007;31(4–5):198–211.
Lostumbo A, Wanamaker C, Tsai J, Suzuki K, Dachman AH. Comparison of 2D and 3D views for evaluation of flat lesions in CT colonography. Acad Radiol. 2010;17(1):39–47.
Lostumbo A, Suzuki K, Dachman AH. Flat lesions in CT colonography. Abdom Imaging. 2010;35(5):578–83.
Suzuki K. Computational intelligence in biomedical imaging. New York: Springer; 2014.
Duda RO, Hart PE, Stork DG. Pattern recognition. 2nd ed. Hoboken: Wiley Interscience; 2001.
Suzuki K, Armato SG 3rd, Li F, Sone S, Doi K. Massive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in low-dose computed tomography. Med Phys. 2003;30(7):1602–17.
Suzuki K. Pixel-based machine learning in medical imaging. Int J Biomed Imaging. 2012;2012:792079.
Arimura H, Katsuragawa S, Suzuki K, et al. Computerized scheme for automated detection of lung nodules in low-dose computed tomography images for lung cancer screening. Acad Radiol. 2004;11(6):617–29.
Li F, Arimura H, Suzuki K, et al. Computer-aided detection of peripheral lung cancers missed at CT: ROC analyses without and with localization. Radiology. 2005;237(2):684–90.
Suzuki K. A supervised ‘lesion-enhancement’ filter by use of a massive-training artificial neural network (MTANN) in computer-aided diagnosis (CAD). Phys Med Biol. 2009;54(18):S31–45.
Suzuki K, Doi K. How can a massive training artificial neural network (MTANN) be trained with a small number of cases in the distinction between nodules and vessels in thoracic CT? Acad Radiol. 2005;12(10):1333–41.
Suzuki K, Shiraishi J, Abe H, MacMahon H, Doi K. False-positive reduction in computer-aided diagnostic scheme for detecting nodules in chest radiographs by means of massive training artificial neural network. Acad Radiol. 2005;12(2):191–201.
Chen S, Suzuki K. Computerized detection of lung nodules by means of “virtual dual-energy” radiography. IEEE Trans Biomed Eng. 2013;60(2):369–78.
Suzuki K, Abe H, Li F, Doi K. Suppression of the contrast of ribs in chest radiographs by means of massive training artificial neural network. Paper presented at Proc. SPIE medical imaging (SPIE MI), San Diego, CA, May 2004.
Suzuki K, Abe H, MacMahon H, Doi K. Image-processing technique for suppressing ribs in chest radiographs by means of massive training artificial neural network (MTANN). IEEE Trans Med Imaging. 2006;25(4):406–16.
Oda S, Awai K, Suzuki K, et al. Performance of radiologists in detection of small pulmonary nodules on chest radiographs: effect of rib suppression with a massive-training artificial neural network. AJR Am J Roentgenol. 2009;193(5):W397–402.
Chen S, Suzuki K. Separation of bones from chest radiographs by means of anatomically specific multiple massive-training ANNs combined with total variation minimization smoothing. IEEE Trans Med Imaging. 2014;33(2):246–57.
Chen S, Zhong S, Yao L, Shang Y, Suzuki K. Enhancement of chest radiographs obtained in the intensive care unit through bone suppression and consistent processing. Phys Med Biol. 2016;61(6):2283–301.
Suzuki K, Li F, Sone S, Doi K. Computer-aided diagnostic scheme for distinction between benign and malignant nodules in thoracic low-dose CT by use of massive training artificial neural network. IEEE Trans Med Imaging. 2005;24(9):1138–50.
Suzuki K, Rockey DC, Dachman AH. CT colonography: advanced computer-aided detection scheme utilizing MTANNs for detection of “missed” polyps in a multicenter clinical trial. Med Phys. 2010;37(1):12–21.
Suzuki K, Yoshida H, Nappi J, Armato SG 3rd, Dachman AH. Mixture of expert 3D massive-training ANNs for reduction of multiple types of false positives in CAD for detection of polyps in CT colonography. Med Phys. 2008;35(2):694–703.
Suzuki K, Yoshida H, Nappi J, Dachman AH. Massive-training artificial neural network (MTANN) for reduction of false positives in computer-aided detection of polyps: suppression of rectal tubes. Med Phys. 2006;33(10):3814–24.
Xu J, Suzuki K. Massive-training support vector regression and Gaussian process for false-positive reduction in computer-aided detection of polyps in CT colonography. Med Phys. 2011;38:1888–902.
Suzuki K, Zhang J, Xu J. Massive-training artificial neural network coupled with Laplacian-eigenfunction-based dimensionality reduction for computer-aided detection of polyps in CT colonography. IEEE Trans Med Imaging. 2010;29(11):1907–17.
Lawrence S, Giles CL, Tsoi AC, Back AD. Face recognition: a convolutional neural-network approach. IEEE Trans Neural Netw. 1997;8(1):98–113.
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Paper presented at advances in neural information processing systems, 2012.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.
Fukunaga K. Introduction to statistical pattern recognition. 2nd ed. San Diego: Academic Press; 1990.
Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature. 1986;323:533–6.
Vapnik VN. The nature of statistical learning theory. Berlin: Springer; 1995.
Hinton G, Osindero S, Teh Y-W. A fast learning algorithm for deep belief nets. Neural Comput. 2006;18(7):1527–54.
Ho TK. Random decision forests. Paper presented at document analysis and recognition, 1995, proceedings of the third international conference on, 1995.
Mairal J, Bach F, Ponce J, Sapiro G. Online dictionary learning for sparse coding. In: Proceedings of the 26th annual international conference on machine learning, Montreal, Quebec, Canada, 2009.
Fukushima K. Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern. 1980;36(4):193–202.
LeCun Y, Boser B, Denker JS, et al. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1989;1(4):541–51.
Suzuki K, Horiba I, Ikegaya K, Nanki M. Recognition of coronary artery stenosis using neural network on DSA system. IEICE Trans Inf Syst. 1994;J77-D-II:1910–6.
Suzuki K, Horiba I, Sugie N, Ikeda S. Improvement of image quality of X-ray fluoroscopy using spatiotemporal neural filter which learns noise reduction, edge enhancement and motion compensation. Paper presented at Proc. Int. Conf. signal processing applications and technology (ICSPAT), Boston, MA, October, 1996.
Suzuki K, Horiba I, N. S. Edge detection from noisy images using a neural edge detector. In: Proc. IEEE Int. workshop on neural networks for signal processing (NNSP). December 2000;10:487–496.
Suzuki K, Liu Y, Higaki T, Funama Y, Awai K. Supervised conversion of ultra-low-dose to higher-dose CT images by using pixel-based machine learning: phantom and initial patient studies. In: Program of scientific assembly and annual meeting of Radiological Society of North America (RSNA), SST14-06, Chicago, IL, 2013.
Suzuki K, Horiba I, Sugie N, Nanki M. Neural filter with selection of input features and its application to image quality improvement of medical image sequences. IEICE Trans Inf Syst. 2002;E85-D(10):1710–8.
Suzuki K, Horiba I, Sugie N. Efficient approximation of neural filters for removing quantum noise from images. IEEE Trans Signal Process. 2002;50(7):1787–99.
Suzuki K, Horiba I, Sugie N. Neural edge enhancer for supervised edge enhancement from noisy images. IEEE Trans Pattern Anal Mach Intell. 2003;25(12):1582–96.
Suzuki K, Horiba I, Sugie N, Nanki M. Extraction of left ventricular contours from left ventriculograms by means of a neural edge detector. IEEE Trans Med Imaging. 2004;23(3):330–9.
Vapnik VN. Statistical learning theory. New York: Wiley; 1998.
Suzuki K. Determining the receptive field of a neural filter. J Neural Eng. 2004;1(4):228–37.
Suzuki K, Horiba I, Sugie N. A simple neural network pruning algorithm with application to filter synthesis. Neural Process Lett. 2001;13(1):43–53.
Tajbakhsh N, Suzuki K. Comparing two classes of end-to-end learning machines for lung nodule detection and classification: MTANNs vs. CNNs. Pattern Recognit. 2017;63:476–86.
Shiraishi J, Li Q, Suzuki K, Engelmann R, Doi K. Computer-aided diagnostic scheme for the detection of lung nodules on chest radiographs: localized search method based on anatomical classification. Med Phys. 2006;33(7):2642–53.
Coppini G, Diciotti S, Falchini M, Villari N, Valli G. Neural networks for computer-aided diagnosis: detection of lung nodules in chest radiograms. IEEE Trans Inf Technol Biomed. 2003;7(4):344–57.
Hardie RC, Rogers SK, Wilson T, Rogers A. Performance analysis of a new computer aided detection system for identifying lung nodules on chest radiographs. Med Image Anal. 2008;12(3):240–58.
Chen S, Suzuki K, MacMahon H. A computer-aided diagnostic scheme for lung nodule detection in chest radiographs by means of two-stage nodule-enhancement with support vector classification. Med Phys. 2011;38:1844–58.
Armato SG 3rd, Giger ML, MacMahon H. Automated detection of lung nodules in CT scans: preliminary results. Med Phys. 2001;28(8):1552–61.
Ye X, Lin X, Dehmeshki J, Slabaugh G, Beddoe G. Shape-based computer-aided detection of lung nodules in thoracic CT images. IEEE Trans Biomed Eng. 2009;56(7):1810–20.
Way TW, Sahiner B, Chan HP, et al. Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys. 2009;36(7):3086–98.
Aoyama M, Li Q, Katsuragawa S, MacMahon H, Doi K. Automated computerized scheme for distinction between benign and malignant solitary pulmonary nodules on chest images. Med Phys. 2002;29(5):701–8.
Aoyama M, Li Q, Katsuragawa S, Li F, Sone S, Doi K. Computerized scheme for determination of the likelihood measure of malignancy for pulmonary nodules on low-dose CT images. Med. Phys. 2003;30(3):387–94.
Shah SK, McNitt-Gray MF, Rogers SR, et al. Computer aided characterization of the solitary pulmonary nodule using volumetric and contrast enhancement features. Acad. Radiol. 2005;12(10):1310–9.
Wu Y, Doi K, Giger ML, Nishikawa RM. Computerized detection of clustered microcalcifications in digital mammograms: applications of artificial neural networks. Med Phys. 1992;19(3):555–60.
El-Naqa I, Yang Y, Wernick MN, Galatsanos NP, Nishikawa RM. A support vector machine approach for detection of microcalcifications. IEEE Trans Med Imaging. 2002;21(12):1552–63.
Yu SN, Li KY, Huang YK. Detection of microcalcifications in digital mammograms using wavelet filter and Markov random field model. Comput Med Imaging Graph. 2006;30(3):163–73.
Ge J, Sahiner B, Hadjiiski LM, et al. Computer aided detection of clusters of microcalcifications on full field digital mammograms. Med Phys. 2006;33(8):2975–88.
Wu YT, Wei J, Hadjiiski LM, et al. Bilateral analysis based false positive reduction for computer-aided mass detection. Med Phys. 2007;34(8):3334–44.
Huo Z, Giger ML, Vyborny CJ, Wolverton DE, Schmidt RA, Doi K. Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol. 1998;5(3):155–68.
Delogu P, Evelina Fantacci M, Kasae P, Retico A. Characterization of mammographic masses using a gradient-based segmentation algorithm and a neural classifier. Comput Biol Med. 2007;37(10):1479–91.
Shi J, Sahiner B, Chan HP, et al. Characterization of mammographic masses based on level set segmentation with new image features and patient information. Med Phys. 2008;35(1):280–90.
Yoshida H, Nappi J. Three-dimensional computer-aided diagnosis scheme for detection of colonic polyps. IEEE Trans Med Imaging. 2001;20(12):1261–74.
Jerebko AK, Summers RM, Malley JD, Franaszek M, Johnson CD. Computer-assisted detection of colonic polyps with CT colonography using neural networks and binary classification trees. Med Phys. 2003;30(1):52–60.
Wang S, Yao J, Summers RM. Improved classifier for computer-aided polyp detection in CT colonography by nonlinear dimensionality reduction. Med Phys. 2008;35(4):1377–86.
Arimura H, Li Q, Korogi Y, et al. Computerized detection of intracranial aneurysms for three-dimensional MR angiography: feature extraction of small protrusions based on a shape-based difference image technique. Med Phys. 2006;33(2):394–401.
Muramatsu C, Li Q, Schmidt RA, et al. Determination of subjective similarity for pairs of masses and pairs of clustered microcalcifications on mammograms: comparison of similarity ranking scores and absolute similarity ratings. Med Phys. 2007;34(7):2890–5.
Muramatsu C, Li Q, Schmidt R, et al. Experimental determination of subjective similarity for pairs of clustered microcalcifications on mammograms: observer study results. Med Phys. 2006;33(9):3460–8.
Muramatsu C, Li Q, Suzuki K, et al. Investigation of psychophysical measure for evaluation of similar images for mammographic masses: preliminary results. Med Phys. 2005;32(7):2295–304.
Lo SB, Lou SA, Lin JS, Freedman MT, Chien MV, Mun SK. Artificial convolution neural network techniques and applications for lung nodule detection. IEEE Trans Med Imaging. 1995;14(4):711–8.
Lo SCB, Chan HP, Lin JS, Li H, Freedman MT, Mun SK. Artificial convolution neural network for medical image pattern recognition. Neural Netw. 1995;8(7–8):1201–14.
Lin JS, Lo SB, Hasegawa A, Freedman MT, Mun SK. Reduction of false positives in lung nodule detection using a two-level neural classification. IEEE Trans Med Imaging. 1996;15(2):206–17.
Lo SC, Li H, Wang Y, Kinnard L, Freedman MT. A multiple circular path convolution neural network system for detection of mammographic masses. IEEE Trans Med Imaging. 2002;21(2):150–8.
Sahiner B, Chan HP, Petrick N, et al. Classification of mass and normal breast tissue: a convolution neural network classifier with spatial domain and texture images. IEEE Trans Med Imaging. 1996;15(5):598–610.
Zhang W, Doi K, Giger ML, Nishikawa RM, Schmidt RA. An improved shift-invariant artificial neural network for computerized detection of clustered microcalcifications in digital mammograms. Med Phys. 1996;23(4):595–601.
Zhang W, Doi K, Giger ML, Wu Y, Nishikawa RM, Schmidt RA. Computerized detection of clustered microcalcifications in digital mammograms using a shift-invariant artificial neural network. Med Phys. 1994;21(4):517–24.
Samala RK, Chan HP, Hadjiiski L, Helvie MA, Wei J, Cha K. Mass detection in digital breast tomosynthesis: deep convolutional neural network with transfer learning from mammography. Med Phys. 2016;43(12):6654.
Teramoto A, Fujita H, Yamamuro O, Tamaki T. Automated detection of pulmonary nodules in PET/CT images: ensemble false-positive reduction using a convolutional neural network technique. Med Phys. 2016;43(6):2821.
Ciompi F, de Hoop B, van Riel SJ, et al. Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box. Med Image Anal. 2015;26(1):195–202.
Kooi T, van Ginneken B, Karssemeijer N, den Heeten A. Discriminating solitary cysts from soft tissue lesions in mammography using a pretrained deep convolutional neural network. Med Phys. 2017;44(3):1017–27.
Lekadir K, Galimzianova A, Betriu A, et al. A convolutional neural network for automatic characterization of plaque composition in carotid ultrasound. IEEE J Biomed Health Inform. 2017;21(1):48–55.
Miki Y, Muramatsu C, Hayashi T, et al. Classification of teeth in cone-beam CT using deep convolutional neural network. Comput Biol Med. 2017;80:24–9.
He L, Chao Y, Suzuki K, Wu K. Fast connected-component labeling. Pattern Recognit. 2009;42:1977–87.
He L, Chao Y, Suzuki K. A run-based two-scan labeling algorithm. IEEE Trans Image Process. 2008;17(5):749–56.
Suzuki K, Horiba I, Sugie N. Linear-time connected-component labeling based on sequential local operations. Comput Vis Image Underst. 2003;89(1):1–23.
Roth HR, Lu L, Seff A, et al. A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Med Image Comput Comput Assist Interv. 2014;17(Pt 1):520–7.
Hasegawa A, Itoh K, Ichioka Y. Generalization of shift invariant neural networks: image processing of corneal endothelium. Neural Netw. 1996;9(2):345–56.
Cha KH, Hadjiiski L, Samala RK, Chan HP, Caoili EM, Cohan RH. Urinary bladder segmentation in CT urography using deep-learning convolutional neural network and level sets. Med Phys. 2016;43(4):1882.
Moeskops P, Viergever MA, Mendrik AM, de Vries LS, Benders MJ, Isgum I. Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans Med Imaging. 2016;35(5):1252–61.
Austin JH, Romney BM, Goldsmith LS. Missed bronchogenic carcinoma: radiographic findings in 27 patients with a potentially resectable lesion evident in retrospect. Radiology. 1992;182(1):115–22.
Shah PK, Austin JH, White CS, et al. Missed non-small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. Radiology. 2003;226(1):235–41.
Ishigaki T, Sakuma S, Horikawa Y, Ikeda M, Yamaguchi H. One-shot dual-energy subtraction imaging. Radiology. 1986;161(1):271–3.
Loog M, van Ginneken B. Segmentation of the posterior ribs in chest radiographs using iterated contextual pixel classification. IEEE Trans Med Imaging. 2006;25(5):602–11.
Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. 1962;160(1):106–54.
The authors are grateful to members in the Computational Intelligence in Biomedical Imaging Laboratory at the Illinois Institute of Technology, and to Suzuki’s Laboratory at the University of Chicago for their valuable contributions to machine-learning and deep-learning research in medical imaging over the past 15 years, and to Ms. Lanzl for her edits.
Machine learning and computer-aided diagnosis technologies developed in the author Kenji Suzuki’s laboratories have been non-exclusively licensed to several companies and have been commercialized. The author Kenji Suzuki received royalty distributions from the companies.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the Institutional Review Board (IRB) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all individual participants included in the studies. The studies complied with the Health Insurance Portability and Accountability Act (HIPAA).
About this article
Cite this article
Suzuki, K. Overview of deep learning in medical imaging. Radiol Phys Technol 10, 257–273 (2017). https://doi.org/10.1007/s12194-017-0406-5
- Deep learning
- Convolutional neural network
- Massive-training artificial neural network
- Computer-aided diagnosis
- Medical image analysis