Japanese Journal of Radiology

, Volume 36, Issue 4, pp 257–272 | Cite as

Deep learning with convolutional neural network in radiology

  • Koichiro Yasaka
  • Hiroyuki Akai
  • Akira Kunimatsu
  • Shigeru Kiryu
  • Osamu Abe


Deep learning with a convolutional neural network (CNN) is gaining attention recently for its high performance in image recognition. Images themselves can be utilized in a learning process with this technique, and feature extraction in advance of the learning process is not required. Important features can be automatically learned. Thanks to the development of hardware and software in addition to techniques regarding deep learning, application of this technique to radiological images for predicting clinically useful information, such as the detection and the evaluation of lesions, etc., are beginning to be investigated. This article illustrates basic technical knowledge regarding deep learning with CNNs along the actual course (collecting data, implementing CNNs, and training and testing phases). Pitfalls regarding this technique and how to manage them are also illustrated. We also described some advanced topics of deep learning, results of recent clinical studies, and the future directions of clinical application of deep learning techniques.


Deep learning Convolutional neural network CT MRI PET 



No specific funding was disclosed.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical statement

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44.CrossRefPubMedGoogle Scholar
  2. 2.
    Fukushima K, Miyake S. Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognit. 1982;15:455–69.CrossRefGoogle Scholar
  3. 3.
    Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. 1962;160:106–54.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Krizhevsky A, Sutskever I, Hinton G. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing System 25 (NIPS 2012). 2012. Accessed 14 Dec 2017.
  5. 5.
    Kahn CE Jr. From images to actions: opportunities for artificial intelligence in radiology. Radiology. 2017;285:719–20.CrossRefPubMedGoogle Scholar
  6. 6.
    Dreyer KJ, Geis JR. When machines think: radiology’s next frontier. Radiology. 2017;285:713–8.CrossRefPubMedGoogle Scholar
  7. 7.
    Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology. 2016;278:563–77.CrossRefPubMedGoogle Scholar
  8. 8.
    Skogen K, Ganeshan B, Good C, Critchley G, Miles K. Measurements of heterogeneity in gliomas on computed tomography relationship to tumour grade. J Neurooncol. 2013;111:213–9.CrossRefPubMedGoogle Scholar
  9. 9.
    Lubner MG, Stabo N, Abel EJ, Del Rio AM, Pickhardt PJ. CT textural analysis of large primary renal cell carcinomas: pretreatment tumor heterogeneity correlates with histologic findings and clinical outcomes. AJR Am J Roentgenol. 2016;207:96–105.CrossRefPubMedGoogle Scholar
  10. 10.
    Yasaka K, Akai H, Nojima H, Shinozaki-Ushiku A, Fukayama M, Nakajima J, et al. Quantitative computed tomography texture analysis for estimating histological subtypes of thymic epithelial tumors. Eur J Radiol. 2017;92:84–92.CrossRefPubMedGoogle Scholar
  11. 11.
    Kickingereder P, Burth S, Wick A, Gotz M, Eidel O, Schlemmer HP, et al. Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology. 2016;280:880–9.CrossRefPubMedGoogle Scholar
  12. 12.
    Huang Y, Liu Z, He L, Chen X, Pan D, Ma Z, et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non-small cell lung cancer. Radiology. 2016;281:947–57.CrossRefPubMedGoogle Scholar
  13. 13.
    Yip C, Landau D, Kozarski R, Ganeshan B, Thomas R, Michaelidou A, et al. Primary esophageal cancer: heterogeneity as potential prognostic biomarker in patients treated with definitive chemotherapy and radiation therapy. Radiology. 2014;270:141–8.CrossRefPubMedGoogle Scholar
  14. 14.
    Ng F, Ganeshan B, Kozarski R, Miles KA, Goh V. Assessment of primary colorectal cancer heterogeneity by using whole-tumor texture analysis: contrast-enhanced CT texture as a biomarker of 5-year survival. Radiology. 2013;266:177–84.CrossRefPubMedGoogle Scholar
  15. 15.
    Kiryu S, Akai H, Nojima M, Hasegawa K, Shinkawa H, Kokudo N, et al. Impact of hepatocellular carcinoma heterogeneity on computed tomography as a prognostic indicator. Sci Rep. 2017;7:12689.CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Li H, Zhu Y, Burnside ES, Drukker K, Hoadley KA, Fan C, et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of MammaPrint, Oncotype DX, and PAM50 gene assays. Radiology. 2016;281:382–91.CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Goh V, Ganeshan B, Nathan P, Juttla JK, Vinayan A, Miles KA. Assessment of response to tyrosine kinase inhibitors in metastatic renal cell cancer: CT texture as a predictive biomarker. Radiology. 2011;261:165–71.CrossRefPubMedGoogle Scholar
  18. 18.
    Le QV, Ranzato M, Monga R, Devin M, Chen K, Corrado GS, et al. Building high-level features using large scale unsupervised learning. International Conference on Machine Learning. 2012. Accessed 14 Dec 2017.
  19. 19.
    Nakao T, Hanaoka S, Nomura Y, Sato I, Nemoto M, Miki S, et al. Deep neural network-based computer-assisted detection of cerebral aneurysms in MR angiography. J Magn Reson Imaging. 2017. Scholar
  20. 20.
    Gonzalez G, Ash SY, Vegas Sanchez-Ferrero G, Onieva Onieva J, Rahaghi FN, Ross JC, et al. Disease staging and prognosis in smokers using deep learning in chest computed tomography. Am J Respir Crit Care Med. 2018;197:193–203.CrossRefPubMedGoogle Scholar
  21. 21.
    Nair V, Hinton G. Rectified linear units improve restricted Boltzmann machines. International Conference on Machine Learning. 2010. Accessed 14 Dec 2017.Google Scholar
  22. 22.
    Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.Google Scholar
  23. 23.
    Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Cornell University Library. 2015. Accessed 30 April 2017.
  24. 24.
    Banerjee I, Crawley A, Bhethanabotla M, Daldrup-Link HE, Rubin DL. Transfer learning on fused multiparametric MR images for classifying histopathological subtypes of rhabdomyosarcoma. Comput Med Imaging Graph. 2017. Scholar
  25. 25.
    Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011;12:2121–59.Google Scholar
  26. 26.
    Kingma DP, Ba JL. Adam: a method for stochastic optimization. Cornell University Library. 2014. Accessed 30 April 2017.
  27. 27.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. Cornell University Library. 2014. Accessed 14 Dec 2017.
  28. 28.
    He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Cornell University Library. 2015. Accessed 14 Dec 2017.
  29. 29.
    Korfiatis P, Kline TL, Lachance DH, Parney IF, Buckner JC, Erickson BJ. Residual deep convolutional neural network predicts MGMT methylation status. J Digit Imaging. 2017;30:622–8.CrossRefPubMedGoogle Scholar
  30. 30.
    Andrearczyk V, Whelan PF. Using filter banks in convolutional neural networks for texture classification. Cornell University Library. 2016. Accessed 30 April 2017.
  31. 31.
    Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. 2015. Accessed 14 Dec 2017.
  32. 32.
    Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Cornell University Library. 2015. Accessed 14 Dec 2017.
  33. 33.
    Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. Cornell University Library. 2015. Accessed 14 Dec 2017.
  34. 34.
    Mao X, Shen C, Yang Y. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Cornell University Library. 2016. Accessed 14 Dec 2017.
  35. 35.
    Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284:574–82.CrossRefPubMedGoogle Scholar
  36. 36.
    Caruana R. Multitask learning. Mach Learn. 1997;28:41–75.CrossRefGoogle Scholar
  37. 37.
    Bengio Y. Deep learning of representations for unsupervised and transfer learning. In: JMLR: Workshop and Conference Proceedings. 2012;17–37.Google Scholar
  38. 38.
    Mohamed AA, Berg WA, Peng H, Luo Y, Jankowitz RC, Wu S. A deep learning method for classifying mammographic breast density categories. Med Phys. 2018;45:314–21.CrossRefPubMedGoogle Scholar
  39. 39.
    Larson DB, Chen MC, Lungren MP, Halabi SS, Stence NV, Langlotz CP. Performance of a deep-learning neural network model in assessing skeletal maturity on pediatric hand radiographs. Radiology. 2017. Scholar
  40. 40.
    Cicero M, Bilbily A, Colak E, Dowdell T, Gray B, Perampaladas K, et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest Radiol. 2017;52:281–7.CrossRefPubMedGoogle Scholar
  41. 41.
    Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Invest Radiol. 2017;52:434–40.CrossRefPubMedGoogle Scholar
  42. 42.
    Prevedello LM, Erdal BS, Ryu JL, Little KJ, Demirer M, Qian S, et al. Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology. 2017;285:923–31.CrossRefPubMedGoogle Scholar
  43. 43.
    Wang X, Yang W, Weinreb J, Han J, Li Q, Kong X, et al. Searching for prostate cancer by fully automated magnetic resonance imaging classification: deep learning versus non-deep learning. Sci Rep. 2017;7:15415.CrossRefPubMedPubMedCentralGoogle Scholar
  44. 44.
    Ghafoorian M, Karssemeijer N, Heskes T, Bergkamp M, Wissink J, Obels J, et al. Deep multi-scale location-aware 3D convolutional neural networks for automated detection of lacunes of presumed vascular origin. Neuroimage Clin. 2017;14:391–9.CrossRefPubMedPubMedCentralGoogle Scholar
  45. 45.
    Jiang H, Ma H, Qian W, Gao M, Li Y. An automatic detection system of lung nodule based on multi-group patch-based deep learning network. IEEE J Biomed Health Inform. 2017. Scholar
  46. 46.
    Parsons DW, Jones S, Zhang X, Lin JC, Leary RJ, Angenendt P, et al. An integrated genomic analysis of human glioblastoma multiforme. Science. 2008;321:1807–12.CrossRefPubMedPubMedCentralGoogle Scholar
  47. 47.
    Yan H, Parsons DW, Jin G, McLendon R, Rasheed BA, Yuan W, et al. IDH1 and IDH2 mutations in gliomas. N Engl J Med. 2009;360:765–73.CrossRefPubMedPubMedCentralGoogle Scholar
  48. 48.
    Hegi ME, Diserens AC, Gorlia T, Hamou MF, de Tribolet N, Weller M, et al. MGMT gene silencing and benefit from temozolomide in glioblastoma. N Engl J Med. 2005;352:997–1003.CrossRefPubMedGoogle Scholar
  49. 49.
    Chang K, Bai HX, Zhou H, Su C, Bi WL, Agbodza E, et al. Residual convolutional neural network for determination of IDH status in low- and high-grade gliomas from MR imaging. Clin Cancer Res. 2017. Scholar
  50. 50.
    Chi J, Walia E, Babyn P, Wang J, Groot G, Eramian M. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging. 2017;30:477–86.CrossRefPubMedPubMedCentralGoogle Scholar
  51. 51.
    Malempati S, Hawkins DS. Rhabdomyosarcoma: review of the Children’s Oncology Group (COG) Soft-Tissue Sarcoma Committee experience and rationale for current COG studies. Pediatr Blood Cancer. 2012;59:5–10.CrossRefPubMedPubMedCentralGoogle Scholar
  52. 52.
    Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging. 2016;35:1207–16.CrossRefPubMedGoogle Scholar
  53. 53.
    Song Q, Zhao L, Luo X, Dou X. Using deep learning for classification of lung nodules on computed tomography images. J Healthc Eng. 2017;2017:8314740.CrossRefPubMedPubMedCentralGoogle Scholar
  54. 54.
    Nibali A, He Z, Wollersheim D. Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg. 2017;12:1799–808.CrossRefPubMedGoogle Scholar
  55. 55.
    Yasaka K, Akai H, Kunimatsu A, Abe O, Kiryu S. Liver fibrosis: deep convolutional neural network for staging by using gadoxetic acid-enhanced hepatobiliary phase MR images. Radiology. 2017. Scholar
  56. 56.
    Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology. 2018;286:887–96.CrossRefPubMedGoogle Scholar
  57. 57.
    Ben-Cohen A, Klang E, Diamant I, Rozendorn N, Raskin SP, Konen E, et al. CT image-based decision support system for categorization of Liver metastases into primary cancer sites: initial results. Acad Radiol. 2017;24:1501–9.CrossRefPubMedGoogle Scholar
  58. 58.
    Lao J, Chen Y, Li ZC, Li Q, Zhang J, Liu J, et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep. 2017;7:10353.CrossRefPubMedPubMedCentralGoogle Scholar
  59. 59.
    Ibragimov B, Xing L. Segmentation of organs-at-risks in head and neck CT images using convolutional neural networks. Med Phys. 2017;44:547–57.CrossRefPubMedPubMedCentralGoogle Scholar
  60. 60.
    Ibragimov B, Toesca D, Chang D, Koong A, Xing L. Combining deep learning with anatomy analysis for segmentation of portal vein for liver SBRT planning. Phys Med Biol. 2017;62:8943–58.PubMedGoogle Scholar
  61. 61.
    Men K, Dai J, Li Y. Automatic segmentation of the clinical target volume and organs at risk in the planning CT for rectal cancer using deep dilated convolutional neural networks. Med Phys. 2017;44:6377–89.CrossRefPubMedGoogle Scholar
  62. 62.
    Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging-based attenuation correction for PET/MR imaging. Radiology. 2018;286:676–84.CrossRefPubMedGoogle Scholar
  63. 63.
    Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y, et al. Direct PseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). J Nucl Med. 2017. Scholar
  64. 64.
    Yasaka K, Katsura M, Akahane M, Sato J, Matsuda I, Ohtomo K. Model-based iterative reconstruction for reduction of radiation dose in abdominopelvic CT: comparison to adaptive statistical iterative reconstruction. Springerplus. 2013;2:209.CrossRefPubMedPubMedCentralGoogle Scholar
  65. 65.
    Katsura M, Matsuda I, Akahane M, Sato J, Akai H, Yasaka K, et al. Model-based iterative reconstruction technique for radiation dose reduction in chest CT: comparison with the adaptive statistical iterative reconstruction technique. Eur Radiol. 2012;22:1613–23.CrossRefPubMedGoogle Scholar
  66. 66.
    Pickhardt PJ, Lubner MG, Kim DH, Tang J, Ruma JA, del Rio AM, et al. Abdominal CT with model-based iterative reconstruction (MBIR): initial results of a prospective trial comparing ultralow-dose with standard-dose imaging. AJR Am J Roentgenol. 2012;199:1266–74.CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Yamada Y, Jinzaki M, Tanami Y, Shiomi E, Sugiura H, Abe T, et al. Model-based iterative reconstruction technique for ultralow-dose computed tomography of the lung: a pilot study. Invest Radiol. 2012;47:482–9.CrossRefPubMedGoogle Scholar
  68. 68.
    Deak Z, Grimm JM, Treitl M, Geyer LL, Linsenmaier U, Korner M, et al. Filtered back projection, adaptive statistical iterative reconstruction, and a model-based iterative reconstruction in abdominal CT: an experimental clinical study. Radiology. 2013;266:197–206.CrossRefPubMedGoogle Scholar
  69. 69.
    Yasaka K, Katsura M, Hanaoka S, Sato J, Ohtomo K. High-resolution CT with new model-based iterative reconstruction with resolution preference algorithm in evaluations of lung nodules: comparison with conventional model-based iterative reconstruction and adaptive statistical iterative reconstruction. Eur J Radiol. 2016;85:599–606.CrossRefPubMedGoogle Scholar
  70. 70.
    Yasaka K, Furuta T, Kubo T, Maeda E, Katsura M, Sato J, et al. Full and hybrid iterative reconstruction to reduce artifacts in abdominal CT for patients scanned without arm elevation. Acta Radiol. 2017;58:1085–93.CrossRefPubMedGoogle Scholar
  71. 71.
    Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, et al. Low-dose CT via convolutional neural network. Biomed Opt Express. 2017;8:679–94.CrossRefPubMedPubMedCentralGoogle Scholar
  72. 72.
    Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, et al. Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans Med Imaging. 2017;36:2524–35.CrossRefPubMedGoogle Scholar
  73. 73.
    Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Measuring computed tomography scanner variability of radiomics features. Invest Radiol. 2015;50:757–65.CrossRefPubMedPubMedCentralGoogle Scholar
  74. 74.
    Yasaka K, Akai H, Mackin D, Court L, Moros E, Ohtomo K, et al. Precision of quantitative computed tomography texture analysis using image filtering: a phantom study for scanner variability. Medicine (Baltimore). 2017;96:e6993.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Japan Radiological Society 2018

Authors and Affiliations

  1. 1.Department of Radiology, The Institute of Medical ScienceThe University of TokyoTokyoJapan
  2. 2.Department of Radiology, Graduate School of Medical SciencesInternational University of Health and WelfareNaritaJapan
  3. 3.Department of Radiology, Graduate School of MedicineThe University of TokyoTokyoJapan

Personalised recommendations