Skip to main content
SpringerLink
Log in

Literature review: efficient deep neural networks techniques for medical image analysis

  • Review
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Significant evolution in deep learning took place in 2010, when software developers started using graphical processing units for general-purpose applications. From that date, the deep neural network (DNN) started progressive steps across different applications ranging from natural language processing to hyperspectral image processing. The convolutional neural network (CNN) mostly triggers the interest, as it is considered one of the most powerful ways to learn useful representations of images and other structured data. The revolution of DNNs in medical imaging (MI) came in 2012, when Li launched ImageNet, a free database of more than 14 million labeled medical images. This state-of-the-art work presents a comprehensive study for the recent DNNs research directions applied in MI analysis. Clinical and pathological analysis through a selected patch of most cited researches is introduced. It will be shown how DNNs are able to tackle medical problems: classification, detection, localization, segmentation, and automatic diagnosis. Datasets comprises a range of imaging technologies: X-Ray, MRI, CT, Ultrasound, PET, Fluorescene Angiography, and even photographic images. This work surveys different patterns of DNNs and focuses somehow on the CNN, which offers an outstanding percentage of solutions compared to other DNNs structures. CNN emphasizes image features and has well-known architectures. On the other hand, limitations beyond DNNs training and execution time will be explained. Problems related to data augmentation and image annotation will be analyzed among a multiple of high standard publications. Finally, a comparative study of existing software frameworks supporting DNNs and future research directions in the area will be presented. From all presented works it could be deduced that the use of DNNs in healthcare is still in its early stages, there are strong initiatives in academia and industry to pursue healthcare projects based on DNNs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10.

Similar content being viewed by others

Availability of data and material

N/A.

Code availability

N/A.

References

  1. The Age of Intelligent Machines, Kurzweil, Ray, Cambridge, MA: MIT Press, 1990

  2. Russell S, Norvig Artificial intelligence: a modern approach, Prentice Hall.

  3. Artificial Intelligence: How knowledge is created, transferred, and used," Elsevier, Scopus Report https://www.elsevier.com/__data/assets/pdf_file/0010/823654/ACAD-RL-AS-RE-ai-report-WEB.pdf

  4. Nath V, Levinson S (2014) Autonomous robotics and deep learning, ISBN: 978–3–319–05603–6, Springer, 2014.

  5. Introduction to Machine Learning, Ethem Alpaydin, 3rd edition, MIT Press, 2015.

  6. Lee J-G et al (2017) Deep learning in medical imaging: general overview. Korean J Radiol 18(4):570–584

    Google Scholar 

  7. Deng L, Dong Y (2014) Deep learning: methods and applications. Found Trends® Signal Process 7(3–4):197–387

    MathSciNet  MATH  Google Scholar 

  8. Patterson J, Gibson A, O'Reilly (2017) Deep learning, Media- USA, 1st edition, 2017.

  9. Ball JE, Anderson DT, Chan CS (2017) Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community. J Appl Remote Sens 11(4):042609

    Google Scholar 

  10. Donghwoon K et al (2017) "A survey of deep learning-based network anomaly detection." Cluster Computing, Springer, pp 1–13, https://doi.org/10.1007/s10586-017-1117-8.

  11. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252

    MathSciNet  Google Scholar 

  12. Gulcehre C (2016) Deep learning software Links. http://deeplearning.net/software_links/. Accessed Feb 2019.

  13. Wang X, Lu L, Shin H, Kim L, Bagheri M, Nogues I, Yao J, Summers RM (2017) “Unsupervised joint mining of deep features and image labels for large-scale radiology image annotation and scene recognition,” IEEE Winter Conf Appl Comput Vis (WACV), pp 998–1007.

  14. Zhang X, Liu W, Dundar M, Badve S, Zhang S (2015) Towards large-scale histopathological image analysis: hashing-based image retrieval. IEEE Trans Med Imag 34(2):496–506

    Google Scholar 

  15. Zhang X, Xing F, Su H, Yang L, Zhang S (2015) High-throughput histopathological image analysis via robust cell segmentation and hashing. J Med Image Anal 26(1):306–315

    Google Scholar 

  16. Dhungel N, Carneiro G, Bradley AP (2015) Deep learning and structured prediction for the segmentation of mass in mammograms. Medical image computing and computer-assisted intervention–MICCAI 2015. Springer, Cham, pp 605–612

    Google Scholar 

  17. Dubrovina A, Kisilev P, Ginsburg B, Hashoul S, Kimmel R (2016) Computational mammography using deep neural networks. In: Workshop on deep learning in medical image analysis (DLMIA).

  18. Zheng Y (2015) Model based 3D cardiac image segmentation with marginal space learning. Medical image recognition, segmentation and parsing: methods, theories and applications. Elsevier, Amsterdam, pp 383–404

    Google Scholar 

  19. Avendi MR, Kheirkhah A, Jafarkhani H (2016) A combined deep-learning and deformable model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Med Image Anal 30:108–119

    Google Scholar 

  20. Zhen X, Wang Z, Islam A, Bhaduri M, Chan I, Li S (2016) Multi-scale deep networks and regression forests for direct bi-ventricular volume estimation. Med Image Anal 30:120–129

    Google Scholar 

  21. Wang S, Yin Y, Cao G, Wei B, Zheng Y, Yang G (2015) Hierarchical retinal blood vessel segmentation based on feature and ensemble learning. Neruocomputing 149:708–717

    Google Scholar 

  22. Li Q, Feng B, Xie L, Liang P, Zhang H, Wang T (2016) A cross-modality learning approach for vessel segmentation in retinal images. IEEE Trans Med Imag 35(1):109–118

    Google Scholar 

  23. Wu A, Xu Z, Gao M, Buty M, Mollura DJ (2016) Deep vessel tracking: a generalized probabilistic approach via deep learning. In: Proceedings of IEEE international symposium on biomedical, imaging, pp 1363–1367.

  24. Xing F, Yang L (2016) Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review. IEEE Rev Biomed Eng 9:234–263

    Google Scholar 

  25. Kaggle B (2017). Kaggle Data Science Bowl (2017) [Online] https: //www.kaggle.com/c/data-science-bowl-2017

  26. Shin H-C et al (2016) Deep convolutional neural networks for computer aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imag 35(5):1285–1298

    Google Scholar 

  27. Yang X et al. (2016) A deep learning approach for tumor tissue image classification. In: Proc. Int. Conf. Biomed. Eng., Calgary, Canada [Online] /https://doi.org/10.2316/P.2016.832-025.

  28. Shin H-C, Orton MR, Collins DJ, Doran SJ, Leach MO (2013) Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4D patient data. IEEE Trans Pattern Anal Mach Intell 35(8):1930–1943

    Google Scholar 

  29. Yao J, Li J, Summers RM (2009) Employing topographical height map in colonic polyp measurement and false positive reduction. Pattern Recogn 42(6):1029–1040

    Google Scholar 

  30. Roth HR et al (2016) Improving computer aided detection using convolutional neural networks and random view aggregation’’. IEEE Transact Med Imag 35(5):1170–1181

    Google Scholar 

  31. Roth H, Lu L et al (2017) Efficient false positive reduction in computer aided detection using convolutional neural networks and random view aggregation. In: Le L, Zheng Y, Carneiro G, Yang L (eds) Deep learning and convolutional neural networks for medical image computing. Springer, Cham

    Google Scholar 

  32. Ker J et al (2018) Deep learning applications in medical image analysis. IEEE Access 6:9375–9389

    Google Scholar 

  33. Veta M, Pluim J, van Diest P, Viergever M (2014) Breast cancer histopathology image analysis: a review. IEEE Trans Biomed Eng 61(5):1400–1411

    Google Scholar 

  34. Kuse M, Wang Y-F, Kalasannavar V, Khan M, Rajpoot N (2011) Local isotropic phase symmetry measure for detection of beta cells and lymphocytes. J Pathol Inform 2(2):2

    Google Scholar 

  35. Al-Kofahi Y, Lassoued W, Lee W, Roysam B (2010) Improved automatic detection and segmentation of cell nuclei in histopathology images. IEEE Trans Biomed Eng 57(4):841–852

    Google Scholar 

  36. Vink JP, Van Leeuwen M, Van Deurzen C, De Haan G (2013) Efficient nucleus detector in histopathology images. J Microsc 249(2):124–135

    Google Scholar 

  37. Ali S, Madabhushi A (2012) An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery. IEEE Trans Med Imag 31(7):1448–1460

    Google Scholar 

  38. Cruz-Roa AA, Ovalle JEA, Madabhushi A, Osorio FAG (2013) A Deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection, in Medical image computing and computer-assisted intervention–MICCAI, Springer, pp. 403–410.

  39. Chao L, Xinggang W, Wenyu L, Longin L (2018) Deep mitosi: mitosis detection via deep detection, verification, and segmentation. Med Image Anal 45:121–133

    Google Scholar 

  40. MITOS-ATYPIA-14 (2014) Mitos-Atypia-14-dataset. https://mitos-atypia-14.grand-challenge.orgldataset/Online; accessed 03.03.2018.

  41. Li W, Li J (2018) Local deep field for electrocardiogram beat classification. IEEE Sens J 18(4):1656–1664

    Google Scholar 

  42. Shehata M et al (2019) Computer-aided diagnostic system for early detection of acute renal transplant rejection using diffusion-weighted MRI. IEEE Trans Biomed Eng 66(2):539–552

    MathSciNet  Google Scholar 

  43. Hamilton PW, Bankhead P, Wang YH, Hutchinson R, Kieran D, McArt DG, James J, SaltoTellez M (2014) Digital pathology and image analysis in tissue biomarker research. Methods 70(1):59–73

    Google Scholar 

  44. Rimm DL (2011) C-path: awatson-like visit to the pathology lab. Sci Trans Med 3:108

    Google Scholar 

  45. Duraisamy S, Emperumal S (2017) Computer-aided mammogram diagnosis system using deep learning convolutional fully complex-valued relaxation neural network classifier. Sci Trans Med 11(8):656–662

    Google Scholar 

  46. American Cancer Society. Cancer facts & figures 2016. Atlanta, American Cancer Society 2016. http://www.cancer.org/acs/groups/content/@research/documents/document/acspc-047079.pdf.

  47. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Google Scholar 

  48. Gutman D et al (2016) Skin lesion analysis toward melanoma detection. In: International symposium on biomedical imaging (ISBI), (International Skin Imaging Collaboration (ISIC), 2016).

  49. Esteva A et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–126

    Google Scholar 

  50. Ramlakhan K, Shang Y (2011) A mobile automated skin lesion classification system. In: 23rd IEEE international conference on tools with artificial intelligence (ICTAI-2011), pp. 138–141.

  51. Celebi M, Schaefer G (2013) Color Medical Image Analysis Springer, pp 63–86.

  52. B. T. Society (1999) The diagnosis, assessment and treatment of diffuse parenchymal lung disease in adults. Thorax, 54 (1).

  53. Anthimopoulos M, Christodoulidis S, Christe A, Mougiakakou S (2014) Classification of interstitial lung disease patterns using local DCT features and random forest. In Proc. 36th Annual Int. Conf. IEEE Eng. Med. Biol. Soc., pp 6040–6043.

  54. Li Q, Cai W, Feng DD (2014) Lung image patch classification with automatic feature learning. In Proc. 36th Annual Int. Conf. IEEE Eng. Med. Biol. Soc., pp 6079–6082.

  55. Li Q et al (2014) Medical image classification with convolutional neural network. In: Proc. 13th Int. Conf. Control Automat. Robot. Vis., pp 844–848.

  56. Anthimopoulos M, Ebner L, Christe A, Mougiakakou S (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imag 35(5):1207–1216

    Google Scholar 

  57. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference learning representation, San Diego, USA.

  58. Krizhevsky A, Sutskever I, and Hinton G (2012) ImageNet classification with deep convolutional neural networks, Advanced Neural Inference Processing Systems.

  59. https://www.alzheimers.net/resources/alzheimers-statistics/ last accessed date: 1st March, 2019

  60. Suk HI, Lee SW, Shen DG (2014) Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. Neuroimage 101:569–582

    Google Scholar 

  61. Hosseini-Asl E, Keynton R, El-Baz A (2016) Alzheimer’s disease diagnostics by adaptation of 3D convolutional network. In: Proc. 2016 IEEE Int. Conf. Image Processing (ICIP), Phoenix, AZ, USA, pp 126–130.

  62. Liu J, Pan Y, Li M, Chen Z, Tang L, Wang J (2018) Applications of deep learning to MRI images: a survey. Big Data Mining and Analytics 1(1):1–18

    Google Scholar 

  63. Zikic D, Y. Ioannou Y, Criminisi A, Brown M (2014) Segmentation of brain tumor tissues with convolutional neural networks. In: Proceedings MICCAI Workshop on Multimodal Brain Tumor Segmentation Challenge, Boston, USA, pp 36–39

  64. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31

    Google Scholar 

  65. Pereira S, Pinto A, Alves V, Silva CA (2016) Braintumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imag 35(5):1240–1251

    Google Scholar 

  66. Kleesiek J, Urban G, Hubert A, Schwarz D, MaierHein K, Bendszus M, Biller A (2016) Deep MRI brain extraction: a 3D convolutional neural network for skull stripping. Neuroimage 129:460–469

    Google Scholar 

  67. Ngo TA, Lu Z, Carneiro G (2017) Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance. Med Image Anal 35:159–171

    Google Scholar 

  68. Havaei M et al (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31

    Google Scholar 

  69. Davy A, Havaei M, Warde-Farley D, Biard A, Tran L, Jodoin PM, Courville A, Larochelle H, Pal C, Bengio Y (2014) Brain tumor segmentation with deep neural networks. In: Proc. of BRATS-MICCAI.

  70. Zikic D, Ioannou Y, Brown M, Criminisi A (2014) Segmentation of brain tumor tissues with convolutional neural networks. In: Proc. of BRATS-MICCAI.

  71. Urban G, Bendszus M, Hamprecht F, Kleesiek J (2014) Multi-modal brain tumor segmentation using deep convolutional neural networks. In: Proc. of BRATS-MICCAI.

  72. Goodfellow I.J. et al. (2013) Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214

  73. Rajchl M et al (2017) DeepCut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE Trans Med Imag 36(2):674–683

    Google Scholar 

  74. Wang G et al (2018) Interactive medical image segmentation using deep learning with image specific fine tuning. IEEE Trans Med Imag 37(7):1562–1572

    Google Scholar 

  75. Menze BH et al (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imag 34(10):1993–2024

    Google Scholar 

  76. Roth HR et al (2014) A new representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI. Springer, Cham, pp 520–527

    Google Scholar 

  77. Kostantinos K et al (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis, Elsevier, Amsterdam

    Google Scholar 

  78. Sun W, Tseng TB, Zhang J, Qian W (2017) Enhancing deep convolutional neural network scheme for breast cancer diagnosis with unlabeled data. Comput Med Imag Graph 57:4–9

    Google Scholar 

  79. Sajjad M, Khan S, Muhammad K, Wu W, Ullah A, Baik SW (2019) Multi-grade brain tumor classification using deep CNN with extensive data augmentation. J Comput Sci 30:174–182

    Google Scholar 

  80. Jia Y et al (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia.

  81. Tianqi C et al (2015) MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems, In: Neural information processing systems, workshop on machine learning systems.

  82. Abadi M et al. (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint arXiv:1603.04467.

  83. Vedaldi A, Lenc K (2015) MatConvNet: convolutional neural networks for MATLAB. In Proc. 23rd ACM Int. Conf. Multimedia, Brisbane, Australia, pp 689–692.

  84. Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a matlab-like environment for machine learning. In BigLearn, NIPS Workshop, No. EPFL-CONF-192376.

  85. Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, Turian J, Warde-Farley D, Bengio Y (2010). Theano: a CPU and GPU math expression compiler. In: Proceedings of the python for scientific computing conference (SciPy).

  86. Ota K et al (2017) Deep learning for mobile multimedia: a survey. ACM Trans Multimed Comput Commun Appl 34:1–32

    Google Scholar 

  87. Choi W et al (2016) Hybrid network-on-chip architectures for accelerating deep learning kernels on heterogeneous many core platforms. In: Proceeding of compliers, architectures, and sythesis of embedded systems (CASES), 2016 IEEE International Conference.

  88. Zhang J, Yeung SH, Shu Y, He B, Wang W. (2019) Efficient memory management for GPU-based deep learning systems; arXiv preprint arXiv:1903.06631.

  89. Zhao H, Han Z, Yang Z, Zhang Q, Yang F, Zhou L, Yang M, Lau FC, Wang Y, Xiong Y, et al. Hived (2020) sharing a {GPU} cluster for deep learning with guarantees, 14th USENIX symposium on operating systems design and implementation (OSDI 20), pp 515–532.

  90. Lin Y, Jiang Z, Gu J, Li W, Dhar S, Ren H, Khailany B, Pan DZ (2020) Dream place eep learning toolkit-enabled GPU acceleration for modern VLSI placemen. IEEE Trans Comput Aid Des Integr Circuits Syst 40:748–61

    Google Scholar 

  91. Hossain S, Lee DJ (2019) Deep learning-based real-time multiple-object detection and tracking from aerial imagery via a flying robot with GPU-based embedded devices. Sensors 19(15):3371

    Google Scholar 

  92. Castro FM, Guil N, Marin-Jimenez MJ, Perez-Serrano J, Ujaldon M (2019) Energy-based tuning of convolutional neural networks on multi-GPUs. Concurr Comput Pract Exp 31(21):4786

    Google Scholar 

  93. Alzubaidi et al (2021) Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J Big Data 8:53. https://doi.org/10.1186/s40537-021-00444-8

    Article  Google Scholar 

  94. Szegedy C et al (2017) Inception-v4, inception-resnet and the impact of residual connections on learning, Thirty-first AAAI conference on artificial intelligence

  95. He K et al (2016) Identity mappings in deep residual networks. Springer, European conference on computer vision

    Google Scholar 

  96. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation, International CONFERENCE ON MEDICAL image computing and computer-assisted intervention, Springer.

  97. Gkioxari G, Girshick R, Malik J (2015) Contextual action recognition with R-CNN, Proceedings of the IEEE international conference on computer vision.

  98. Ren S et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks, NIPS'15: Proceedings of the 28th international conference on neural information processing systems, Vol 1, pp 91–99.

  99. Du J (2018) Understanding of object detection based on CNN family and YOLO. J Phys: Conf Ser 1004:012029

    Google Scholar 

  100. Liu W et al (2016) SSD: single shot multibox detector. Springer, European conference on computer vision

    Google Scholar 

  101. Khan SU, Islam N, Jan Z, Din IU, Rodrigues JJPC (2019) A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recogn Lett 125:1–6

    Google Scholar 

  102. Zhang Q, Yang LT, Chen Z, Li P, Bu F (2018) An adaptive dropout deep computation model for industrial IoT big data learning with crowdsourcing to cloud computing. IEEE Trans Ind Inf 15(4):2330–2337

    Google Scholar 

  103. Hang Yu, Yang LT, Zhang Q, David Armstrong M, Deen J (2021) Convolutional neural networks for medical image analysis: state-of-theart, comparisons, improvement and perspectives. Neurocomputing 444:92–110

    Google Scholar 

  104. Altaf F et al (2019) Going deep in medical image analysis: concepts, methods, challenges and future directions. IEEE Access 7:99540

    Google Scholar 

  105. Baltrusaitis T, Ahuja C, Morency L (2019) Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Learn 41(2):423–443

    Google Scholar 

Download references

Funding

N/A.

Author information

Authors and Affiliations

  1. Knowledge Based Systems Dept., Informatics Research Institute, City of Scientific Research and Technological Applications, Alexandria, Egypt

    Mohamed A. Abdou

Authors
  1. Mohamed A. Abdou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N/A.

Corresponding author

Correspondence to Mohamed A. Abdou.

Ethics declarations

Conflict of interest

The author declares that there is not conflicts of interest.

Ethical approval

N/A.

Consent to participate

The author has consented to the submission of this survey to the journal.

Consent for publication

The author has consented to the publication of this survey to the journal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdou, M.A. Literature review: efficient deep neural networks techniques for medical image analysis. Neural Comput & Applic 34, 5791–5812 (2022). https://doi.org/10.1007/s00521-022-06960-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-06960-9

Keywords

Search

Navigation