Deep semantic segmentation of natural and medical images: a review

Abstract

The semantic image segmentation task consists of classifying each pixel of an image into an instance, where each instance corresponds to a class. This task is a part of the concept of scene understanding or better explaining the global context of an image. In the medical image analysis domain, image segmentation can be used for image-guided interventions, radiotherapy, or improved radiological diagnostics. In this review, we categorize the leading deep learning-based medical and non-medical image segmentation solutions into six main groups of deep architectural, data synthesis-based, loss function-based, sequenced models, weakly supervised, and multi-task methods and provide a comprehensive review of the contributions in each of these groups. Further, for each group, we analyze each variant of these groups and discuss the limitations of the current approaches and present potential future research directions for semantic image segmentation.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

References

  1. Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. https://github.com/matterport/Mask_RCNN

  2. Abhishek K, Hamarneh G (2019) Mask2Lesion: mask-constrained adversarial skin lesion image synthesis. In: Medical image computing and computer-assisted intervention workshop on simulation and synthesis in medical imaging, pp 71–80

  3. Abhishek K, Hamarneh G, Drew MS (2020) Illumination-based transformations improve skin lesion segmentation in dermoscopic images. arXiv:200310111

  4. Adams RA, Fournier JJ (2003) Sobolev spaces. Elsevier, Amsterdam

    Google Scholar 

  5. Afshari S, BenTaieb A, Mirikharaji Z, Hamarneh G (2019) Weakly supervised fully convolutional network for PET lesion segmentation. In: Medical imaging 2019: image processing, international society for optics and photonics, vol 10949, p 109491K

  6. Alom MZ, Yakopcic C, Hasan M, Taha TM, Asari VK (2019) Recurrent residual U-Net for medical image segmentation. J Med Imag 6(1):14006

    Article  Google Scholar 

  7. Amirul Islam M, Rochan M, Bruce ND, Wang Y (2017) Gated feedback refinement network for dense image labeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3751–3759

  8. Amit Y (2019) Deep learning with asymmetric connections and hebbian updates. Front Comput Neurosci. https://doi.org/10.3389/fncom.2019.00018

    Article  Google Scholar 

  9. Anantharaman R, Velazquez M, Lee Y (2018) Utilizing Mask R-CNN for detection and segmentation of oral diseases. In: 2018 IEEE international conference on bioinformatics and biomedicine, pp 2197–2204

  10. Badrinarayanan V, Handa A, Cipolla R (2015) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:151100561

  11. Bai W, Suzuki H, Qin C, Tarroni G, Oktay O, Matthews PM, Rueckert D (2018) Recurrent neural networks for aortic image sequence segmentation with sparse annotations. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 586–594

  12. Bellec G, Scherr F, Hajek E, Salaj D, Legenstein R, Maass W (2019) Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets. arXiv:190109049

  13. Bengio Y, Frasconi P (1994) Credit assignment through time: alternatives to backpropagation. In: Advances in neural information processing systems, pp 75–82

  14. Benoit-Cattin H, Collewet G, Belaroussi B, Saint-Jalmes H, Odet C (2005) The SIMRI project: a versatile and interactive MRI simulator. J Magn Reson 173(1):97–115. https://doi.org/10.1016/j.jmr.2004.09.027

    Article  Google Scholar 

  15. BenTaieb A, Hamarneh G (2016) Topology aware fully convolutional networks for histology gland segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 460–468

  16. Berman M, Blaschko MB, Triki AR, Yu J (2018a) Yes, IoU loss is submodular-as a function of the mispredictions. arXiv:180901845

  17. Berman M, Rannen Triki A, Blaschko MB (2018b) The Lovász-Softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4413–4421

  18. Bischke B, Helber P, Folz J, Borth D, Dengel A (2019) Multi-task learning for segmentation of building footprints with deep neural networks. In: 2019 IEEE international conference on image processing. IEEE, pp 1480–1484

  19. Bonta LR, Kiran NU (2019) Efficient segmentation of medical images using dilated residual networks. In: Computer aided intervention and diagnostics in clinical and medical images. Springer, pp 39–47

  20. Borji A, Cheng MM, Hou Q, Jiang H, Li J (2019) Salient object detection: a survey. Comput Vis Media 5(2):117–150. https://doi.org/10.1007/s41095-019-0149-9

    Article  Google Scholar 

  21. Brostow GJ, Shotton J, Fauqueur J, Cipolla R (2008) Segmentation and recognition using structure from motion point clouds. In: Lecture notes in computer science. Springer, Berlin, pp 44–57. https://doi.org/10.1007/978-3-540-88682-2_5

  22. Brostow GJ, Fauqueur J, Cipolla R (2009) Semantic object classes in video: a high-definition ground truth database. Pattern Recognit Lett 30(2):88–97. https://doi.org/10.1016/j.patrec.2008.04.005

    Article  Google Scholar 

  23. Brügger R, Baumgartner CF, Konukoglu E (2019) A partially reversible U-Net for memory-efficient volumetric image segmentation. arXiv:190606148

  24. Caliva F, Iriondo C, Martinez AM, Majumdar S, Pedoia V (2019) Distance map loss penalty term for semantic segmentation. In: International conference on medical imaging with deep learning

  25. Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

    MathSciNet  Article  Google Scholar 

  26. Chaichulee S, Villarroel M, Jorge J, Arteta C, Green G, McCormick K, Zisserman A, Tarassenko L (2017) Multi-task convolutional neural network for patient detection and skin segmentation in continuous non-contact vital sign monitoring. In: 2017 12th IEEE international conference on automatic face & gesture recognition. IEEE, pp 266–272

  27. Chakravarty A, Sivaswamy J (2018) RACE-Net: a recurrent neural network for biomedical image segmentation. IEEE J Biomed Health Inform 23(3):1151–1162

    Article  Google Scholar 

  28. Challenge G (2020) Grand challenges in biomedical image analysis. https://grand-challenge.org/challenges/

  29. Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 3–13

  30. Chen LC, Yang Y, Wang J, Xu W, Yuille AL (2016) Attention to scale: scale-aware semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3640–3649

  31. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017a) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  32. Chen LC, Papandreou G, Schroff F, Adam H (2017b) Rethinking atrous convolution for semantic image segmentation. arXiv:170605587

  33. Chen LC, Collins M, Zhu Y, Papandreou G, Zoph B, Schroff F, Adam H, Shlens J (2018a) Searching for efficient multi-scale architectures for dense image prediction. In: Advances in neural information processing systems, pp 8699–8710

  34. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018b) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision, pp 801–818

  35. Chen X, Williams BM, Vallabhaneni SR, Czanner G, Williams R, Zheng Y (2019) Learning active contour models for medical image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 11632–11640

  36. Cherian A, Sullivan A (2019) Sem-GAN: semantically-consistent image-to-image translation. In: 2019 IEEE winter conference on applications of computer vision (WACV). IEEE. https://doi.org/10.1109/wacv.2019.00196

  37. Choi J, Kim T, Kim C (2019) Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 6830–6840

  38. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  39. Cireşan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3642–3649

  40. Cireşan DC, Meier U, Masci J, Gambardella LM, Schmidhuber J (2011) High-performance neural networks for visual object classification. arXiv:11020183

  41. Cohen JP, Luck M, Honari S (2018) Distribution matching losses can hallucinate features in medical image translation. In: Medical image computing and computer assisted intervention – MICCAI 2018. Springer, pp 529–536. https://doi.org/10.1007/978-3-030-00928-1_60

  42. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  43. Costa P, Galdran A, Meyer MI, Abràmoff MD, Niemeijer M, Mendonça AM, Campilho A (2017) Towards adversarial retinal image synthesis. arXiv:170108974

  44. Couprie C, Farabet C, Najman L, LeCun Y (2013) Indoor semantic segmentation using depth information. arXiv:13013572

  45. Czarnecki WM, Osindero S, Jaderberg M, Swirszcz G, Pascanu R (2017) Sobolev training for neural networks. In: Advances in neural information processing systems, pp 4278–4287

  46. Dai W, Dong N, Wang Z, Liang X, Zhang H, Xing EP (2018) SCAN: structure correcting adversarial network for organ segmentation in chest X-rays. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 263–273. https://doi.org/10.1007/978-3-030-00889-5_30

  47. Drobnjak I, Gavaghan D, Süli E, Pitt-Francis J, Jenkinson M (2006) Development of a functional magnetic resonance imaging simulator for modeling realistic rigid-body motion artifacts. Magn Reson Med 56(2):364–380. https://doi.org/10.1002/mrm.20939

    Article  Google Scholar 

  48. Drobnjak I, Pell GS, Jenkinson M (2010) Simulating the effects of time-varying magnetic fields with a realistic simulated scanner. Magn Reson Imaging 28(7):1014–1021. https://doi.org/10.1016/j.mri.2010.03.029

    Article  Google Scholar 

  49. Drozdzal M, Chartrand G, Vorontsov E, Shakeri M, Di Jorio L, Tang A, Romero A, Bengio Y, Pal C, Kadoury S (2018) Learning normalized inputs for iterative estimation in medical image segmentation. Med Image Anal 44:1–13

    Article  Google Scholar 

  50. Everingham M, Gool LV, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  51. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2012) The PASCAL visual object classes challenge 2012 (VOC2012) results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html

  52. Everingham M, Eslami SA, Van Gool L, Williams CK, Winn J, Zisserman A (2015) The PASCAL visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136

    Article  Google Scholar 

  53. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019a) Dual attention network for scene segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3146–3154

  54. Fu J, Liu J, Wang Y, Zhou J, Wang C, Lu H (2019b) Stacked deconvolutional network for semantic segmentation. IEEE Trans Image Process

  55. Galdran A, Alvarez-Gila A, Meyer MI, Saratxaga CL, Araújo T, Garrote E, Aresta G, Costa P, Mendonça AM, Campilho A (2017) Data-driven color augmentation techniques for deep skin image analysis. arXiv:170303702

  56. Gamage H, Wijesinghe W, Perera I (2019) Instance-based segmentation for boundary detection of neuropathic ulcers through Mask-RCNN. In: International conference on artificial neural networks. Springer, pp 511–522

  57. Gao Y, Phillips JM, Zheng Y, Min R, Fletcher PT, Gerig G (2018) Fully convolutional structured LSTM networks for joint 4D medical image segmentation. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 1104–1108

  58. Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Martinez-Gonzalez P, Garcia-Rodriguez J (2018) A survey on deep learning techniques for image and video semantic segmentation. Appl Soft Comput 70:41–65

    Article  Google Scholar 

  59. Glatard T, Lartizien C, Gibaud B, da Silva RF, Forestier G, Cervenansky F, Alessandrini M, Benoit-Cattin H, Bernard O, Camarasu-Pop S, Cerezo N, Clarysse P, Gaignard A, Hugonnard P, Liebgott H, Marache S, Marion A, Montagnat J, Tabary J, Friboulet D (2013) A virtual imaging platform for multi-modality medical image simulation. IEEE Trans Med Imaging 32(1):110–118. https://doi.org/10.1109/tmi.2012.2220154

    Article  Google Scholar 

  60. Goceri E (2019a) Challenges and recent solutions for image segmentation in the era of deep learning. In: 2019 ninth international conference on image processing theory, tools and applications (IPTA). IEEE. https://doi.org/10.1109/ipta.2019.8936087

  61. Goceri E (2019b) Diagnosis of alzheimerś disease with sobolev gradient-based optimization and 3d convolutional neural network. Int J Numer Methods Biomed Eng. https://doi.org/10.1002/cnm.3225

    MathSciNet  Article  Google Scholar 

  62. Goceri E (2020) CapsNet topology to classify tumours from brain images and comparative evaluation. IET Image Process 14(5):882–889. https://doi.org/10.1049/iet-ipr.2019.0312

    Article  Google Scholar 

  63. Goceri E, Goceri N (2017) Deep learning in medical image analysis: recent advances and future trends. In: Proceedings of the IADIS international conference computer graphics, visualization, computer vision and image processing (CGVCVIP) 2017, pp 305–310

  64. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  65. Gros C, De Leener B, Badji A, Maranzano J, Eden D, Dupont SM, Talbott J, Zhuoquiong R, Liu Y, Granberg T et al (2019) Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks. Neuroimage 184:901–915

    Article  Google Scholar 

  66. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging 38:2281–2292

    Article  Google Scholar 

  67. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. In: Advances in neural information processing systems, pp 5767–5777

  68. Guo Y, Liu Y, Georgiou T, Lew MS (2018) A review of semantic segmentation using deep neural networks. Int J Multimed Inf Retr 7(2):87–93

    Article  Google Scholar 

  69. Hamarneh G, Jassi P (2010) VascuSynth: simulating vascular trees for generating volumetric image data with ground-truth segmentation and tree analysis. Comput Med Imaging Graphics 34(8):605–616. https://doi.org/10.1016/j.compmedimag.2010.06.002

    Article  Google Scholar 

  70. Han C, Murao K, Satoh S, Nakayama H (2019) Learning more with less: GAN-based medical image augmentation. Med Imaging Technol 37(3):137–142

    Google Scholar 

  71. Han S, Liu X, Mao H, Pu J, Pedram A, Horowitz MA, Dally WJ (2016) EIE: efficient inference engine on compressed deep neural network. In: 2016 ACM/IEEE 43rd annual international symposium on computer architecture. IEEE, pp 243–254

  72. Haralick RM, Shapiro LG (1992) Computer and robot vision. Addison-Wesley, Boston

    Google Scholar 

  73. Harrison R, Lewellen T (2012) The SimSET program. In: Monte Carlo calculations in nuclear medicine, Second Edition. Taylor & Francis, pp 87–110. https://doi.org/10.1201/b13073-7

  74. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  75. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  76. He T, Guo J, Wang J, Xu X, Yi Z (2019) Multi-task learning for the segmentation of thoracic organs at risk in CT images. In: SegTHOR@ISBI

  77. Hesamian MH, Jia W, He X, Kennedy P (2019) Deep learning techniques for medical image segmentation: achievements and challenges. J Digit Imaging 32:582–596

    Article  Google Scholar 

  78. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  79. Honari S, Yosinski J, Vincent P, Pal C (2016) Recombinator networks: learning coarse-to-fine feature aggregation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5743–5752

  80. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:170404861

  81. Hu J, Shen L, Sun G (2018a) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  82. Hu Y, Chen Z, Lin W (2018b) RGB-D semantic segmentation: a review. In: 2018 IEEE international conference on multimedia & expo workshops. IEEE, pp 1–6

  83. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  84. Huang SW, Lin CT, Chen SP, Wu YY, Hsu PH, Lai SH (2018) AugGAN: Cross domain adaptation with GAN-based data augmentation. In: Proceedings of the European conference on computer vision (ECCV). Springer, Berlin, pp 731–744. https://doi.org/10.1007/978-3-030-01240-3_44

  85. Huo Y, Xu Z, Bao S, Assad A, Abramson RG, Landman BA (2018) Adversarial synthesis learning enables segmentation without target modality ground truth. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 1217–1220

  86. Hussain MA, Amir-Khalili A, Hamarneh G, Abugharbieh R (2017) Segmentation-free kidney localization and volume estimation using aggregated orthogonal decision CNNs. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 612–620

  87. Isensee F, Petersen J, Klein A, Zimmerer D, Jaeger PF, Kohl S, Wasserthal J, Koehler G, Norajitra T, Wirkert S, et al. (2019) nnU-Net: self-adapting framework for U-Net-based medical image segmentation. In: Bildverarbeitung für die Medizin 2019. Springer, pp 22–22

  88. Jaeger PF, Kohl SA, Bickelhaupt S, Isensee F, Kuder TA, Schlemmer HP, Maier-Hein KH (2018) Retina U-Net: embarrassingly simple exploitation of segmentation supervision for medical object detection. arXiv:181108661

  89. Jégou S, Drozdzal M, Vazquez D, Romero A, Bengio Y (2017) The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 11–19

  90. Jensen J, Svendsen N (1992) Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers. IEEE Trans Ultrason Ferroelectr Freq Control 39(2):262–267. https://doi.org/10.1109/58.139123

    Article  Google Scholar 

  91. Jensen JA (1996) Field: A program for simulating ultrasound systems. In: 10th Nordic-Baltic conference on biomedical imaging, Volume 34, Supplement 1, Part 1, pp 351–353

  92. Jin D, Xu Z, Tang Y, Harrison AP, Mollura DJ (2018) CT-realistic lung nodule simulation from 3D conditional generative adversarial networks for robust lung segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 732–740

  93. Jin W, Fatehi M, Abhishek K, Mallya M, Toyota B, Hamarneh G (2020) Artificial intelligence in glioma imaging: challenges and advances. J Neural Eng 17(2):021002. https://doi.org/10.1088/1741-2552/ab8131

    Article  Google Scholar 

  94. Johnson JW (2018) Adapting mask R-CNN for automatic nucleus segmentation. arXiv:180500500

  95. Karimi D, Salcudean SE (2019) Reducing the Hausdorff distance in medical image segmentation with convolutional neural networks. arXiv:190410030

  96. Karimi D, Dou H, Warfield SK, Gholipour A (2019) Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. arXiv:191202911

  97. Ke R, Bugeau A, Papadakis N, Schütz P, Schönlieb CB (2019) A multi-task U-Net for segmentation with lazy labels. arXiv:1906.12177

  98. Kervadec H, Bouchtiba J, Desrosiers C, Granger E, Dolz J, Ben Ayed I (2019a) Boundary loss for highly unbalanced segmentation. In: Proceedings of the 2nd international conference on medical imaging with deep learning, PMLR, London, United Kingdom, proceedings of machine learning research, vol 102, pp 285–296. http://proceedings.mlr.press/v102/kervadec19a.html

  99. Kervadec H, Dolz J, Tang M, Granger E, Boykov Y, Ayed IB (2019) Constrained-CNN losses for weakly supervised segmentation. Med Image Anal 54:88–99. https://doi.org/10.1016/j.media.2019.02.009

    Article  Google Scholar 

  100. Khosravan N, Mortazi A, Wallace M, Bagci U (2019) PAN: projective adversarial network for medical image segmentation. arXiv:190604378

  101. Kim B, Ye JC (2019) Multiphase level-set loss for semi-supervised and unsupervised segmentation with deep learning. arXiv:190402872

  102. Kim HE, Hwang S (2016) Deconvolutional feature stacking for weakly-supervised semantic segmentation. arXiv:160204984

  103. Kim YD, Park E, Yoo S, Choi T, Yang L, Shin D (2015) Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv:151106530

  104. Kopelowitz E, Engelhard G (2019) Lung nodules detection and segmentation using 3D Mask R-CNN. arXiv:1907.07676

  105. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  106. Kuntimad G, Ranganath H (1999) Perfect image segmentation using pulse coupled neural networks. IEEE Trans Neural Netw 10(3):591–598. https://doi.org/10.1109/72.761716

    Article  Google Scholar 

  107. Lateef F, Ruichek Y (2019) Survey on semantic segmentation using deep learning techniques. Neurocomputing 338:321–348

    Article  Google Scholar 

  108. Le TLT, Thome N, Bernard S, Bismuth V, Patoureaux F (2019) Multitask classification and segmentation for cancer diagnosis in mammography. arXiv:190905397

  109. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  110. Lee DH, Zhang S, Fischer A, Bengio Y (2015) Difference target propagation. In: Machine learning and knowledge discovery in databases. Springer, pp 498–515. https://doi.org/10.1007/978-3-319-23528-8_31

  111. Lee J, Kim E, Lee S, Lee J, Yoon S (2019) Ficklenet: weakly and semi-supervised semantic image segmentation using stochastic inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5267–5276

  112. Leroux S, Molchanov P, Simoens P, Dhoedt B, Breuel T, Kautz J (2018) IamNN: iterative and adaptive mobile neural network for efficient image classification. arXiv:180410123

  113. Li H, Xiong P, An J, Wang L (2018) Pyramid attention network for semantic segmentation. arXiv:180510180

  114. Li H, Li J, Lin X, Qian X (2019a) Pancreas segmentation via spatial context based U-Net and bidirectional LSTM. arXiv:190300832

  115. Li S, Dong M, Du G, Mu X (2019b) Attention dense-U-Net for automatic breast mass segmentation in digital mammogram. IEEE Access 7:59037–59047

    Article  Google Scholar 

  116. Li X, Liu Z, Luo P, Change Loy C, Tang X (2017) Not all pixels are equal: difficulty-aware semantic segmentation via deep layer cascade. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3193–3202

  117. Li X, Yu L, Chen H, Fu CW, Heng PA (2019c) Transformation consistent self-ensembling model for semi-supervised medical image segmentation. arXiv:190300348

  118. Lian S, Luo Z, Zhong Z, Lin X, Su S, Li S (2018) Attention guided U-Net for accurate iris segmentation. J Vis Commun Image Represent 56:296–304

    Article  Google Scholar 

  119. Lin D, Ji Y, Lischinski D, Cohen-Or D, Huang H (2018) Multi-scale context intertwining for semantic segmentation. In: Proceedings of the European conference on computer vision, pp 603–619

  120. Lin G, Milan A, Shen C, Reid I (2017a) RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1925–1934

  121. Lin TY, Goyal P, Girshick R, He K, Dollár P (2017b) Focal loss for dense object detection. arXiv:170802002

  122. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  123. Liu C, Chen LC, Schroff F, Adam H, Hua W, Yuille AL, Fei-Fei L (2019a) Auto-deeplab: hierarchical neural architecture search for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 82–92

  124. Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, Pietikäinen M (2019b) Deep learning for generic object detection: a survey. Int J Comput Vis 128(2):261–318. https://doi.org/10.1007/s11263-019-01247-4

    Article  Google Scholar 

  125. Liu Y, Perona P, Meister M (2019c) Panda: panoptic data augmentation. arXiv:191112317

  126. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  127. Luc P, Couprie C, Chintala S, Verbeek J (2016) Semantic segmentation using adversarial networks. arXiv:161108408

  128. Luo P, Wang G, Lin L, Wang X (2017) Deep dual learning for semantic image segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 2718–2726

  129. Ma WDK, Lewis J, Kleijn WB (2019) The hsic bottleneck: Deep learning without back-propagation. arXiv:190801580

  130. Marion A, Forestier G, Benoit-Cattin H, Camarasu-Pop S, Clarysse P, da SilvaRF, Gibaud B, Glatard T, Hugonnard P, Lartizien C, Liebgott H, Specovius S,Tabary J, Valette S, Friboulet D (2011) Multi-modality medical image simulation of biological models with the virtual imaging platform (VIP). In: 2011 24th international symposium on computer-based medical systems(CBMS). IEEE. https://doi.org/10.1109/cbms.2011.5999141

  131. Mehta S, Mercan E, Bartlett J, Weaver D, Elmore JG, Shapiro L (2018) Y-Net: Joint segmentation and classification for diagnosis of breast biopsy images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 893–901

  132. Meyer BJ, Harwood B, Drummond T (2018) Deep metric learning and image classification with nearest neighbour Gaussian kernels. In: 2018 25th IEEE international conference on image processing. IEEE, pp 151–155

  133. Milletari F, Navab N, Ahmadi SA (2016) V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision. IEEE, pp 565–571

  134. Mirikharaji Z, Hamarneh G (2018) Star shape prior in fully convolutional networks for skin lesion segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 737–745

  135. Mirikharaji Z, Yan Y, Hamarneh G (2019) Learning to segment skin lesions from noisy annotations. In: International workshop on medical image learning with less labels and imperfect data

  136. Moeskops P, Veta M, Lafarge MW, Eppenhof KA, Pluim JP (2017) Adversarial training and dilated convolutions for brain MRI segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 56–64

  137. Mohajerani S, Asad R, Abhishek K, Sharma N, van Duynhoven A, Saeedi P (2019) Cloudmaskgan: a content-aware unpaired image-to-image translation algorithm for remote sensing imagery. In: 2019 IEEE international conference on image processing. IEEE, pp 1965–1969

  138. Mohanty SP (2018) Crowdai mapping challenge 2018: baseline with mask RCNN. https://github.com/crowdai/crowdai-mapping-challenge-mask-rcnn

  139. Mottaghi R, Chen X, Liu X, Cho NG, Lee SW, Fidler S, Urtasun R, Yuille A (2014) The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 891–898

  140. Mukherjee S, Cheng I, Miller S, Guo T, Chau V, Basu A (2019) A fast segmentation-free fully automated approach to white matter injury detection in preterm infants. Med Biol Eng Comput 57(1):71–87

    Article  Google Scholar 

  141. Mumford D, Shah J (1989) Optimal approximations by piecewise smooth functions and associated variational problems. Commun Pure Appl Math 42(5):577–685

    MathSciNet  Article  Google Scholar 

  142. Neff T, Payer C, Stern D, Urschler M (2017) Generative adversarial network based synthesis for supervised medical image segmentation. In: Proceedings of OAGM and ARW joint workshop

  143. Neff T, Payer C, Štern D, Urschler M (2018) Generative adversarial networks to synthetically augment data for deep learning based image segmentation. In: Proceedings of the OAGM workshop 2018: medical image analysis. Verlag der Technischen Universität Graz, pp 22–29

  144. Nguyen HH, Fang F, Yamagishi J, Echizen I (2019) Multi-task learning for detecting and segmenting manipulated facial images and videos. arXiv:1906.06876

  145. Ni ZL, Bian GB, Xie XL, Hou ZG, Zhou XH, Zhou YJ (2019) RASNet: segmentation for tracking surgical instruments in surgical videos using refined attention segmentation network. arXiv:190508663

  146. Nie D, Gao Y, Wang L, Shen D (2018) ASDNet: Attention based semi-supervised deep networks for medical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 370–378

  147. Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1520–1528

  148. Nøkland A, Eidnes LH (2019) Training neural networks with local error signals. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, PMLR, Long Beach, California, USA, proceedings of machine learning research, vol 97, pp 4839–4850. http://proceedings.mlr.press/v97/nokland19a.html

  149. Nosrati MS, Hamarneh G (2016) Incorporating prior knowledge in medical image segmentation: a survey. arXiv:160701092

  150. Nowozin S (2014) Optimal decisions from probabilistic models: the intersection-over-union case. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 548–555

  151. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, et al. (2018) Attention U-Net: learning where to look for the pancreas. arXiv:180403999

  152. Paschali M, Gasperini S, Roy AG, Fang MYS, Navab N (2019) 3DQ: compact quantized neural networks for volumetric whole brain segmentation. arXiv:190403110

  153. Peng C, Zhang X, Yu G, Luo G, Sun J (2017) Large kernel matters–improve semantic segmentation by global convolutional network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4353–4361

  154. Peng J, Kervadec H, Dolz J, Ayed IB, Pedersoli M, Desrosiers C (2019) Discretely-constrained deep network for weakly supervised segmentation. arXiv:190805770

  155. Perone CS, Cohen-Adad J (2018) Deep semi-supervised segmentation with weight-averaged consistency targets. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 12–19

  156. Perone CS, Cohen-Adad J (2019) Promises and limitations of deep learning for medical image segmentation. J Med Artif Intell 2. http://jmai.amegroups.com/article/view/4659

  157. Perone CS, Calabrese E, Cohen-Adad J (2018) Spinal cord gray matter segmentation using deep dilated convolutions. Sci Rep 8(1). https://doi.org/10.1038/s41598-018-24304-3

  158. Perone CS, Ballester P, Barros RC, Cohen-Adad J (2019) Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. Neuroimage 194:1–11

    Article  Google Scholar 

  159. Pohlen T, Hermans A, Mathias M, Leibe B (2017) Full-resolution residual networks for semantic segmentation in street scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4151–4160

  160. Proenca H, Neves JC (2019) Segmentation-less and non-holistic deep-learning frameworks for iris recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

  161. Qin Y, Kamnitsas K, Ancha S, Nanavati J, Cottrell G, Criminisi A, Nori A (2018) Autofocus layer for semantic segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 603–611

  162. Raghu M, Zhang C, Kleinberg J, Bengio S (2019) Transfusion: understanding transfer learning with applications to medical imaging. arXiv:190207208

  163. Reddick W, Glass J, Cook E, Elkin T, Deaton R (1997) Automated segmentation and classification of multispectral magnetic resonance images of brain using artificial neural networks. IEEE Trans Med Imaging 16(6):911–918. https://doi.org/10.1109/42.650887

    Article  Google Scholar 

  164. Reilhac A, Batan G, Michel C, Grova C, Tohka J, Collins D, Costes N, Evans A (2005) PET-SORTEO: validation and development of database of simulated PET volumes. IEEE Trans Nucl Sci 52(5):1321–1328. https://doi.org/10.1109/tns.2005.858242

    Article  Google Scholar 

  165. Remillard J (2018) Images to OSM. https://github.com/jremillard/images-to-osm

  166. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  167. Rezaei M, Harmuth K, Gierke W, Kellermeier T, Fischer M, Yang H, Meinel C (2017) A conditional adversarial network for semantic segmentation of brain tumor. In: International conference on medical image computing and computer assisted intervention, Brainlesion Workshop. Springer, pp 241–252

  168. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 234–241

  169. Roy AG, Conjeti S, Sheet D, Katouzian A, Navab N, Wachinger C (2017) Error corrective boosting for learning fully convolutional networks with limited data. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 231–239

  170. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536. https://doi.org/10.1038/323533a0

    Article  MATH  Google Scholar 

  171. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173. https://doi.org/10.1007/s11263-007-0090-8

    Article  Google Scholar 

  172. Russell SJ, Norvig P (2016) Artificial intelligence: a modern approach. Pearson Education Limited, Kuala Lumpur

    Google Scholar 

  173. Salehi SSM, Erdogmus D, Gholipour A (2017) Tversky loss function for image segmentation using 3D fully convolutional deep networks. In: International workshop on machine learning in medical imaging. Springer, pp 379–387

  174. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  175. Saxena S, Verbeek J (2016) Convolutional neural fabrics. In: Advances in neural information processing systems, pp 4053–4061

  176. Schlemper J, Oktay O, Schaap M, Heinrich M, Kainz B, Glocker B, Rueckert D (2019) Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal 53:197–207

    Article  Google Scholar 

  177. Shahriari S, Garcia D (2018) Meshfree simulations of ultrasound vector flow imaging using smoothed particle hydrodynamics. Phys Med Biol 63(20):205011. https://doi.org/10.1088/1361-6560/aae3c3

    Article  Google Scholar 

  178. Shaw A, Hunter D, Landola F, Sidhu S (2019) SqueezeNAS: fast neural architecture search for faster semantic segmentation. In: Proceedings of the IEEE international conference on computer vision workshops

  179. Shin HC, Tenenholtz NA, Rogers JK, Schwarz CG, Senjem ML, Gunter JL, Andriole KP, Michalski M (2018) Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 1–11

  180. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data. https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  181. Sifre L (2014) Rigid-motion scattering for image classification. PhD thesis, CMAP, Ecole Polytechnique

  182. Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the seventh international conference on document analysis and recognition—Volume 2. IEEE Computer Society, USA, ICDAR ’03, p 958

  183. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  184. Simpson AL, Antonelli M, Bakas S, Bilello M, Farahani K, van Ginneken B, Kopp-Schneider A, Landman BA, Litjens G, Menze B, et al. (2019) A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv:190209063

  185. Sinha A, Dolz J (2019) Multi-scale guided attention for medical image segmentation. arXiv:190602849

  186. Son J, Park SJ, Jung KH (2017) Retinal vessel segmentation in fundoscopic images with generative adversarial networks. arXiv:170609318

  187. Song G, Myeong H, Mu Lee K (2018) Seednet: automatic seed generation with deep reinforcement learning for robust interactive segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1760–1768

  188. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  189. Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:150500387

  190. Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127

    Article  Google Scholar 

  191. SUYEgit (2018) Mask R-CNN for surgery robot. https://github.com/SUYEgit/Surgery-Robot-Detection-Segmentation/

  192. Tabary J, Hugonnard P, Mathy F (2007) SINDBAD: a realistic multi-purpose and scalable X-ray simulation tool for NDT applications. In: DIR 2007: international symposium on digital industrial radiology and computed tomography

  193. Taghanaki SA, Duggan N, Ma H, Hou X, Celler A, Benard F, Hamarneh G (2018) Segmentation-free direct tumor volume and metabolic activity estimation from pet scans. Comput Med Imaging Graphics 63:52–66

    Article  Google Scholar 

  194. Taghanaki SA, Abhishek K, Azizi S, Hamarneh G (2019a) A kernelized manifold mapping to diminish the effect of adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 11340–11349

  195. Taghanaki SA, Abhishek K, Hamarneh G (2019b) Improved inference via deep input transfer. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 819–827

  196. Taghanaki SA, Bentaieb A, Sharma A, Zhou SK, Zheng Y, Georgescu B, Sharma P, Grbic S, Xu Z, Comaniciu D, et al. (2019c) Select, attend, and transfer: light, learnable skip connections. In: Medical image computing and computer-assisted intervention workshop on machine learning in medical imaging

  197. Taghanaki SA, Havaei M, Berthier T, Dutil F, Di Jorio L, Hamarneh G, Bengio Y (2019d) InfoMask: masked variational latent representation to localize chest disease. In: International conference on medical image computing and computer assisted intervention

  198. Taghanaki SA, Zheng Y, Zhou SK, Georgescu B, Sharma P, Xu D, Comaniciu D, Hamarneh G (2019e) Combo loss: handling input and output imbalance in multi-organ segmentation. Comput Med Imaging Graphics 75:24–33

    Article  Google Scholar 

  199. Tajbakhsh N, Jeyaseelan L, Li Q, Chiang JN, Wu Z, Ding X (2020) Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med Image Anal 63:101693. https://doi.org/10.1016/j.media.2020.101693

    Article  Google Scholar 

  200. Tarvainen A, Valpola H (2017) Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in neural information processing systems, pp 1195–1204

  201. Tsai HF, Gajda J, Sloan TF, Rares A, Shen AQ (2019) Usiigaci: instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning. SoftwareX 9:230–237

    Article  Google Scholar 

  202. Vorontsov E, Molchanov P, Byeon W, De Mello S, Jampani V, Liu MY, Kadoury S, Kautz J (2019) Towards semi-supervised segmentation via image-to-image translation. arXiv:190401636

  203. Vuola AO, Akram SU, Kannala J (2019) Mask R-CNN and U-net ensembled for nuclei segmentation. arXiv:190110170

  204. Wang EK, Zhang X, Pan L, Cheng C, Dimitrakopoulou-Strauss A, Li Y, Zhe N (2019a) Multi-path dilated residual network for nuclei segmentation and detection. Cells 8(5):499

    Article  Google Scholar 

  205. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017a) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  206. Wang G, Luo P, Lin L, Wang X (2017b) Learning object interactions and descriptions for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5859–5867

  207. Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, Cottrell G (2018a) Understanding convolution for semantic segmentation. In: 2018 IEEE winter conference on applications of computer vision. IEEE, pp 1451–1460

  208. Wang S, Rong R, Yang DM, Cai L, Yang L, Luo D, Yao B, Xu L, Wang T, Zhan X, et al. (2019b) Computational staining of pathology images to study tumor microenvironment in lung cancer. Available at SSRN 3391381

  209. Wang W, Lai Q, Fu H, Shen J, Ling H (2019c) Salient object detection in the deep learning era: an in-depth survey. arXiv:190409146

  210. Wang X, Girshick R, Gupta A, He K (2018b) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  211. Wang X, Wang H, Niu S, Zhang J (2019d) Detection and localization of image forgeries using improved mask regional convolutional neural network. Math Biosci Eng 16:4581–4593

    Article  Google Scholar 

  212. Wang Z, Sarcar S, Liu J, Zheng Y, Ren X (2018c) Outline objects using deep reinforcement learning. arXiv:180404603

  213. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Advances in neural information processing systems, pp 2074–2082

  214. Weng Y, Zhou T, Li Y, Qiu X (2019a) NAS-Unet: neural architecture search for medical image segmentation. IEEE Access 7:44247–44257

    Article  Google Scholar 

  215. Weng Y, Zhou T, Li Y, Qiu X (2019b) NAS-unet: neural architecture search for medical image segmentation. IEEE Access 7:44247–44257. https://doi.org/10.1109/access.2019.2908991

    Article  Google Scholar 

  216. Wessel J, Heinrich MP, von Berg J, Franz A, Saalbach A (2019) Sequential rib labeling and segmentation in chest X-ray using Mask R-CNN. In: International conference on medical imaging with deep learning—extended abstract track, London, United Kingdom. https://openreview.net/forum?id=SJxuHzLjFV

  217. Wojna Z, Ferrari V, Guadarrama S, Silberman N, Chen LC, Fathi A, Uijlings J (2017) The devil is in the decoder. arXiv:170705847

  218. Wong KC, Moradi M, Tang H, Syeda-Mahmood T (2018) 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes. In: International conference on medical image computing and computer assisted intervention. Springer, pp 612–619

  219. Wu Y, He K (2018) Group normalization. In: Proceedings of the European conference on computer vision, pp 3–19

  220. Wu Z, Shen C, Van Den Hengel A (2019) Wider or deeper: revisiting the resnet model for visual recognition. Pattern Recognit 90:119–133

    Article  Google Scholar 

  221. Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A (2010) SUN database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 3485–3492

  222. Xie S, Tu Z (2015) Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision, pp 1395–1403

  223. Xie X, Niu J, Liu X, Chen Z, Tang S (2020) A survey on domain knowledge powered deep learning for medical image analysis. arXiv:200412150

  224. Xu X, Lu Q, Yang L, Hu S, Chen D, Hu Y, Shi Y (2018) Quantization of fully convolutional networks for accurate biomedical image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8300–8308

  225. Xue Y, Xu T, Zhang H, Long LR, Huang X (2018) SegAN: adversarial network with multi-scale L1 loss for medical image segmentation. Neuroinformatics 16(3–4):383–392

    Article  Google Scholar 

  226. Yang D, Xu D, Zhou SK, Georgescu B, Chen M, Grbic S, Metaxas D, Comaniciu D (2017a) Automatic liver segmentation using an adversarial image-to-image network. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 507–515

  227. Yang Q, Li N, Zhao Z, Fan X, Chang EI, Xu Y, et al. (2018) MRI cross-modality neuroimage-to-neuroimage translation. arXiv:180106940

  228. Yang X, Yu L, Wu L, Wang Y, Ni D, Qin J, Heng PA (2017b) Fine-grained recurrent neural networks for automatic prostate segmentation in ultrasound images. In: Thirty-first AAAI conference on artificial intelligence

  229. Yu B, Zhou L, Wang L, Fripp J, Bourgeat P (2018a) 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 626–630

  230. Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018b) BiSeNet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV). Springer, pp 334–349. https://doi.org/10.1007/978-3-030-01261-8_20

  231. Yuan Y (2017) Automatic skin lesion segmentation with fully convolutional-deconvolutional networks. arXiv:170305165

  232. Zamir AR, Sax A, Shen W, Guibas LJ, Malik J, Savarese S (2018) Taskonomy: disentangling task transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3712–3722

  233. Zhang P, Zhong Y, Deng Y, Tang X, Li X (2019) A survey on deep learning of small sample in biomedical image analysis. arXiv:190800473

  234. Zhang W, Witharana C, Liljedahl A, Kanevskiy M (2018a) Deep convolutional neural networks for automated characterization of arctic ice-wedge polygons in very high spatial resolution aerial imagery. Remote Sens 10(9):1487

    Article  Google Scholar 

  235. Zhang Y, Yang L, Chen J, Fredericksen M, Hughes DP, Chen DZ (2017) Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 408–416

  236. Zhang Y, Miao S, Mansi T, Liao R (2018b) Task driven generative modeling for unsupervised domain adaptation: application to X-ray image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 599–607

  237. Zhang Z, Yang L, Zheng Y (2018c) Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9242–9251

  238. Zhang Z, Zhang X, Peng C, Xue X, Sun J (2018d) Exfuse: enhancing feature fusion for semantic segmentation. In: Proceedings of the European conference on computer vision, pp 269–284

  239. Zhao H, Li H, Cheng L (2017a) Synthesizing filamentary structured images with GANs. arXiv:170602185

  240. Zhao H, Shi J, Qi X, Wang X, Jia J (2017b) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  241. Zhao M, Hamarneh G (2019a) Retinal image classification viavasculature-guided sequential attention. In: International conference on computer vision workshop on visual recognition for medical images

  242. Zhao M, Hamarneh G (2019b) Tree-LSTM: using LSTM to encode memory in anatomical tree prediction from 3D images. In: Medical image computing and computer-assisted intervention workshop on machine learning in medical imaging

  243. Zhao T, Yang Y, Niu H, Wang D, Chen Y (2018) Comparing U-Net convolutional network with Mask R-CNN in the performances of pomegranate tree canopy segmentation. In: Asia-pacific remote sensing

  244. Zhao ZQ, Zheng P, Xu ST, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232. https://doi.org/10.1109/tnnls.2018.2876865

    Article  Google Scholar 

  245. Zhen X, Li S (2015) Towards direct medical image analysis without segmentation. arXiv:151006375

  246. Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. In: Advances in neural information processing systems, pp 487–495

  247. Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, Torralba A (2017) Scene parsing through ADE20K dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 633–641

  248. Zhou S, Nie D, Adeli E, Yin J, Lian J, Shen D (2019a) High-resolution encoder-decoder networks for low-contrast medical image segmentation. IEEE Trans Image Process 29:461–475

    MathSciNet  Article  Google Scholar 

  249. Zhou T, Ruan S, Canu S (2019b) A review: deep learning for medical image segmentation using multi-modality fusion. Array 3–4:100004. https://doi.org/10.1016/j.array.2019.100004

    Article  Google Scholar 

  250. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2018) UNet++: a nested U-Net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 3–11

  251. Zhu X, Zhou H, Yang C, Shi J, Lin D (2018) Penalizing top performers: conservative loss for semantic segmentation adaptation. In: Proceedings of the European conference on computer vision, pp 568–583

  252. Zhu Z, Liu C, Yang D, Yuille A, Xu D (2019) V-NAS: neural architecture search for volumetric medical image segmentation. In: 2019 international conference on 3D vision (3DV). IEEE. https://doi.org/10.1109/3dv.2019.00035

  253. Zoph B, Le QV (2016) Neural architecture search with reinforcement learning. arXiv:161101578

  254. Zou Z, Shi Z, Guo Y, Ye J (2019) Object detection in 20 years: a survey. arXiv:190505055

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Saeid Asgari Taghanaki.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Saeid Asgari Taghanaki and Kumar Abhishek Joint first authors.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 112 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Asgari Taghanaki, S., Abhishek, K., Cohen, J.P. et al. Deep semantic segmentation of natural and medical images: a review. Artif Intell Rev 54, 137–178 (2021). https://doi.org/10.1007/s10462-020-09854-1

Download citation

Keywords

  • Semantic image segmentation
  • Deep learning