Skip to main content
SpringerLink
Log in

Deep semantic segmentation of natural and medical images: a review

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

The semantic image segmentation task consists of classifying each pixel of an image into an instance, where each instance corresponds to a class. This task is a part of the concept of scene understanding or better explaining the global context of an image. In the medical image analysis domain, image segmentation can be used for image-guided interventions, radiotherapy, or improved radiological diagnostics. In this review, we categorize the leading deep learning-based medical and non-medical image segmentation solutions into six main groups of deep architectural, data synthesis-based, loss function-based, sequenced models, weakly supervised, and multi-task methods and provide a comprehensive review of the contributions in each of these groups. Further, for each group, we analyze each variant of these groups and discuss the limitations of the current approaches and present potential future research directions for semantic image segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. https://github.com/matterport/Mask_RCNN

  • Abhishek K, Hamarneh G (2019) Mask2Lesion: mask-constrained adversarial skin lesion image synthesis. In: Medical image computing and computer-assisted intervention workshop on simulation and synthesis in medical imaging, pp 71–80

  • Abhishek K, Hamarneh G, Drew MS (2020) Illumination-based transformations improve skin lesion segmentation in dermoscopic images. arXiv:200310111

  • Adams RA, Fournier JJ (2003) Sobolev spaces. Elsevier, Amsterdam

    MATH  Google Scholar 

  • Afshari S, BenTaieb A, Mirikharaji Z, Hamarneh G (2019) Weakly supervised fully convolutional network for PET lesion segmentation. In: Medical imaging 2019: image processing, international society for optics and photonics, vol 10949, p 109491K

  • Alom MZ, Yakopcic C, Hasan M, Taha TM, Asari VK (2019) Recurrent residual U-Net for medical image segmentation. J Med Imag 6(1):14006

    Article  Google Scholar 

  • Amirul Islam M, Rochan M, Bruce ND, Wang Y (2017) Gated feedback refinement network for dense image labeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3751–3759

  • Amit Y (2019) Deep learning with asymmetric connections and hebbian updates. Front Comput Neurosci. https://doi.org/10.3389/fncom.2019.00018

    Article  Google Scholar 

  • Anantharaman R, Velazquez M, Lee Y (2018) Utilizing Mask R-CNN for detection and segmentation of oral diseases. In: 2018 IEEE international conference on bioinformatics and biomedicine, pp 2197–2204

  • Badrinarayanan V, Handa A, Cipolla R (2015) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:151100561

  • Bai W, Suzuki H, Qin C, Tarroni G, Oktay O, Matthews PM, Rueckert D (2018) Recurrent neural networks for aortic image sequence segmentation with sparse annotations. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 586–594

  • Bellec G, Scherr F, Hajek E, Salaj D, Legenstein R, Maass W (2019) Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets. arXiv:190109049

  • Bengio Y, Frasconi P (1994) Credit assignment through time: alternatives to backpropagation. In: Advances in neural information processing systems, pp 75–82

  • Benoit-Cattin H, Collewet G, Belaroussi B, Saint-Jalmes H, Odet C (2005) The SIMRI project: a versatile and interactive MRI simulator. J Magn Reson 173(1):97–115. https://doi.org/10.1016/j.jmr.2004.09.027

    Article  Google Scholar 

  • BenTaieb A, Hamarneh G (2016) Topology aware fully convolutional networks for histology gland segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 460–468

  • Berman M, Blaschko MB, Triki AR, Yu J (2018a) Yes, IoU loss is submodular-as a function of the mispredictions. arXiv:180901845

  • Berman M, Rannen Triki A, Blaschko MB (2018b) The Lovász-Softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4413–4421

  • Bischke B, Helber P, Folz J, Borth D, Dengel A (2019) Multi-task learning for segmentation of building footprints with deep neural networks. In: 2019 IEEE international conference on image processing. IEEE, pp 1480–1484

  • Bonta LR, Kiran NU (2019) Efficient segmentation of medical images using dilated residual networks. In: Computer aided intervention and diagnostics in clinical and medical images. Springer, pp 39–47

  • Borji A, Cheng MM, Hou Q, Jiang H, Li J (2019) Salient object detection: a survey. Comput Vis Media 5(2):117–150. https://doi.org/10.1007/s41095-019-0149-9

    Article  Google Scholar 

  • Brostow GJ, Shotton J, Fauqueur J, Cipolla R (2008) Segmentation and recognition using structure from motion point clouds. In: Lecture notes in computer science. Springer, Berlin, pp 44–57. https://doi.org/10.1007/978-3-540-88682-2_5

  • Brostow GJ, Fauqueur J, Cipolla R (2009) Semantic object classes in video: a high-definition ground truth database. Pattern Recognit Lett 30(2):88–97. https://doi.org/10.1016/j.patrec.2008.04.005

    Article  Google Scholar 

  • Brügger R, Baumgartner CF, Konukoglu E (2019) A partially reversible U-Net for memory-efficient volumetric image segmentation. arXiv:190606148

  • Caliva F, Iriondo C, Martinez AM, Majumdar S, Pedoia V (2019) Distance map loss penalty term for semantic segmentation. In: International conference on medical imaging with deep learning

  • Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

    Article  MathSciNet  Google Scholar 

  • Chaichulee S, Villarroel M, Jorge J, Arteta C, Green G, McCormick K, Zisserman A, Tarassenko L (2017) Multi-task convolutional neural network for patient detection and skin segmentation in continuous non-contact vital sign monitoring. In: 2017 12th IEEE international conference on automatic face & gesture recognition. IEEE, pp 266–272

  • Chakravarty A, Sivaswamy J (2018) RACE-Net: a recurrent neural network for biomedical image segmentation. IEEE J Biomed Health Inform 23(3):1151–1162

    Article  Google Scholar 

  • Challenge G (2020) Grand challenges in biomedical image analysis. https://grand-challenge.org/challenges/

  • Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 3–13

  • Chen LC, Yang Y, Wang J, Xu W, Yuille AL (2016) Attention to scale: scale-aware semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3640–3649

  • Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017a) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  • Chen LC, Papandreou G, Schroff F, Adam H (2017b) Rethinking atrous convolution for semantic image segmentation. arXiv:170605587

  • Chen LC, Collins M, Zhu Y, Papandreou G, Zoph B, Schroff F, Adam H, Shlens J (2018a) Searching for efficient multi-scale architectures for dense image prediction. In: Advances in neural information processing systems, pp 8699–8710

  • Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018b) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision, pp 801–818

  • Chen X, Williams BM, Vallabhaneni SR, Czanner G, Williams R, Zheng Y (2019) Learning active contour models for medical image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 11632–11640

  • Cherian A, Sullivan A (2019) Sem-GAN: semantically-consistent image-to-image translation. In: 2019 IEEE winter conference on applications of computer vision (WACV). IEEE. https://doi.org/10.1109/wacv.2019.00196

  • Choi J, Kim T, Kim C (2019) Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 6830–6840

  • Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  • Cireşan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3642–3649

  • Cireşan DC, Meier U, Masci J, Gambardella LM, Schmidhuber J (2011) High-performance neural networks for visual object classification. arXiv:11020183

  • Cohen JP, Luck M, Honari S (2018) Distribution matching losses can hallucinate features in medical image translation. In: Medical image computing and computer assisted intervention – MICCAI 2018. Springer, pp 529–536. https://doi.org/10.1007/978-3-030-00928-1_60

  • Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  • Costa P, Galdran A, Meyer MI, Abràmoff MD, Niemeijer M, Mendonça AM, Campilho A (2017) Towards adversarial retinal image synthesis. arXiv:170108974

  • Couprie C, Farabet C, Najman L, LeCun Y (2013) Indoor semantic segmentation using depth information. arXiv:13013572

  • Czarnecki WM, Osindero S, Jaderberg M, Swirszcz G, Pascanu R (2017) Sobolev training for neural networks. In: Advances in neural information processing systems, pp 4278–4287

  • Dai W, Dong N, Wang Z, Liang X, Zhang H, Xing EP (2018) SCAN: structure correcting adversarial network for organ segmentation in chest X-rays. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 263–273. https://doi.org/10.1007/978-3-030-00889-5_30

  • Drobnjak I, Gavaghan D, Süli E, Pitt-Francis J, Jenkinson M (2006) Development of a functional magnetic resonance imaging simulator for modeling realistic rigid-body motion artifacts. Magn Reson Med 56(2):364–380. https://doi.org/10.1002/mrm.20939

    Article  Google Scholar 

  • Drobnjak I, Pell GS, Jenkinson M (2010) Simulating the effects of time-varying magnetic fields with a realistic simulated scanner. Magn Reson Imaging 28(7):1014–1021. https://doi.org/10.1016/j.mri.2010.03.029

    Article  Google Scholar 

  • Drozdzal M, Chartrand G, Vorontsov E, Shakeri M, Di Jorio L, Tang A, Romero A, Bengio Y, Pal C, Kadoury S (2018) Learning normalized inputs for iterative estimation in medical image segmentation. Med Image Anal 44:1–13

    Article  Google Scholar 

  • Everingham M, Gool LV, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  • Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2012) The PASCAL visual object classes challenge 2012 (VOC2012) results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html

  • Everingham M, Eslami SA, Van Gool L, Williams CK, Winn J, Zisserman A (2015) The PASCAL visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136

    Article  Google Scholar 

  • Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019a) Dual attention network for scene segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3146–3154

  • Fu J, Liu J, Wang Y, Zhou J, Wang C, Lu H (2019b) Stacked deconvolutional network for semantic segmentation. IEEE Trans Image Process

  • Galdran A, Alvarez-Gila A, Meyer MI, Saratxaga CL, Araújo T, Garrote E, Aresta G, Costa P, Mendonça AM, Campilho A (2017) Data-driven color augmentation techniques for deep skin image analysis. arXiv:170303702

  • Gamage H, Wijesinghe W, Perera I (2019) Instance-based segmentation for boundary detection of neuropathic ulcers through Mask-RCNN. In: International conference on artificial neural networks. Springer, pp 511–522

  • Gao Y, Phillips JM, Zheng Y, Min R, Fletcher PT, Gerig G (2018) Fully convolutional structured LSTM networks for joint 4D medical image segmentation. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 1104–1108

  • Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Martinez-Gonzalez P, Garcia-Rodriguez J (2018) A survey on deep learning techniques for image and video semantic segmentation. Appl Soft Comput 70:41–65

    Article  Google Scholar 

  • Glatard T, Lartizien C, Gibaud B, da Silva RF, Forestier G, Cervenansky F, Alessandrini M, Benoit-Cattin H, Bernard O, Camarasu-Pop S, Cerezo N, Clarysse P, Gaignard A, Hugonnard P, Liebgott H, Marache S, Marion A, Montagnat J, Tabary J, Friboulet D (2013) A virtual imaging platform for multi-modality medical image simulation. IEEE Trans Med Imaging 32(1):110–118. https://doi.org/10.1109/tmi.2012.2220154

    Article  Google Scholar 

  • Goceri E (2019a) Challenges and recent solutions for image segmentation in the era of deep learning. In: 2019 ninth international conference on image processing theory, tools and applications (IPTA). IEEE. https://doi.org/10.1109/ipta.2019.8936087

  • Goceri E (2019b) Diagnosis of alzheimerś disease with sobolev gradient-based optimization and 3d convolutional neural network. Int J Numer Methods Biomed Eng. https://doi.org/10.1002/cnm.3225

    Article  MathSciNet  Google Scholar 

  • Goceri E (2020) CapsNet topology to classify tumours from brain images and comparative evaluation. IET Image Process 14(5):882–889. https://doi.org/10.1049/iet-ipr.2019.0312

    Article  Google Scholar 

  • Goceri E, Goceri N (2017) Deep learning in medical image analysis: recent advances and future trends. In: Proceedings of the IADIS international conference computer graphics, visualization, computer vision and image processing (CGVCVIP) 2017, pp 305–310

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  • Gros C, De Leener B, Badji A, Maranzano J, Eden D, Dupont SM, Talbott J, Zhuoquiong R, Liu Y, Granberg T et al (2019) Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks. Neuroimage 184:901–915

    Article  Google Scholar 

  • Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging 38:2281–2292

    Article  Google Scholar 

  • Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. In: Advances in neural information processing systems, pp 5767–5777

  • Guo Y, Liu Y, Georgiou T, Lew MS (2018) A review of semantic segmentation using deep neural networks. Int J Multimed Inf Retr 7(2):87–93

    Article  Google Scholar 

  • Hamarneh G, Jassi P (2010) VascuSynth: simulating vascular trees for generating volumetric image data with ground-truth segmentation and tree analysis. Comput Med Imaging Graphics 34(8):605–616. https://doi.org/10.1016/j.compmedimag.2010.06.002

    Article  Google Scholar 

  • Han C, Murao K, Satoh S, Nakayama H (2019) Learning more with less: GAN-based medical image augmentation. Med Imaging Technol 37(3):137–142

    Google Scholar 

  • Han S, Liu X, Mao H, Pu J, Pedram A, Horowitz MA, Dally WJ (2016) EIE: efficient inference engine on compressed deep neural network. In: 2016 ACM/IEEE 43rd annual international symposium on computer architecture. IEEE, pp 243–254

  • Haralick RM, Shapiro LG (1992) Computer and robot vision. Addison-Wesley, Boston

    Google Scholar 

  • Harrison R, Lewellen T (2012) The SimSET program. In: Monte Carlo calculations in nuclear medicine, Second Edition. Taylor & Francis, pp 87–110. https://doi.org/10.1201/b13073-7

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  • He T, Guo J, Wang J, Xu X, Yi Z (2019) Multi-task learning for the segmentation of thoracic organs at risk in CT images. In: SegTHOR@ISBI

  • Hesamian MH, Jia W, He X, Kennedy P (2019) Deep learning techniques for medical image segmentation: achievements and challenges. J Digit Imaging 32:582–596

    Article  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Honari S, Yosinski J, Vincent P, Pal C (2016) Recombinator networks: learning coarse-to-fine feature aggregation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5743–5752

  • Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:170404861

  • Hu J, Shen L, Sun G (2018a) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  • Hu Y, Chen Z, Lin W (2018b) RGB-D semantic segmentation: a review. In: 2018 IEEE international conference on multimedia & expo workshops. IEEE, pp 1–6

  • Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  • Huang SW, Lin CT, Chen SP, Wu YY, Hsu PH, Lai SH (2018) AugGAN: Cross domain adaptation with GAN-based data augmentation. In: Proceedings of the European conference on computer vision (ECCV). Springer, Berlin, pp 731–744. https://doi.org/10.1007/978-3-030-01240-3_44

  • Huo Y, Xu Z, Bao S, Assad A, Abramson RG, Landman BA (2018) Adversarial synthesis learning enables segmentation without target modality ground truth. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 1217–1220

  • Hussain MA, Amir-Khalili A, Hamarneh G, Abugharbieh R (2017) Segmentation-free kidney localization and volume estimation using aggregated orthogonal decision CNNs. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 612–620

  • Isensee F, Petersen J, Klein A, Zimmerer D, Jaeger PF, Kohl S, Wasserthal J, Koehler G, Norajitra T, Wirkert S, et al. (2019) nnU-Net: self-adapting framework for U-Net-based medical image segmentation. In: Bildverarbeitung für die Medizin 2019. Springer, pp 22–22

  • Jaeger PF, Kohl SA, Bickelhaupt S, Isensee F, Kuder TA, Schlemmer HP, Maier-Hein KH (2018) Retina U-Net: embarrassingly simple exploitation of segmentation supervision for medical object detection. arXiv:181108661

  • Jégou S, Drozdzal M, Vazquez D, Romero A, Bengio Y (2017) The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 11–19

  • Jensen J, Svendsen N (1992) Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers. IEEE Trans Ultrason Ferroelectr Freq Control 39(2):262–267. https://doi.org/10.1109/58.139123

    Article  Google Scholar 

  • Jensen JA (1996) Field: A program for simulating ultrasound systems. In: 10th Nordic-Baltic conference on biomedical imaging, Volume 34, Supplement 1, Part 1, pp 351–353

  • Jin D, Xu Z, Tang Y, Harrison AP, Mollura DJ (2018) CT-realistic lung nodule simulation from 3D conditional generative adversarial networks for robust lung segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 732–740

  • Jin W, Fatehi M, Abhishek K, Mallya M, Toyota B, Hamarneh G (2020) Artificial intelligence in glioma imaging: challenges and advances. J Neural Eng 17(2):021002. https://doi.org/10.1088/1741-2552/ab8131

    Article  Google Scholar 

  • Johnson JW (2018) Adapting mask R-CNN for automatic nucleus segmentation. arXiv:180500500

  • Karimi D, Salcudean SE (2019) Reducing the Hausdorff distance in medical image segmentation with convolutional neural networks. arXiv:190410030

  • Karimi D, Dou H, Warfield SK, Gholipour A (2019) Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. arXiv:191202911

  • Ke R, Bugeau A, Papadakis N, Schütz P, Schönlieb CB (2019) A multi-task U-Net for segmentation with lazy labels. arXiv:1906.12177

  • Kervadec H, Bouchtiba J, Desrosiers C, Granger E, Dolz J, Ben Ayed I (2019a) Boundary loss for highly unbalanced segmentation. In: Proceedings of the 2nd international conference on medical imaging with deep learning, PMLR, London, United Kingdom, proceedings of machine learning research, vol 102, pp 285–296. http://proceedings.mlr.press/v102/kervadec19a.html

  • Kervadec H, Dolz J, Tang M, Granger E, Boykov Y, Ayed IB (2019) Constrained-CNN losses for weakly supervised segmentation. Med Image Anal 54:88–99. https://doi.org/10.1016/j.media.2019.02.009

    Article  Google Scholar 

  • Khosravan N, Mortazi A, Wallace M, Bagci U (2019) PAN: projective adversarial network for medical image segmentation. arXiv:190604378

  • Kim B, Ye JC (2019) Multiphase level-set loss for semi-supervised and unsupervised segmentation with deep learning. arXiv:190402872

  • Kim HE, Hwang S (2016) Deconvolutional feature stacking for weakly-supervised semantic segmentation. arXiv:160204984

  • Kim YD, Park E, Yoo S, Choi T, Yang L, Shin D (2015) Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv:151106530

  • Kopelowitz E, Engelhard G (2019) Lung nodules detection and segmentation using 3D Mask R-CNN. arXiv:1907.07676

  • Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • Kuntimad G, Ranganath H (1999) Perfect image segmentation using pulse coupled neural networks. IEEE Trans Neural Netw 10(3):591–598. https://doi.org/10.1109/72.761716

    Article  Google Scholar 

  • Lateef F, Ruichek Y (2019) Survey on semantic segmentation using deep learning techniques. Neurocomputing 338:321–348

    Article  Google Scholar 

  • Le TLT, Thome N, Bernard S, Bismuth V, Patoureaux F (2019) Multitask classification and segmentation for cancer diagnosis in mammography. arXiv:190905397

  • LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  • Lee DH, Zhang S, Fischer A, Bengio Y (2015) Difference target propagation. In: Machine learning and knowledge discovery in databases. Springer, pp 498–515. https://doi.org/10.1007/978-3-319-23528-8_31

  • Lee J, Kim E, Lee S, Lee J, Yoon S (2019) Ficklenet: weakly and semi-supervised semantic image segmentation using stochastic inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5267–5276

  • Leroux S, Molchanov P, Simoens P, Dhoedt B, Breuel T, Kautz J (2018) IamNN: iterative and adaptive mobile neural network for efficient image classification. arXiv:180410123

  • Li H, Xiong P, An J, Wang L (2018) Pyramid attention network for semantic segmentation. arXiv:180510180

  • Li H, Li J, Lin X, Qian X (2019a) Pancreas segmentation via spatial context based U-Net and bidirectional LSTM. arXiv:190300832

  • Li S, Dong M, Du G, Mu X (2019b) Attention dense-U-Net for automatic breast mass segmentation in digital mammogram. IEEE Access 7:59037–59047

    Article  Google Scholar 

  • Li X, Liu Z, Luo P, Change Loy C, Tang X (2017) Not all pixels are equal: difficulty-aware semantic segmentation via deep layer cascade. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3193–3202

  • Li X, Yu L, Chen H, Fu CW, Heng PA (2019c) Transformation consistent self-ensembling model for semi-supervised medical image segmentation. arXiv:190300348

  • Lian S, Luo Z, Zhong Z, Lin X, Su S, Li S (2018) Attention guided U-Net for accurate iris segmentation. J Vis Commun Image Represent 56:296–304

    Article  Google Scholar 

  • Lin D, Ji Y, Lischinski D, Cohen-Or D, Huang H (2018) Multi-scale context intertwining for semantic segmentation. In: Proceedings of the European conference on computer vision, pp 603–619

  • Lin G, Milan A, Shen C, Reid I (2017a) RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1925–1934

  • Lin TY, Goyal P, Girshick R, He K, Dollár P (2017b) Focal loss for dense object detection. arXiv:170802002

  • Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  • Liu C, Chen LC, Schroff F, Adam H, Hua W, Yuille AL, Fei-Fei L (2019a) Auto-deeplab: hierarchical neural architecture search for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 82–92

  • Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, Pietikäinen M (2019b) Deep learning for generic object detection: a survey. Int J Comput Vis 128(2):261–318. https://doi.org/10.1007/s11263-019-01247-4

    Article  Google Scholar 

  • Liu Y, Perona P, Meister M (2019c) Panda: panoptic data augmentation. arXiv:191112317

  • Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  • Luc P, Couprie C, Chintala S, Verbeek J (2016) Semantic segmentation using adversarial networks. arXiv:161108408

  • Luo P, Wang G, Lin L, Wang X (2017) Deep dual learning for semantic image segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 2718–2726

  • Ma WDK, Lewis J, Kleijn WB (2019) The hsic bottleneck: Deep learning without back-propagation. arXiv:190801580

  • Marion A, Forestier G, Benoit-Cattin H, Camarasu-Pop S, Clarysse P, da SilvaRF, Gibaud B, Glatard T, Hugonnard P, Lartizien C, Liebgott H, Specovius S,Tabary J, Valette S, Friboulet D (2011) Multi-modality medical image simulation of biological models with the virtual imaging platform (VIP). In: 2011 24th international symposium on computer-based medical systems(CBMS). IEEE. https://doi.org/10.1109/cbms.2011.5999141

  • Mehta S, Mercan E, Bartlett J, Weaver D, Elmore JG, Shapiro L (2018) Y-Net: Joint segmentation and classification for diagnosis of breast biopsy images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 893–901

  • Meyer BJ, Harwood B, Drummond T (2018) Deep metric learning and image classification with nearest neighbour Gaussian kernels. In: 2018 25th IEEE international conference on image processing. IEEE, pp 151–155

  • Milletari F, Navab N, Ahmadi SA (2016) V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision. IEEE, pp 565–571

  • Mirikharaji Z, Hamarneh G (2018) Star shape prior in fully convolutional networks for skin lesion segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 737–745

  • Mirikharaji Z, Yan Y, Hamarneh G (2019) Learning to segment skin lesions from noisy annotations. In: International workshop on medical image learning with less labels and imperfect data

  • Moeskops P, Veta M, Lafarge MW, Eppenhof KA, Pluim JP (2017) Adversarial training and dilated convolutions for brain MRI segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 56–64

  • Mohajerani S, Asad R, Abhishek K, Sharma N, van Duynhoven A, Saeedi P (2019) Cloudmaskgan: a content-aware unpaired image-to-image translation algorithm for remote sensing imagery. In: 2019 IEEE international conference on image processing. IEEE, pp 1965–1969

  • Mohanty SP (2018) Crowdai mapping challenge 2018: baseline with mask RCNN. https://github.com/crowdai/crowdai-mapping-challenge-mask-rcnn

  • Mottaghi R, Chen X, Liu X, Cho NG, Lee SW, Fidler S, Urtasun R, Yuille A (2014) The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 891–898

  • Mukherjee S, Cheng I, Miller S, Guo T, Chau V, Basu A (2019) A fast segmentation-free fully automated approach to white matter injury detection in preterm infants. Med Biol Eng Comput 57(1):71–87

    Article  Google Scholar 

  • Mumford D, Shah J (1989) Optimal approximations by piecewise smooth functions and associated variational problems. Commun Pure Appl Math 42(5):577–685

    Article  MathSciNet  Google Scholar 

  • Neff T, Payer C, Stern D, Urschler M (2017) Generative adversarial network based synthesis for supervised medical image segmentation. In: Proceedings of OAGM and ARW joint workshop

  • Neff T, Payer C, Štern D, Urschler M (2018) Generative adversarial networks to synthetically augment data for deep learning based image segmentation. In: Proceedings of the OAGM workshop 2018: medical image analysis. Verlag der Technischen Universität Graz, pp 22–29

  • Nguyen HH, Fang F, Yamagishi J, Echizen I (2019) Multi-task learning for detecting and segmenting manipulated facial images and videos. arXiv:1906.06876

  • Ni ZL, Bian GB, Xie XL, Hou ZG, Zhou XH, Zhou YJ (2019) RASNet: segmentation for tracking surgical instruments in surgical videos using refined attention segmentation network. arXiv:190508663

  • Nie D, Gao Y, Wang L, Shen D (2018) ASDNet: Attention based semi-supervised deep networks for medical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 370–378

  • Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1520–1528

  • Nøkland A, Eidnes LH (2019) Training neural networks with local error signals. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, PMLR, Long Beach, California, USA, proceedings of machine learning research, vol 97, pp 4839–4850. http://proceedings.mlr.press/v97/nokland19a.html

  • Nosrati MS, Hamarneh G (2016) Incorporating prior knowledge in medical image segmentation: a survey. arXiv:160701092

  • Nowozin S (2014) Optimal decisions from probabilistic models: the intersection-over-union case. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 548–555

  • Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, et al. (2018) Attention U-Net: learning where to look for the pancreas. arXiv:180403999

  • Paschali M, Gasperini S, Roy AG, Fang MYS, Navab N (2019) 3DQ: compact quantized neural networks for volumetric whole brain segmentation. arXiv:190403110

  • Peng C, Zhang X, Yu G, Luo G, Sun J (2017) Large kernel matters–improve semantic segmentation by global convolutional network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4353–4361

  • Peng J, Kervadec H, Dolz J, Ayed IB, Pedersoli M, Desrosiers C (2019) Discretely-constrained deep network for weakly supervised segmentation. arXiv:190805770

  • Perone CS, Cohen-Adad J (2018) Deep semi-supervised segmentation with weight-averaged consistency targets. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 12–19

  • Perone CS, Cohen-Adad J (2019) Promises and limitations of deep learning for medical image segmentation. J Med Artif Intell 2. http://jmai.amegroups.com/article/view/4659

  • Perone CS, Calabrese E, Cohen-Adad J (2018) Spinal cord gray matter segmentation using deep dilated convolutions. Sci Rep 8(1). https://doi.org/10.1038/s41598-018-24304-3

  • Perone CS, Ballester P, Barros RC, Cohen-Adad J (2019) Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. Neuroimage 194:1–11

    Article  Google Scholar 

  • Pohlen T, Hermans A, Mathias M, Leibe B (2017) Full-resolution residual networks for semantic segmentation in street scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4151–4160

  • Proenca H, Neves JC (2019) Segmentation-less and non-holistic deep-learning frameworks for iris recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

  • Qin Y, Kamnitsas K, Ancha S, Nanavati J, Cottrell G, Criminisi A, Nori A (2018) Autofocus layer for semantic segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 603–611

  • Raghu M, Zhang C, Kleinberg J, Bengio S (2019) Transfusion: understanding transfer learning with applications to medical imaging. arXiv:190207208

  • Reddick W, Glass J, Cook E, Elkin T, Deaton R (1997) Automated segmentation and classification of multispectral magnetic resonance images of brain using artificial neural networks. IEEE Trans Med Imaging 16(6):911–918. https://doi.org/10.1109/42.650887

    Article  Google Scholar 

  • Reilhac A, Batan G, Michel C, Grova C, Tohka J, Collins D, Costes N, Evans A (2005) PET-SORTEO: validation and development of database of simulated PET volumes. IEEE Trans Nucl Sci 52(5):1321–1328. https://doi.org/10.1109/tns.2005.858242

    Article  Google Scholar 

  • Remillard J (2018) Images to OSM. https://github.com/jremillard/images-to-osm

  • Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  • Rezaei M, Harmuth K, Gierke W, Kellermeier T, Fischer M, Yang H, Meinel C (2017) A conditional adversarial network for semantic segmentation of brain tumor. In: International conference on medical image computing and computer assisted intervention, Brainlesion Workshop. Springer, pp 241–252

  • Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer assisted intervention. Springer, pp 234–241

  • Roy AG, Conjeti S, Sheet D, Katouzian A, Navab N, Wachinger C (2017) Error corrective boosting for learning fully convolutional networks with limited data. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 231–239

  • Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536. https://doi.org/10.1038/323533a0

    Article  MATH  Google Scholar 

  • Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173. https://doi.org/10.1007/s11263-007-0090-8

    Article  Google Scholar 

  • Russell SJ, Norvig P (2016) Artificial intelligence: a modern approach. Pearson Education Limited, Kuala Lumpur

    MATH  Google Scholar 

  • Salehi SSM, Erdogmus D, Gholipour A (2017) Tversky loss function for image segmentation using 3D fully convolutional deep networks. In: International workshop on machine learning in medical imaging. Springer, pp 379–387

  • Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  • Saxena S, Verbeek J (2016) Convolutional neural fabrics. In: Advances in neural information processing systems, pp 4053–4061

  • Schlemper J, Oktay O, Schaap M, Heinrich M, Kainz B, Glocker B, Rueckert D (2019) Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal 53:197–207

    Article  Google Scholar 

  • Shahriari S, Garcia D (2018) Meshfree simulations of ultrasound vector flow imaging using smoothed particle hydrodynamics. Phys Med Biol 63(20):205011. https://doi.org/10.1088/1361-6560/aae3c3

    Article  Google Scholar 

  • Shaw A, Hunter D, Landola F, Sidhu S (2019) SqueezeNAS: fast neural architecture search for faster semantic segmentation. In: Proceedings of the IEEE international conference on computer vision workshops

  • Shin HC, Tenenholtz NA, Rogers JK, Schwarz CG, Senjem ML, Gunter JL, Andriole KP, Michalski M (2018) Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 1–11

  • Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data. https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  • Sifre L (2014) Rigid-motion scattering for image classification. PhD thesis, CMAP, Ecole Polytechnique

  • Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the seventh international conference on document analysis and recognition—Volume 2. IEEE Computer Society, USA, ICDAR ’03, p 958

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  • Simpson AL, Antonelli M, Bakas S, Bilello M, Farahani K, van Ginneken B, Kopp-Schneider A, Landman BA, Litjens G, Menze B, et al. (2019) A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv:190209063

  • Sinha A, Dolz J (2019) Multi-scale guided attention for medical image segmentation. arXiv:190602849

  • Son J, Park SJ, Jung KH (2017) Retinal vessel segmentation in fundoscopic images with generative adversarial networks. arXiv:170609318

  • Song G, Myeong H, Mu Lee K (2018) Seednet: automatic seed generation with deep reinforcement learning for robust interactive segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1760–1768

  • Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:150500387

  • Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127

    Article  Google Scholar 

  • SUYEgit (2018) Mask R-CNN for surgery robot. https://github.com/SUYEgit/Surgery-Robot-Detection-Segmentation/

  • Tabary J, Hugonnard P, Mathy F (2007) SINDBAD: a realistic multi-purpose and scalable X-ray simulation tool for NDT applications. In: DIR 2007: international symposium on digital industrial radiology and computed tomography

  • Taghanaki SA, Duggan N, Ma H, Hou X, Celler A, Benard F, Hamarneh G (2018) Segmentation-free direct tumor volume and metabolic activity estimation from pet scans. Comput Med Imaging Graphics 63:52–66

    Article  Google Scholar 

  • Taghanaki SA, Abhishek K, Azizi S, Hamarneh G (2019a) A kernelized manifold mapping to diminish the effect of adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 11340–11349

  • Taghanaki SA, Abhishek K, Hamarneh G (2019b) Improved inference via deep input transfer. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 819–827

  • Taghanaki SA, Bentaieb A, Sharma A, Zhou SK, Zheng Y, Georgescu B, Sharma P, Grbic S, Xu Z, Comaniciu D, et al. (2019c) Select, attend, and transfer: light, learnable skip connections. In: Medical image computing and computer-assisted intervention workshop on machine learning in medical imaging

  • Taghanaki SA, Havaei M, Berthier T, Dutil F, Di Jorio L, Hamarneh G, Bengio Y (2019d) InfoMask: masked variational latent representation to localize chest disease. In: International conference on medical image computing and computer assisted intervention

  • Taghanaki SA, Zheng Y, Zhou SK, Georgescu B, Sharma P, Xu D, Comaniciu D, Hamarneh G (2019e) Combo loss: handling input and output imbalance in multi-organ segmentation. Comput Med Imaging Graphics 75:24–33

    Article  Google Scholar 

  • Tajbakhsh N, Jeyaseelan L, Li Q, Chiang JN, Wu Z, Ding X (2020) Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med Image Anal 63:101693. https://doi.org/10.1016/j.media.2020.101693

    Article  Google Scholar 

  • Tarvainen A, Valpola H (2017) Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in neural information processing systems, pp 1195–1204

  • Tsai HF, Gajda J, Sloan TF, Rares A, Shen AQ (2019) Usiigaci: instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning. SoftwareX 9:230–237

    Article  Google Scholar 

  • Vorontsov E, Molchanov P, Byeon W, De Mello S, Jampani V, Liu MY, Kadoury S, Kautz J (2019) Towards semi-supervised segmentation via image-to-image translation. arXiv:190401636

  • Vuola AO, Akram SU, Kannala J (2019) Mask R-CNN and U-net ensembled for nuclei segmentation. arXiv:190110170

  • Wang EK, Zhang X, Pan L, Cheng C, Dimitrakopoulou-Strauss A, Li Y, Zhe N (2019a) Multi-path dilated residual network for nuclei segmentation and detection. Cells 8(5):499

    Article  Google Scholar 

  • Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017a) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  • Wang G, Luo P, Lin L, Wang X (2017b) Learning object interactions and descriptions for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5859–5867

  • Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, Cottrell G (2018a) Understanding convolution for semantic segmentation. In: 2018 IEEE winter conference on applications of computer vision. IEEE, pp 1451–1460

  • Wang S, Rong R, Yang DM, Cai L, Yang L, Luo D, Yao B, Xu L, Wang T, Zhan X, et al. (2019b) Computational staining of pathology images to study tumor microenvironment in lung cancer. Available at SSRN 3391381

  • Wang W, Lai Q, Fu H, Shen J, Ling H (2019c) Salient object detection in the deep learning era: an in-depth survey. arXiv:190409146

  • Wang X, Girshick R, Gupta A, He K (2018b) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  • Wang X, Wang H, Niu S, Zhang J (2019d) Detection and localization of image forgeries using improved mask regional convolutional neural network. Math Biosci Eng 16:4581–4593

    Article  Google Scholar 

  • Wang Z, Sarcar S, Liu J, Zheng Y, Ren X (2018c) Outline objects using deep reinforcement learning. arXiv:180404603

  • Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Advances in neural information processing systems, pp 2074–2082

  • Weng Y, Zhou T, Li Y, Qiu X (2019a) NAS-Unet: neural architecture search for medical image segmentation. IEEE Access 7:44247–44257

    Article  Google Scholar 

  • Weng Y, Zhou T, Li Y, Qiu X (2019b) NAS-unet: neural architecture search for medical image segmentation. IEEE Access 7:44247–44257. https://doi.org/10.1109/access.2019.2908991

    Article  Google Scholar 

  • Wessel J, Heinrich MP, von Berg J, Franz A, Saalbach A (2019) Sequential rib labeling and segmentation in chest X-ray using Mask R-CNN. In: International conference on medical imaging with deep learning—extended abstract track, London, United Kingdom. https://openreview.net/forum?id=SJxuHzLjFV

  • Wojna Z, Ferrari V, Guadarrama S, Silberman N, Chen LC, Fathi A, Uijlings J (2017) The devil is in the decoder. arXiv:170705847

  • Wong KC, Moradi M, Tang H, Syeda-Mahmood T (2018) 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes. In: International conference on medical image computing and computer assisted intervention. Springer, pp 612–619

  • Wu Y, He K (2018) Group normalization. In: Proceedings of the European conference on computer vision, pp 3–19

  • Wu Z, Shen C, Van Den Hengel A (2019) Wider or deeper: revisiting the resnet model for visual recognition. Pattern Recognit 90:119–133

    Article  Google Scholar 

  • Xiao J, Hays J, Ehinger KA, Oliva A, Torralba A (2010) SUN database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 3485–3492

  • Xie S, Tu Z (2015) Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision, pp 1395–1403

  • Xie X, Niu J, Liu X, Chen Z, Tang S (2020) A survey on domain knowledge powered deep learning for medical image analysis. arXiv:200412150

  • Xu X, Lu Q, Yang L, Hu S, Chen D, Hu Y, Shi Y (2018) Quantization of fully convolutional networks for accurate biomedical image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8300–8308

  • Xue Y, Xu T, Zhang H, Long LR, Huang X (2018) SegAN: adversarial network with multi-scale L1 loss for medical image segmentation. Neuroinformatics 16(3–4):383–392

    Article  Google Scholar 

  • Yang D, Xu D, Zhou SK, Georgescu B, Chen M, Grbic S, Metaxas D, Comaniciu D (2017a) Automatic liver segmentation using an adversarial image-to-image network. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 507–515

  • Yang Q, Li N, Zhao Z, Fan X, Chang EI, Xu Y, et al. (2018) MRI cross-modality neuroimage-to-neuroimage translation. arXiv:180106940

  • Yang X, Yu L, Wu L, Wang Y, Ni D, Qin J, Heng PA (2017b) Fine-grained recurrent neural networks for automatic prostate segmentation in ultrasound images. In: Thirty-first AAAI conference on artificial intelligence

  • Yu B, Zhou L, Wang L, Fripp J, Bourgeat P (2018a) 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation. In: 2018 IEEE 15th international symposium on biomedical imaging. IEEE, pp 626–630

  • Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018b) BiSeNet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV). Springer, pp 334–349. https://doi.org/10.1007/978-3-030-01261-8_20

  • Yuan Y (2017) Automatic skin lesion segmentation with fully convolutional-deconvolutional networks. arXiv:170305165

  • Zamir AR, Sax A, Shen W, Guibas LJ, Malik J, Savarese S (2018) Taskonomy: disentangling task transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3712–3722

  • Zhang P, Zhong Y, Deng Y, Tang X, Li X (2019) A survey on deep learning of small sample in biomedical image analysis. arXiv:190800473

  • Zhang W, Witharana C, Liljedahl A, Kanevskiy M (2018a) Deep convolutional neural networks for automated characterization of arctic ice-wedge polygons in very high spatial resolution aerial imagery. Remote Sens 10(9):1487

    Article  Google Scholar 

  • Zhang Y, Yang L, Chen J, Fredericksen M, Hughes DP, Chen DZ (2017) Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 408–416

  • Zhang Y, Miao S, Mansi T, Liao R (2018b) Task driven generative modeling for unsupervised domain adaptation: application to X-ray image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 599–607

  • Zhang Z, Yang L, Zheng Y (2018c) Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9242–9251

  • Zhang Z, Zhang X, Peng C, Xue X, Sun J (2018d) Exfuse: enhancing feature fusion for semantic segmentation. In: Proceedings of the European conference on computer vision, pp 269–284

  • Zhao H, Li H, Cheng L (2017a) Synthesizing filamentary structured images with GANs. arXiv:170602185

  • Zhao H, Shi J, Qi X, Wang X, Jia J (2017b) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  • Zhao M, Hamarneh G (2019a) Retinal image classification viavasculature-guided sequential attention. In: International conference on computer vision workshop on visual recognition for medical images

  • Zhao M, Hamarneh G (2019b) Tree-LSTM: using LSTM to encode memory in anatomical tree prediction from 3D images. In: Medical image computing and computer-assisted intervention workshop on machine learning in medical imaging

  • Zhao T, Yang Y, Niu H, Wang D, Chen Y (2018) Comparing U-Net convolutional network with Mask R-CNN in the performances of pomegranate tree canopy segmentation. In: Asia-pacific remote sensing

  • Zhao ZQ, Zheng P, Xu ST, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232. https://doi.org/10.1109/tnnls.2018.2876865

    Article  Google Scholar 

  • Zhen X, Li S (2015) Towards direct medical image analysis without segmentation. arXiv:151006375

  • Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. In: Advances in neural information processing systems, pp 487–495

  • Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, Torralba A (2017) Scene parsing through ADE20K dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 633–641

  • Zhou S, Nie D, Adeli E, Yin J, Lian J, Shen D (2019a) High-resolution encoder-decoder networks for low-contrast medical image segmentation. IEEE Trans Image Process 29:461–475

    Article  MathSciNet  Google Scholar 

  • Zhou T, Ruan S, Canu S (2019b) A review: deep learning for medical image segmentation using multi-modality fusion. Array 3–4:100004. https://doi.org/10.1016/j.array.2019.100004

    Article  Google Scholar 

  • Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2018) UNet++: a nested U-Net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 3–11

  • Zhu X, Zhou H, Yang C, Shi J, Lin D (2018) Penalizing top performers: conservative loss for semantic segmentation adaptation. In: Proceedings of the European conference on computer vision, pp 568–583

  • Zhu Z, Liu C, Yang D, Yuille A, Xu D (2019) V-NAS: neural architecture search for volumetric medical image segmentation. In: 2019 international conference on 3D vision (3DV). IEEE. https://doi.org/10.1109/3dv.2019.00035

  • Zoph B, Le QV (2016) Neural architecture search with reinforcement learning. arXiv:161101578

  • Zou Z, Shi Z, Guo Y, Ye J (2019) Object detection in 20 years: a survey. arXiv:190505055

Download references

Author information

Authors and Affiliations

  1. School of Computing Science, Simon Fraser University, Burnaby, Canada

    Saeid Asgari Taghanaki, Kumar Abhishek & Ghassan Hamarneh

  2. Mila, Université de Montréal, Montreal, Canada

    Joseph Paul Cohen

  3. NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montréal, Montreal, Canada

    Julien Cohen-Adad

Authors
  1. Saeid Asgari Taghanaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kumar Abhishek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joseph Paul Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julien Cohen-Adad
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ghassan Hamarneh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saeid Asgari Taghanaki.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Saeid Asgari Taghanaki and Kumar Abhishek Joint first authors.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 112 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Asgari Taghanaki, S., Abhishek, K., Cohen, J.P. et al. Deep semantic segmentation of natural and medical images: a review. Artif Intell Rev 54, 137–178 (2021). https://doi.org/10.1007/s10462-020-09854-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-020-09854-1

Keywords

Search

Navigation