Volumetric Medical Image Segmentation: A 3D Deep Coarse-to-Fine Framework and Its Adversarial Examples

  • Yingwei Li
  • Zhuotun Zhu
  • Yuyin Zhou
  • Yingda Xia
  • Wei Shen
  • Elliot K. Fishman
  • Alan L. YuilleEmail author
Part of the Advances in Computer Vision and Pattern Recognition book series (ACVPR)


Although deep neural networks have been a dominant method for many 2D vision tasks, it is still challenging to apply them to 3D tasks, such as medical image segmentation, due to the limited amount of annotated 3D data and limited computational resources. In this chapter, by rethinking the strategy to apply 3D Convolutional Neural Networks to segment medical images, we propose a novel 3D-based coarse-to-fine framework to efficiently tackle these challenges. The proposed 3D-based framework outperforms their 2D counterparts by a large margin since it can leverage the rich spatial information along all three axes. We further analyze the threat of adversarial attacks on the proposed framework and show how to defend against the attack. We conduct experiments on three datasets, the NIH pancreas dataset, the JHMI pancreas dataset and the JHMI pathological cyst dataset, where the first two and the last one contain healthy and pathological pancreases, respectively, and achieve the current state of the art in terms of Dice-Sørensen Coefficient (DSC) on all of them. Especially, on the NIH pancreas dataset, we outperform the previous best by an average of over \(2\%\), and the worst case is improved by \(7\%\) to reach almost \(70\%\), which indicates the reliability of our framework in clinical applications.


  1. 1.
    Bui TD, Shin J, Moon T (2017) 3D densely convolution networks for volumetric segmentation. arXiv:1709.03199
  2. 2.
    Cai J, Lu L, Xie Y, Xing F, Yang L (2017) Improving deep pancreas segmentation in CT and MRI images via recurrent neural contextual learning and direct loss functionGoogle Scholar
  3. 3.
    Chen H, Dou Q, Yu L, Qin J, Heng PA (2017) Voxresnet: deep voxelwise residual networks for brain segmentation from 3D MR images. NeuroImageGoogle Scholar
  4. 4.
    Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2016) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:1606.00915
  5. 5.
    Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAIGoogle Scholar
  6. 6.
    Dou Q, Yu L, Chen H, Jin Y, Yang X, Qin J, Heng PA (2017) 3D deeply supervised network for automated segmentation of volumetric medical images. MIA 41:40–54Google Scholar
  7. 7.
    Fang Y, Xie J, Dai G, Wang M, Zhu F, Xu T, Wong E (2015) 3D deep shape descriptor. In: CVPRGoogle Scholar
  8. 8.
    Gibson E, Giganti F, Hu Y, Bonmati E, Bandula S, Gurusamy K, Davidson B, Pereira SP, Clarkson MJ, Barratt DC (2018) Automatic multi-organ segmentation on abdominal ct with dense v-networks. TMIGoogle Scholar
  9. 9.
    Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: AISTATSGoogle Scholar
  10. 10.
    Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: ICLRGoogle Scholar
  11. 11.
    Gravel P, Beaudoin G, De Guise JA (2004) A method for modeling noise in medical images. TMI 23(10):1221–1232Google Scholar
  12. 12.
    Havaei M, Davy A, Warde-Farley D, Biard A, Courville AC, Bengio Y, Pal C, Jodoin P, Larochelle H (2017) Brain tumor segmentation with deep neural networks. MIA 35:18–31Google Scholar
  13. 13.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPRGoogle Scholar
  14. 14.
    Huang Y, Würfl T, Breininger K, Liu L, Lauritsch G, Maier A (2018) Some investigations on robustness of deep learning in limited angle tomography. In: MICCAIGoogle Scholar
  15. 15.
    Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICMLGoogle Scholar
  16. 16.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) CAFFE: convolutional architecture for fast feature embedding. MMGoogle Scholar
  17. 17.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: NIPSGoogle Scholar
  18. 18.
    Kurakin A, Goodfellow I, Bengio S (2017) Adversarial machine learning at scale. In: ICLRGoogle Scholar
  19. 19.
    Lasboo AA, Rezai P, Yaghmai V (2010) Morphological analysis of pancreatic cystic masses. Acad Radiol 17(3):348–351CrossRefGoogle Scholar
  20. 20.
    Lee CY, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply-supervised nets. In: AISTATSGoogle Scholar
  21. 21.
    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: CVPRGoogle Scholar
  22. 22.
    Madry A, Makelov A, Schmidt L, Tsipras D, Vladu, A (2018) Towards deep learning models resistant to adversarial attacks. In: ICLRGoogle Scholar
  23. 23.
    Merkow J, Marsden A, Kriegman D, Tu Z (2016) Dense volume-to-volume vascular boundary detection. In: MICCAIGoogle Scholar
  24. 24.
    Milletari F, Navab N, Ahmadi SA (2016) V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3DVGoogle Scholar
  25. 25.
    Moeskops P, Wolterink JM, van der Velden BHM, Gilhuijs KGA, Leiner T, Viergever MA, Isgum I (2017) Deep learning for multi-task medical image segmentation in multiple modalities. CoRR arXiv:1704.03379
  26. 26.
    Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: ICMLGoogle Scholar
  27. 27.
    Paschali M, Conjeti S, Navarro F, Navab N (2018) Generalizability versus robustness: adversarial examples for medical imaging. In: MICCAIGoogle Scholar
  28. 28.
    Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted interventionGoogle Scholar
  29. 29.
    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: MICCAIGoogle Scholar
  30. 30.
    Roth H, Oda M, Shimizu N, Oda H, Hayashi Y, Kitasaka T, Fujiwara M, Misawa K, Mori K (2018) Towards dense volumetric pancreas segmentation in CT using 3D fully convolutional networks. In: SPIEGoogle Scholar
  31. 31.
    Roth HR, Lu L, Farag A, Shin HC, Liu J, Turkbey EB, Summers RM (2015) Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. In: MICCAIGoogle Scholar
  32. 32.
    Roth HR, Lu L, Farag A, Sohn A, Summers RM (2016) Spatial aggregation of holistically-nested networks for automated pancreas segmentation. In: MICCAIGoogle Scholar
  33. 33.
    Shen W, Wang B, Jiang Y, Wang Y, Yuille AL (2017) Multi-stage multi-recursive-input fully convolutional networks for neuronal boundary detection. In: ICCV, pp 2410–2419Google Scholar
  34. 34.
    Shen W, Wang X, Wang Y, Bai X, Zhang Z (2015) Deepcontour: a deep convolutional feature learned by positive-sharing loss for contour detection. In: CVPRGoogle Scholar
  35. 35.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  36. 36.
    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: CVPRGoogle Scholar
  37. 37.
    Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2014) Intriguing properties of neural networks. In: ICLRGoogle Scholar
  38. 38.
    Tsipras D, Santurkar S, Engstrom L, Turner A, Madry A (2018) Robustness may be at odds with accuracy, p 1. arXiv:1805.12152
  39. 39.
    Wang Y, Zhou Y, Shen W, Park S, Fishman EK, Yuille AL (2018) Abdominal multi-organ segmentation with organ-attention networks and statistical fusion. CoRR. arXiv:1804.08414
  40. 40.
    Wang Y, Zhou Y, Tang P, Shen W, Fishman EK, Yuille AL (2018) Training multi-organ segmentation networks with sample selection by relaxed upper confident bound. In: Proceedings of MICCAI, pp. 434–442CrossRefGoogle Scholar
  41. 41.
    Xie C, Wang J, Zhang Z, Zhou Y, Xie L, Yuille A (2017) Adversarial examples for semantic segmentation and object detection. In: ICCVGoogle Scholar
  42. 42.
    Xie S, Tu Z (2015) Holistically-nested edge detection. In: ICCVGoogle Scholar
  43. 43.
    Yu L, Cheng JZ, Dou Q, Yang X, Chen H, Qin J, Heng PA (2017) Automatic 3D cardiovascular MR segmentation with densely-connected volumetric convnets. In: MICCAIGoogle Scholar
  44. 44.
    Yu L, Yang X, Chen H, Qin J, Heng P (2017) Volumetric convnets with mixed residual connections for automated prostate segmentation from 3D MR images. In: AAAIGoogle Scholar
  45. 45.
    Zhou Y, Xie L, Fishman EK, Yuille AL (2017) Deep supervision for pancreatic cyst segmentation in abdominal CT scans. In: MICCAIGoogle Scholar
  46. 46.
    Zhou Y, Xie L, Shen W, Wang Y, Fishman EK, Yuille AL (2017) A fixed-point model for pancreas segmentation in abdominal CT scans. In: MICCAIGoogle Scholar
  47. 47.
    Zhu Z, Wang X, Bai S, Yao C, Bai X (2016) Deep learning representation using autoencoder for 3D shape retrieval. Neurocomputing 204:41–50CrossRefGoogle Scholar
  48. 48.
    Zhu Z, Xia Y, Shen W, Fishman EK, Yuille AL (2018) A 3d coarse-to-fine framework for volumetric medical image segmentation. In: International conference on 3D vision, pp 682–690Google Scholar
  49. 49.
    Zhu Z, Xia Y, Xie L, Fishman EK, Yuille AL (2018) Multi-scale coarse-to-fine segmentation for screening pancreatic ductal adenocarcinoma. arXiv:1807.02941

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yingwei Li
    • 1
  • Zhuotun Zhu
    • 1
  • Yuyin Zhou
    • 1
  • Yingda Xia
    • 1
  • Wei Shen
    • 1
  • Elliot K. Fishman
    • 2
  • Alan L. Yuille
    • 1
    Email author
  1. 1.Johns Hopkins UniversityBaltimoreUSA
  2. 2.Johns Hopkins University School of MedicineBaltimoreUSA

Personalised recommendations