Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets

  • Peijun Hu
  • Fa Wu
  • Jialin Peng
  • Yuanyuan Bao
  • Feng Chen
  • Dexing Kong
Original Article



Multi-organ segmentation from CT images is an essential step for computer-aided diagnosis and surgery planning. However, manual delineation of the organs by radiologists is tedious, time-consuming and poorly reproducible. Therefore, we propose a fully automatic method for the segmentation of multiple organs from three-dimensional abdominal CT images.


The proposed method employs deep fully convolutional neural networks (CNNs) for organ detection and segmentation, which is further refined by a time-implicit multi-phase evolution method. Firstly, a 3D CNN is trained to automatically localize and delineate the organs of interest with a probability prediction map. The learned probability map provides both subject-specific spatial priors and initialization for subsequent fine segmentation. Then, for the refinement of the multi-organ segmentation, image intensity models, probability priors as well as a disjoint region constraint are incorporated into an unified energy functional. Finally, a novel time-implicit multi-phase level-set algorithm is utilized to efficiently optimize the proposed energy functional model.


Our method has been evaluated on 140 abdominal CT scans for the segmentation of four organs (liver, spleen and both kidneys). With respect to the ground truth, average Dice overlap ratios for the liver, spleen and both kidneys are 96.0, 94.2 and 95.4%, respectively, and average symmetric surface distance is less than 1.3 mm for all the segmented organs. The computation time for a CT volume is 125 s in average. The achieved accuracy compares well to state-of-the-art methods with much higher efficiency.


A fully automatic method for multi-organ segmentation from abdominal CT images was developed and evaluated. The results demonstrated its potential in clinical usage with high effectiveness, robustness and efficiency.


Multi-organ segmentation Deep CNN Time-implicit multi-phase level sets 3D CT 



This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 11271323, 91330105, 11401231), Zhejiang Provincial Natural Science Foundation of China (Grant No. LZ13A010002), Natural Science Foundation of Fujian Province (Grant No. 2015J01254) and Science Technology Foundation for Middle-aged and Young Teacher of Fujian Province (Grant No. JA14021). J. Peng was also supported by Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

This article does not contain any studies with human participants or animal performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Brosch T, Tang LY, Yoo Y, Li DK, Traboulsee A, Tam R (2016) Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans Med Imaging 35(5):1229–1239CrossRefPubMedGoogle Scholar
  2. 2.
    Cha KH, Hadjiiski L, Samala RK, Chan HP, Caoili EM, Cohan RH (2016) Urinary bladder segmentation in ct urography using deep-learning convolutional neural network and level sets. Med Phys 43(4):1882–1896CrossRefPubMedGoogle Scholar
  3. 3.
    Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2015) Semantic image segmentation with deep convolutional nets and fully connected crfs. In: International conference on learning representations (ICLR)Google Scholar
  4. 4.
    Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Ourselin S, Joskowicz L, Sabuncu MR, Unal G, Wells W (eds) International conference on medical image computing and computer-assisted intervention, (MICCAI), Springer, pp 424–432Google Scholar
  5. 5.
    Ciresan D, Giusti A, Gambardella LM, Schmidhuber J (2012) Deep neural networks segment neuronal membranes in electron microscopy images. In: Advances in neural information processing systems, pp 2843–2851Google Scholar
  6. 6.
    Criminisi A, Robertson D, Konukoglu E, Shotton J, Pathak S, White S, Siddiqui K (2013) Regression forests for efficient anatomy detection and localization in computed tomography scans. Med Image Anal 17(8):1293–1303CrossRefPubMedGoogle Scholar
  7. 7.
    Dou Q, Chen H, Jin Y, Yu L, Qin J, Heng PA (2016) 3d deeply supervised network for automatic liver segmentation from ct volumes. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 149–157Google Scholar
  8. 8.
    Gauriau R, Cuingnet R, Lesage D, Bloch I (2015) Multi-organ localization with cascaded global-to-local regression and shape prior. Med Image Anal 23(1):70–83CrossRefPubMedGoogle Scholar
  9. 9.
    He B, Huang C, Jia F (2015) Fully automatic multi-organ segmentation based on multi-boost learning and statistical shape model search. VISCERAL@ ISBI 2015 VISCERAL Anatomy3 Organ Segmentation Challenge, p 18Google Scholar
  10. 10.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034Google Scholar
  11. 11.
    Heimann T, van Ginneken B, Styner M, Arzhaeva Y, Aurich V, Bauer C, Beck A, Becker C, Beichel R, Bekes G, Bello F, Binnig G, Bischof H, Bornik A, Cashman P, Chi Y, Cordova A, Dawant B, Fidrich M, Furst J, Furukawa D, Grenacher L, Hornegger J, Kainmuller D, Kitney R, Kobatake H, Lamecker H, Lange T, Lee J, Lennon B, Li R, Li S, Meinzer HP, Nemeth G, Raicu D, Rau AM, vanRikxoort E, Rousson M, Rusko L, Saddi K, Schmidt G, Seghers D, Shimizu A, Slagmolen P, Sorantin E, Soza G, Susomboon R, Waite J, Wimmer A, Wolf I (2009) Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans Med Imaging 28(8):1251–1265Google Scholar
  12. 12.
    Hu P, Wu F, Peng J, Liang P, Kong D (2016) Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution. Phys Med Biol 61:8676–8698. doi: 10.1088/1361-6560/61/24/8676
  13. 13.
    Jawarneh MS, Mandava R, Ramachandram D, Shuaib IL (2010) Automatic initialization of contour for level set algorithms guided by integration of multiple views to segment abdominal ct scans. In: 2010 Second international conference on computational intelligence, modelling and simulation (CIMSiM). IEEE, pp 315–320Google Scholar
  14. 14.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  15. 15.
    Lai M (2015) Deep learning for medical image segmentation. arXiv preprint arXiv:1505.02000
  16. 16.
    Lee CY, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply-supervised nets. In: AISTATS, vol 2, p 6Google Scholar
  17. 17.
    Linguraru MG, Pura JA, Pamulapati V, Summers RM (2012) Statistical 4d graphs for multi-organ abdominal segmentation from multiphase ct. Med Image Anal 16(4):904–914CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440Google Scholar
  19. 19.
    Lu F, Wu F, Hu P, Peng Z, Kong D. Automatic 3d liver location and segmentation via convolutional neural networks and graph cut. Int J Comput Assist Radiol Surg. doi: 10.1007/s11548-016-1467-3
  20. 20.
    Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. arXiv preprint arXiv:1606.04797
  21. 21.
    Okada T, Linguraru MG, Hori M, Summers RM, Tomiyama N, Sato Y (2015) Abdominal multi-organ segmentation from ct images using conditional shape-location and unsupervised intensity priors. Med Image Anal 26(1):1–18CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Peng J, Dong F, Chen Y, Kong D (2014) A region-appearance-based adaptive variational model for 3d liver segmentation. Med Phys 41(4):043,502CrossRefGoogle Scholar
  23. 23.
    Peng J, Hu P, Lu F, Peng Z, Kong D, Zhang H (2015) 3d liver segmentation using multiple region appearances and graph cuts. Med Phys 42(12):6840–6852CrossRefPubMedGoogle Scholar
  24. 24.
    Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639CrossRefGoogle Scholar
  25. 25.
    Potts RB (1952) Some generalized order-disorder transformations. In: Mathematical proceedings of the Cambridge philosophical society, vol 48. Cambridge University Press, pp 106–109Google Scholar
  26. 26.
    Rajchl M, Baxter JS, Bae E, Tai XC, Fenster A, Peters TM, Yuan J (2015) Variational time-implicit multiphase level-sets. In: Energy minimization methods in computer vision and pattern recognition. Springer, pp 278–291Google Scholar
  27. 27.
    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241Google Scholar
  28. 28.
    Roth HR, Lu L, Farag A, Shin HC, Liu J, Turkbey EB, Summers RM (2015) Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. In: Medical image computing and computer-assisted intervention. Springer, pp 556–564Google Scholar
  29. 29.
    Saito A, Nawano S, Shimizu A (2016) Joint optimization of segmentation and shape prior from level-set-based statistical shape model, and its application to the automated segmentation of abdominal organs. Med Image Anal 28:46–65CrossRefPubMedGoogle Scholar
  30. 30.
    Selver MA (2014) Segmentation of abdominal organs from ct using a multi-level, hierarchical neural network strategy. Comput Methods Programs Biomed 113(3):830–852CrossRefPubMedGoogle Scholar
  31. 31.
    Tong T, Wolz R, Wang Z, Gao Q, Misawa K, Fujiwara M, Mori K, Hajnal JV, Rueckert D (2015) Discriminative dictionary learning for abdominal multi-organ segmentation. Med Image Anal 23(1):92–104CrossRefPubMedGoogle Scholar
  32. 32.
    Vese LA, Chan TF (2002) A multiphase level set framework for image segmentation using the mumford and shah model. Int J Comput Vis 50(3):271–293CrossRefGoogle Scholar
  33. 33.
    Wang C, Smedby O (2014) Automatic multi-organ segmentation using fast model based level set method and hierarchical shape priors. Proc VISCERAL Chall ISBI 1194:25–31Google Scholar
  34. 34.
    Wolz R, Chu C, Misawa K, Fujiwara M, Mori K, Rueckert D (2013) Automated abdominal multi-organ segmentation with subject-specific atlas generation. IEEE Trans Med Imaging 32(9):1723–1730CrossRefPubMedGoogle Scholar
  35. 35.
    Xu Z, Burke RP, Lee CP, Baucom RB, Poulose BK, Abramson RG, Landman BA (2015) Efficient multi-atlas abdominal segmentation on clinically acquired ct with simple context learning. Med Image Anal 24(1):18–27CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    Yuan J, Bae E, Tai XC, Boykov Y (2010) A continuous max-flow approach to potts model. In: European conference on computer vision. Springer, pp 379–392Google Scholar
  37. 37.
    Yuan J, Ukwatta E, Tai X, Fenster A, Schnoerr C (2012) A fast global optimization-based approach to evolving contours with generic shape prior. University of California, Los Angeles. Technical report CAM 12, vol 38Google Scholar
  38. 38.
    Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, Shen D (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage 108:214–224CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© CARS 2016

Authors and Affiliations

  1. 1.School of Mathematical SciencesZhejiang UniversityHangzhouChina
  2. 2.College of Computer Science and TechnologyHuaqiao UniversityXiamenChina
  3. 3.Department of Radiology, First Affiliated HospitalZhejiang UniversityHangzhouChina

Personalised recommendations