The Visual Computer

, Volume 33, Issue 6–8, pp 1061–1071 | Cite as

Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation

  • Lei Bi
  • Jinman KimEmail author
  • Ashnil Kumar
  • Michael Fulham
  • Dagan Feng
Original Article


The automated segmentation of regions of interest (ROIs) in medical imaging is the fundamental requirement for the derivation of high-level semantics for image analysis in clinical decision support systems. Traditional segmentation approaches such as region-based depend heavily upon hand-crafted features and a priori knowledge of the user. As such, these methods are difficult to adopt within a clinical environment. Recently, methods based on fully convolutional networks (FCN) have achieved great success in the segmentation of general images. FCNs leverage a large labeled dataset to hierarchically learn the features that best correspond to the shallow appearance as well as the deep semantics of the images. However, when applied to medical images, FCNs usually produce coarse ROI detection and poor boundary definitions primarily due to the limited number of labeled training data and limited constraints of label agreement among neighboring similar pixels. In this paper, we propose a new stacked FCN architecture with multi-channel learning (SFCN-ML). We embed the FCN in a stacked architecture to learn the foreground ROI features and background non-ROI features separately and then integrate these different channels to produce the final segmentation result. In contrast to traditional FCN methods, our SFCN-ML architecture enables the visual attributes and semantics derived from both the fore- and background channels to be iteratively learned and inferred. We conducted extensive experiments on three public datasets with a variety of visual challenges. Our results show that our SFCN-ML is more effective and robust than a routine FCN and its variants, and other state-of-the-art methods.


Fully convolutional networks (FCNs) Segmentation Regions of interest (ROI) 


  1. 1.
    Musen, M.A., et al.: Clinical decision-support systems. Biomedical informatics. Springer, London (2014)CrossRefGoogle Scholar
  2. 2.
    Doi, K.: Current status and future potential of computer-aided diagnosis in medical imaging. Br. J. Radiol. 78, s3–s19 (2005)Google Scholar
  3. 3.
    Bi, L., et al.: Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies, Comput. Med. Imaging Graph. (2016). doi: 10.1016/j.compmedimag.2016.11.008
  4. 4.
    Chen, X., et al.: Medical image segmentation by combining graph cuts and oriented active appearance models. IEEE Trans. Image Process. 21, 2035–2046 (2012)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Ahn, E., et al.: Saliency-based Lesion Segmentation via Background Detection in Dermoscopic Images. IEEE J. Biomed. Health Inform. (2017). doi: 10.1109/JBHI.2017.2653179
  6. 6.
    Li, B.N., et al.: Integrating spatial fuzzy clustering with level set methods for automated medical image segmentation. Comput. Biol. Med. 41, 1–10 (2011)CrossRefGoogle Scholar
  7. 7.
    Long, J., et al.: Fully convolutional networks for semantic segmentation, In: Proc. IEEE CVPR, 3431–3440 (2015)Google Scholar
  8. 8.
    Chen, L.-C., et al.: Semantic image segmentation with deep convolutional nets and fully connected crfs, In: Proc. ICLR (2015)Google Scholar
  9. 9.
    Zheng, S., et al.: Conditional random fields as recurrent neural networks, In: Proc. IEEE ICCV, 1529–1537 (2015)Google Scholar
  10. 10.
    Chen, H., et al.: DCAN: Deep contour-aware networks for object instance segmentation from histology images. Med. Image Anal. 36, 135–146 (2017)CrossRefGoogle Scholar
  11. 11.
    Dou, Q., et al.: 3d deeply supervised network for automatic liver segmentation from ct volumes, In: Proc. MICCAI, 149–157 (2016)Google Scholar
  12. 12.
    Xu, Y., et al.: Gland instance segmentation by deep multichannel side supervision. In: Proc. MICCAI, 496–504 (2016)Google Scholar
  13. 13.
    BenTaieb, A., et al.: Topology Aware Fully Convolutional Networks for Histology Gland Segmentation. In: Proc. MICCAI, 460–468 (2016)Google Scholar
  14. 14.
    Paulano, F., et al.: 3D segmentation and labeling of fractured bone from CT images. Vis. Comput. 30, 939–948 (2014)CrossRefGoogle Scholar
  15. 15.
    Lin, L., et al.: Inference with collaborative model for interactive tumor segmentation in medical image sequences. IEEE Trans. Cybern. 46, 2796–2809 (2016)CrossRefGoogle Scholar
  16. 16.
    Bi, L., et al.: Cellular automata and anisotropic diffusion filter based interactive tumor segmentation for positron emission tomography, IEEE EMBC, 5453–5456 (2013)Google Scholar
  17. 17.
    ONeill, G.T., et al.: Segmentation of cam-type femurs from CT scans. Vis. Comput. 28, 205–218 (2012)CrossRefGoogle Scholar
  18. 18.
    Silveira, M., et al.: Comparison of segmentation methods for melanoma diagnosis in dermoscopy images. IEEE J. Sel. Top. Signal Process. 3, 35–45 (2009)CrossRefGoogle Scholar
  19. 19.
    Emre Celebi, M., et al.: Lesion border detection in dermoscopy images using ensembles of thresholding methods. Skin Res. Technol. 19, e252–e258 (2013)CrossRefGoogle Scholar
  20. 20.
    Li, C., et al.: Minimization of region-scalable fitting energy for image segmentation. IEEE Trans. Image Process. 17, 1940–1949 (2008)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Li, C., et al.: Distance regularized level set evolution and its application to image segmentation. IEEE Trans. Image Process. 19, 3243–3254 (2010)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Wong, A., et al.: Automatic skin lesion segmentation via iterative stochastic region merging. IEEE Trans. Inf. Technol. Biomed. 15, 929–936 (2011)CrossRefGoogle Scholar
  23. 23.
    Somkantha, K., et al.: Boundary detection in medical images using edge following algorithm based on intensity gradient and texture gradient features. IEEE Trans. Biomed. Eng. 58, 567–573 (2011)CrossRefGoogle Scholar
  24. 24.
    Roy, A., et al.: JCLMM: A finite mixture model for clustering of circular–linear data and its application to psoriatic plaque segmentation, Pattern Recognit. (2017)Google Scholar
  25. 25.
    Nock, R., et al.: Statistical region merging. IEEE Trans. Pattern Anal. Mach. Intel. 26, 1452–1458 (2004)CrossRefGoogle Scholar
  26. 26.
    van Rikxoort, E.M., et al.: Adaptive local multi-atlas segmentation: application to the heart and the caudate nucleus. Med. Image Anal. 14, 39 (2010)CrossRefGoogle Scholar
  27. 27.
    Song, Y., et al.: Similarity guided feature labeling for lesion detection, In Proc. MICCAI, 284–291 (2013)Google Scholar
  28. 28.
    Li, C., et al.: Automated PET-guided liver segmentation from low-contrast CT volumes using probabilistic atlas. Comput. Method Program Biomed. 107, 164–174 (2012)CrossRefGoogle Scholar
  29. 29.
    Bi, L., et al.: Automatic Descending Aorta Segmentation in Whole-Body PET-CT Studies for PERCIST-Based Thresholding, In: Proc. DICTA, 1–6 (2012)Google Scholar
  30. 30.
    Li, C., et al.: Joint probabilistic model of shape and intensity for multiple abdominal organ segmentation from volumetric CT images. IEEE J. Biomed. Health Inform. 17, 92–102 (2013)CrossRefGoogle Scholar
  31. 31.
    Isgum, I., et al.: Multi-atlas-based segmentation with local decision fusion: application to cardiac and aortic segmentation in CT scans. IEEE Trans. Med. Imaging 28, 1000–1010 (2009)CrossRefGoogle Scholar
  32. 32.
    Bi, L., et al.: Multi-stage thresholded region classification for whole-body PET–CT lymphoma studies, In: Proc. MICCAI, 569–576 (2014)Google Scholar
  33. 33.
    Song, Y., et al.: A multistage discriminative model for tumor and lymph node detection in thoracic images. IEEE Trans. Med. Imaging 31, 1061–1075 (2012)CrossRefGoogle Scholar
  34. 34.
    Lartizien, C., et al.: Computer-aided staging of lymphoma patients with FDG PET/CT imaging based on textural information. IEEE J. Biomed. Health Inform. 18, 946–955 (2014)CrossRefGoogle Scholar
  35. 35.
    Ciresan, D., et al.: Deep neural networks segment neuronal membranes in electron microscopy images, In: Proc. NIPS, 2843–2851 (2012)Google Scholar
  36. 36.
    Cha, K.H., et al.: Urinary bladder segmentation in CT urography using deeplearning convolutional neural network and level sets. Med. Phys. 43, 1882–1896 (2016)CrossRefGoogle Scholar
  37. 37.
    Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)CrossRefGoogle Scholar
  38. 38.
    Roth, H.R., et al.: Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation, In: Proc. MICCAI, 556–564 (2015)Google Scholar
  39. 39.
    Farag, A., et al.: A bottom-up approach for pancreas segmentation using cascaded superpixels and (deep) image patch labeling. IEEE Trans. Image Process. 26, 386–399 (2017)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Ronneberger, O., et al.: U-net: convolutional networks for biomedical image segmentation. In: Proc. MICCAI, 234–241 (2015)Google Scholar
  41. 41.
    Fu, H., et al.: Retinal vessel segmentation via deep learning network and fully-connected conditional random fields, In: Proc. IEEE ISBI, 698–701 (2016)Google Scholar
  42. 42.
    Lu, F., et al.: Automatic 3D liver location and segmentation via convolutional neural network and graph cut. Int. J. Comput. Assist. Radiol. Surg. 12, 171–182 (2017)Google Scholar
  43. 43.
    Hu, P., et al.: Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets, Int. J. Comput. Assist. Radiol. Surg. 1–13, (2016)Google Scholar
  44. 44.
    Chatfield, K., et al.: Return of the devil in the details: delving deep into convolutional nets, BMVC (2014)Google Scholar
  45. 45.
    Wang, L., et al.: Saliency detection with recurrent fully convolutional networks, In: Proc. ECCV, 825–841 (2016)Google Scholar
  46. 46.
    Dean, J., et al.: Large scale distributed deep networks, In: NIPS, 1223–1231 (2012)Google Scholar
  47. 47.
    Hogeweg, L., et al.: Clavicle segmentation in chest radiographs. Med. Image Anal. 16, 1490–1502 (2012)CrossRefGoogle Scholar
  48. 48.
    Sirinukunwattana, K., et al.: A stochastic polygons model for glandular structures in colon histology images. IEEE Trans. Med. Imaging 34, 2366–2378 (2015)CrossRefGoogle Scholar
  49. 49.
    Shin, H.-C., et al.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016)CrossRefGoogle Scholar
  50. 50.
    Deng, J., et al.: Imagenet: a large-scale hierarchical image database, In: Proc. CVPR, 248–255, (2009)Google Scholar
  51. 51.
    Kumar, A., et al.: An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE J. Biomed. Health Inform. 21, 31–40 (2017)CrossRefGoogle Scholar
  52. 52.
    Vedaldi, A., et al.: Matconvnet: Convolutional neural networks for matlab, In: Proc. ACM MM, 689–692 (2015)Google Scholar
  53. 53.
    Krizhevsky, A., et al.: Imagenet classification with deep convolutional neural networks, In: NIPS, 1097–1105 (2012)Google Scholar
  54. 54.
    Li, X., et al.: Contextual hypergraph modeling for salient object detection, In: Proc. ICCV, 3328–3335 (2013)Google Scholar
  55. 55.
    Chan, T.F., Vese, L.A.: Active contour and segmentation models using geometric PDEs for medical imaging. Geometric methods in bio-medical image processing. Springer, Berlin (2002)zbMATHGoogle Scholar
  56. 56.
    Zhang, Y., et al.: Medical image segmentation using new hybrid level-set method, In: Proc. MEDIVIS, 71–76 (2008)Google Scholar
  57. 57.
    Shotton, J., et al.: Semantic texton forests for image categorization and segmentation, In: Proc. CVPR, 1–8 (2008)Google Scholar
  58. 58.
    Bai, X., et al.: Video snapcut: robust video object cutout using localized classifiers, In: ACM Trans. Graph. (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Lei Bi
    • 1
  • Jinman Kim
    • 1
    Email author
  • Ashnil Kumar
    • 1
  • Michael Fulham
    • 1
    • 2
    • 3
  • Dagan Feng
    • 1
    • 4
  1. 1.School of Information TechnologiesThe University of SydneySydneyAustralia
  2. 2.Department of Molecular ImagingRoyal Prince Alfred HospitalSydneyAustralia
  3. 3.Sydney Medical SchoolThe University of SydneySydneyAustralia
  4. 4.Med-X Research InstituteShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations