Pancreas Segmentation in MRI Using Graph-Based Decision Fusion on Convolutional Neural Networks

  • Jinzheng Cai
  • Le Lu
  • Yuanpu Xie
  • Fuyong Xing
  • Lin YangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)


Deep neural networks have demonstrated very promising performance on accurate segmentation of challenging organs (e.g., pancreas) in abdominal CT and MRI scans. The current deep learning approaches conduct pancreas segmentation by processing sequences of 2D image slices independently through deep, dense per-pixel masking for each image, without explicitly enforcing spatial consistency constraint on segmentation of successive slices. We propose a new convolutional/recurrent neural network architecture to address the contextual learning and segmentation consistency problem. A deep convolutional sub-network is first designed and pre-trained from scratch. The output layer of this network module is then connected to recurrent layers and can be fine-tuned for contextual learning, in an end-to-end manner. Our recurrent sub-network is a type of Long short-term memory (LSTM) network that performs segmentation on an image by integrating its neighboring slice segmentation predictions, in the form of a dependent sequence processing. Additionally, a novel segmentation-direct loss function (named Jaccard Loss) is proposed and deep networks are trained to optimize Jaccard Index (JI) directly. Extensive experiments are conducted to validate our proposed deep models, on quantitative pancreas segmentation using both CT and MRI scans. Our method outperforms the state-of-the-art work on CT [11] and MRI pancreas segmentation [1], respectively.


  1. 1.
    Cai, J., Lu, L., Zhang, Z., Xing, F., Yang, L., Yin, Q.: Pancreas segmentation in MRI using graph-based decision fusion on convolutional neural networks. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 442–450. Springer, Cham (2016). doi: 10.1007/978-3-319-46723-8_51 CrossRefGoogle Scholar
  2. 2.
    Chen, J., Yang, L., Zhang, Y., Alber, M.S., Chen, D.Z.: Combining fully convolutional and recurrent neural networks for 3D biomedical image segmentation. CoRR abs/1609.01006 (2016)Google Scholar
  3. 3.
    Farag, A., Lu, L., Roth, H.R., Liu, J., Turkbey, E., Summers, R.M.: A bottom-up approach for pancreas segmentation using cascaded superpixels and (deep) image patch labeling. IEEE Trans. Image Process. 26(1), 386–399 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Kamnitsas, K., Ledig, C., Newcombe, V.F.J., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. CoRR abs/1603.05959 (2016)Google Scholar
  5. 5.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE CVPR, pp. 3431–3440, June 2015Google Scholar
  6. 6.
    Merkow, J., Kriegman, D.J., Marsden, A., Tu, Z.: Dense volume-to-volume vascular boundary detection. CoRR abs/1605.08401 (2016)Google Scholar
  7. 7.
    Milletari, F., Navab, N., Ahmadi, S.: V-net: fully convolutional neural networks for volumetric medical image segmentation. CoRR abs/1606.04797 (2016)Google Scholar
  8. 8.
    Oda, M., et al.: Regression forest-based atlas localization and direction specific atlas generation for pancreas segmentation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 556–563. Springer, Cham (2016). doi: 10.1007/978-3-319-46723-8_64 CrossRefGoogle Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi: 10.1007/978-3-319-24574-4_28 CrossRefGoogle Scholar
  10. 10.
    Roth, H.R., Lu, L., Farag, A., Shin, H.-C., Liu, J., Turkbey, E.B., Summers, R.M.: DeepOrgan: multi-level deep convolutional networks for automated pancreas segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 556–564. Springer, Cham (2015). doi: 10.1007/978-3-319-24553-9_68 CrossRefGoogle Scholar
  11. 11.
    Roth, H.R., Lu, L., Farag, A., Sohn, A., Summers, R.M.: Spatial aggregation of holistically-nested networks for automated pancreas segmentation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 451–459. Springer, Cham (2016). doi: 10.1007/978-3-319-46723-8_52 CrossRefGoogle Scholar
  12. 12.
    Roth, H.R., Lu, L., Lay, N., Harrison, A.P., Farag, A., Summers, R.M.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. CoRR abs/1702.00045 (2017)Google Scholar
  13. 13.
    Shi, X., Chen, Z., Wang, H., Yeung, D., Wong, W., Woo, W.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. CoRR abs/1506.04214 (2015)Google Scholar
  14. 14.
    Stollenga, M.F., Byeon, W., Liwicki, M., Schmidhuber, J.: Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. CoRR abs/1506.07452 (2015)Google Scholar
  15. 15.
    Tong, T., Wolz, R., Wang, Z., Gao, Q., Misawa, K., Fujiwara, M., Mori, K., Hajnal, J.V., Rueckert, D.: Discriminative dictionary learning for abdominal multi-organ segmentation. Med. Image Anal. 23(1), 92–104 (2015)CrossRefGoogle Scholar
  16. 16.
    Wolz, R., Chu, C., Misawa, K., Fujiwara, M., Mori, K., Rueckert, D.: Automated abdominal multi-organ segmentation with subject-specific atlas generation. IEEE Trans. Med. Imaging 32(9), 1723–1730 (2013)CrossRefGoogle Scholar
  17. 17.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: IEEE ICCV, pp. 1395–1403 (2015)Google Scholar
  18. 18.
    Zhou, Y., Xie, L., Shen, W., Fishman, E., Yuille, A.L.: Pancreas segmentation in abdominal CT scan: a coarse-to-fine approach. CoRR abs/1612.08230 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jinzheng Cai
    • 1
  • Le Lu
    • 3
  • Yuanpu Xie
    • 1
  • Fuyong Xing
    • 2
  • Lin Yang
    • 1
    • 2
    Email author
  1. 1.Department of Biomedical EngineeringUniversity of FloridaGainesvilleUSA
  2. 2.Department of Electrical and Computer EngineeringUniversity of FloridaGainesvilleUSA
  3. 3.Department of Radiology and Imaging SciencesNational Institutes of Health Clinical CenterBethesdaUSA

Personalised recommendations