Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network

  • Liyan Sun
  • Zhiwen Fan
  • Xinghao DingEmail author
  • Yue Huang
  • John Paisley
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11492)


The need for fast acquisition and automatic analysis of MRI data is growing. Although compressed sensing magnetic resonance imaging (CS-MRI) has been studied to accelerate MRI by reducing k-space measurements, current techniques overlook downstream applications such as segmentation when doing image reconstruction. In this paper, we test the utility of CS-MRI when performing automatic segmentation and propose a unified deep neural network architecture called SegNetMRI for simultaneous CS-MRI reconstruction and segmentation. SegNetMRI uses an MRI reconstruction network with multiple cascaded blocks, each containing an encoder-decoder unit and a data fidelity unit, and a parallel MRI segmentation network having the same encoder-decoder structure. The two subnetworks are pre-trained and fine-tuned with shared reconstruction encoders. The outputs are merged into the final segmentation. Our experiments show that SegNetMRI can improve both the reconstruction and segmentation performance when using compressed measurements.


Compressed sensing Magnetic resonance imaging Medical image segmentation 



This work was supported in part by the National Natural Science Foundation of China under Grants 61571382, 81671766, 61571005, 81671674, 61671309 and U1605252, in part by the Fundamental Research Funds for the Central Universities under Grant 20720160075, 20720180059, in part by the CCF-Tencent open fund and, the Natural Science Foundation of Fujian Province of China (No. 2017J01126). L. Sun conducted portions of this work at Columbia University under China Scholarship Council grant No. 201806310090.


  1. 1.
    Aljabar, P., Heckemann, R.A., Hammers, A., Hajnal, J.V., Rueckert, D.: Multi-atlas based segmentation of brain images: atlas selection and its effect on accuracy. Neuroimage 46(3), 726–738 (2009)CrossRefGoogle Scholar
  2. 2.
    Artaechevarria, X., Munoz-Barrutia, A., Ortiz-de Solórzano, C.: Combination strategies in multi-atlas image segmentation: application to brain MR data. IEEE Trans. Med. Imaging 28(8), 1266–1277 (2009)CrossRefGoogle Scholar
  3. 3.
    Atkinson, D., et al.: Automatic compensation of motion artifacts in MRI. Magn. Reson. Med. 41(1), 163–170 (1999)CrossRefGoogle Scholar
  4. 4.
    Bahrami, K., Rekik, I., Shi, F., Shen, D.: Joint reconstruction and segmentation of 7T-like MR images from 3T MRI based on cascaded convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 764–772. Springer, Cham (2017). Scholar
  5. 5.
    Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chen, H., Dou, Q., Yu, L., Qin, J., Heng, P.A.: VoxResNet: deep voxelwise residual networks for brain segmentation from 3D MR images. NeuroImage 170, 446–455 (2017)CrossRefGoogle Scholar
  7. 7.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). Scholar
  8. 8.
    Dong, H., Yang, G., Liu, F., Mo, Y., Guo, Y.: Automatic brain tumor detection and segmentation using U-net based fully convolutional networks. In: Valdés Hernández, M., González-Castro, V. (eds.) MIUA 2017. CCIS, vol. 723, pp. 506–517. Springer, Cham (2017). Scholar
  9. 9.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Fessler, J.A.: Medical image reconstruction: a brief overview of past milestones and future directions. arXiv preprint arXiv:1707.05927 (2017)
  11. 11.
    Hricak, H., Amparo, E.: Body MRI: alleviation of claustrophobia by prone positioning. Radiology 152(3), 819–819 (1984)CrossRefGoogle Scholar
  12. 12.
    Huang, Y., Paisley, J., Lin, Q., Ding, X., Fu, X., Zhang, X.P.: Bayesian nonparametric dictionary learning for compressed sensing MRI. IEEE Trans. Image Process. 23(12), 5007–5019 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Lai, Z., et al.: Image reconstruction of compressed sensing MRI using graph-based redundant wavelet transform. Med. Image Anal. 27, 93–104 (2016)CrossRefGoogle Scholar
  14. 14.
    Lee, D., Yoo, J., Ye, J.C.: Deep residual learning for compressed sensing MRI. In: ISBI, pp. 15–18. IEEE (2017)Google Scholar
  15. 15.
    Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: AOD-Net: all-in-one dehazing network. In: ICCV, October 2017Google Scholar
  16. 16.
    Liu, D., Wen, B., Liu, X., Wang, Z., Huang, T.S.: When image denoising meets high-level vision tasks: a deep learning approach. In: IJCAI (2018)Google Scholar
  17. 17.
    Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007)CrossRefGoogle Scholar
  18. 18.
    Mendrik, A.M., et al.: MRBrainS challenge: online evaluation framework for brain image segmentation in 3T MRI scans. Comput. Intell. Neurosci. 2015, 1 (2015)CrossRefGoogle Scholar
  19. 19.
    Moeskops, P., Viergever, M.A., Mendrik, A.M., de Vries, L.S., Benders, M.J., Išgum, I.: Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1252–1261 (2016)CrossRefGoogle Scholar
  20. 20.
    Nie, D., Wang, L., Gao, Y., Sken, D.: Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: ISBI, pp. 1342–1345. IEEE (2016)Google Scholar
  21. 21.
    Qu, X., Hou, Y., Lam, F., Guo, D., Zhong, J., Chen, Z.: Magnetic resonance image reconstruction from undersampled measurements using a patch-based nonlocal operator. Med. Image Anal. 18(6), 843–856 (2014)CrossRefGoogle Scholar
  22. 22.
    Ravishankar, S., Bresler, Y.: MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imaging 30(5), 1028–1041 (2011)CrossRefGoogle Scholar
  23. 23.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  24. 24.
    Schlemper, J., Caballero, J., Hajnal, J.V., Price, A., Rueckert, D.: A deep cascade of convolutional neural networks for MR image reconstruction. In: Niethammer, M., Styner, M., Aylward, S., Zhu, H., Oguz, I., Yap, P.-T., Shen, D. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 647–658. Springer, Cham (2017). Scholar
  25. 25.
    Steenwijk, M.D., et al.: Accurate white matter lesion segmentation by k nearest neighbor classification with tissue type priors (kNN-TTPs). NeuroImage Clin. 3, 462–469 (2013)CrossRefGoogle Scholar
  26. 26.
    Stollenga, M.F., Byeon, W., Liwicki, M., Schmidhuber, J.: Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. In: NIPS, pp. 2998–3006 (2015)Google Scholar
  27. 27.
    Wang, L., et al.: Links: learning-based multi-source integration framework for segmentation of infant brain images. NeuroImage 108, 160–172 (2015)CrossRefGoogle Scholar
  28. 28.
    Wang, S., et al.: Accelerating magnetic resonance imaging via deep learning. In: ISBI, pp. 514–517. IEEE (2016)Google Scholar
  29. 29.
    Zhang, H., Yang, J., Zhang, Y., Nasrabadi, N.M., Huang, T.S.: Close the loop: joint blind image restoration and recognition with sparse representation prior. In: ICCV, pp. 770–777. IEEE (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Liyan Sun
    • 1
  • Zhiwen Fan
    • 1
  • Xinghao Ding
    • 1
    Email author
  • Yue Huang
    • 1
  • John Paisley
    • 2
  1. 1.Fujian Key Laboratory of Sensing and Computing for Smart CityXiamen UniversityFujianChina
  2. 2.Department of Electrical EngineeringColumbia UniversityNew YorkUSA

Personalised recommendations