Deep Learning for Multi-task Medical Image Segmentation in Multiple Modalities

  • Pim MoeskopsEmail author
  • Jelmer M. Wolterink
  • Bas H. M. van der Velden
  • Kenneth G. A. Gilhuijs
  • Tim Leiner
  • Max A. Viergever
  • Ivana Išgum
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9901)


Automatic segmentation of medical images is an important task for many clinical applications. In practice, a wide range of anatomical structures are visualised using different imaging modalities. In this paper, we investigate whether a single convolutional neural network (CNN) can be trained to perform different segmentation tasks.

A single CNN is trained to segment six tissues in MR brain images, the pectoral muscle in MR breast images, and the coronary arteries in cardiac CTA. The CNN therefore learns to identify the imaging modality, the visualised anatomical structures, and the tissue classes.

For each of the three tasks (brain MRI, breast MRI and cardiac CTA), this combined training procedure resulted in a segmentation performance equivalent to that of a CNN trained specifically for that task, demonstrating the high capacity of CNN architectures. Hence, a single system could be used in clinical practice to automatically perform diverse segmentation tasks without task-specific training.


Deep learning Convolutional neural networks Medical image segmentation Brain MRI Breast MRI Cardiac CTA 


  1. 1.
    de Brébisson, A., Montana, G.: Deep neural networks for anatomical brain segmentation. In: CVPR Bioimage Computing Workshop (2015)Google Scholar
  2. 2.
    Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). In: ICLR (2016)Google Scholar
  3. 3.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 142–158 (2016)CrossRefGoogle Scholar
  4. 4.
    Gubern-Mérida, A., Kallenberg, M., Martí, R., Karssemeijer, N.: Segmentation of the pectoral muscle in breast MRI using atlas-based approaches. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7511, pp. 371–378. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33418-4_46 CrossRefGoogle Scholar
  5. 5.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  6. 6.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  7. 7.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  8. 8.
    Landman, B.A., Ribbens, A., Lucas, B., Davatzikos, C., Avants, B., Ledig, C., Ma, D., Rueckert, D., Vandermeulen, D., Maes, F., et al.: MICCAI 2012 Workshop on Multi-atlas Labeling. CreateSpace Independent Publishing Platform (2012).
  9. 9.
    Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C., Buckner, R.L.: Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J. Cogn. Neurosci. 19(9), 1498–1507 (2007)CrossRefGoogle Scholar
  10. 10.
    Moeskops, P., Viergever, M.A., Mendrik, A.M., de Vries, L.S., Benders, M.J., Išgum, I.: Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1252–1261 (2016)CrossRefGoogle Scholar
  11. 11.
    Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., Nielsen, M.: Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 246–253. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40763-5_31 CrossRefGoogle Scholar
  12. 12.
    Roth, H.R., Lu, L., Farag, A., Shin, H.-C., Liu, J., Turkbey, E.B., Summers, R.M.: DeepOrgan: multi-level deep convolutional networks for automated pancreas segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 556–564. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_68 CrossRefGoogle Scholar
  13. 13.
    Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (2016)Google Scholar
  14. 14.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  15. 15.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  16. 16.
    van der Velden, B.H., Dmitriev, I., Loo, C.E., Pijnappel, R.M., Gilhuijs, K.G.: Association between parenchymal enhancement of the contralateral breast in dynamic contrast-enhanced MR imaging and outcome of patients with unilateral invasive breast cancer. Radiology 276(3), 675–685 (2015)CrossRefGoogle Scholar
  17. 17.
    de Vos, B., Wolterink, J., de Jong, P., Viergever, M., Išgum, I.: 2D image classification for 3D anatomy localization; employing deep convolutional neural networks. In: SPIE Medical Imaging, p. 97841Y (2016)Google Scholar
  18. 18.
    Wolterink, J.M., Leiner, T., Viergever, M.A., Išgum, I.: Automatic coronary calcium scoring in cardiac CT angiography using convolutional neural networks. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 589–596. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_72 CrossRefGoogle Scholar
  19. 19.
    Zheng, Y., Loziczonek, M., Georgescu, B., Zhou, S.K., Vega-Higuera, F., Comaniciu, D.: Machine learning based vesselness measurement for coronary artery segmentation in cardiac CT volumes. In: SPIE Medical Imaging, p. 79621K (2011)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Pim Moeskops
    • 1
    • 2
    Email author
  • Jelmer M. Wolterink
    • 1
  • Bas H. M. van der Velden
    • 1
  • Kenneth G. A. Gilhuijs
    • 1
  • Tim Leiner
    • 3
  • Max A. Viergever
    • 1
  • Ivana Išgum
    • 1
  1. 1.Image Sciences InstituteUniversity Medical Center UtrechtUtrechtThe Netherlands
  2. 2.Medical Image AnalysisEindhoven University of TechnologyEindhovenThe Netherlands
  3. 3.Department of RadiologyUniversity Medical Center UtrechtUtrechtThe Netherlands

Personalised recommendations