One Network to Segment Them All: A General, Lightweight System for Accurate 3D Medical Image Segmentation

  • Mathias PerslevEmail author
  • Erik Bjørnager Dam
  • Akshay Pai
  • Christian Igel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11765)


Many recent medical segmentation systems rely on powerful deep learning models to solve highly specific tasks. To maximize performance, it is standard practice to evaluate numerous pipelines with varying model topologies, optimization parameters, pre- & postprocessing steps, and even model cascades. It is often not clear how the resulting pipeline transfers to different tasks.

We propose a simple and thoroughly evaluated deep learning framework for segmentation of arbitrary medical image volumes. The system requires no task-specific information, no human interaction and is based on a fixed model topology and a fixed hyperparameter set, eliminating the process of model selection and its inherent tendency to cause method-level over-fitting. The system is available in open source and does not require deep learning expertise to use. Without task-specific modifications, the system performed better than or similar to highly specialized deep learning methods across 3 separate segmentation tasks. In addition, it ranked 5-th and 6-th in the first and second round of the 2018 Medical Segmentation Decathlon comprising another 10 tasks.

The system relies on multi-planar data augmentation which facilitates the application of a single 2D architecture based on the familiar U-Net. Multi-planar training combines the parameter efficiency of a 2D fully convolutional neural network with a systematic train- and test-time augmentation scheme, which allows the 2D model to learn a representation of the 3D image volume that fosters generalization.



We would like to thank both Microsoft and NVIDIA for providing computational resources on the Azure platform for this project.

Supplementary material

490275_1_En_4_MOESM1_ESM.pdf (493 kb)
Supplementary material 1 (pdf 493 KB)
490275_1_En_4_MOESM2_ESM.mp4 (8 mb)
Supplementary material 2 (mp4 8199 KB)


  1. 1.
    Simpson, A.L., et al.: A large annotated medical image dataset for the development and evaluation of segmentation algorithms. CoRR abs/1902.09063 (2019)Google Scholar
  2. 2.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  3. 3.
    Louring Koch, T., Perslev, M., Igel, C., Brand, S.S.: Accurate segmentation of dental panoramic radiographs with U-Nets. In: International Symposium on Biomedical Imaging (ISBI). IEEE (2019)Google Scholar
  4. 4.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (ICML), pp. 448–456. PMLR (2015)Google Scholar
  5. 5.
    Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016)Google Scholar
  6. 6.
    Simard, P.Y., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: International Conference on Document Analysis and Recognition (ICDAR). IEEE (2003)Google Scholar
  7. 7.
    Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C., Buckner, R.L.: Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J. Cogn. Neurosci. 19(9), 1498–1507 (2007)CrossRefGoogle Scholar
  8. 8.
    Boccardi, M., et al.: Training labels for hippocampal segmentation based on the EADC-ADNI harmonized hippocampal protocol. Alzheimer’s Dementia 11(2), 175–183 (2015)CrossRefGoogle Scholar
  9. 9.
    Dam, E., Lillholm, M., Marques, J., Nielsen, M.: Automatic segmentation of high-and low-field knee MRIs using knee image quantification with data from the osteoarthritis initiative. J. Med. Imaging 2(2), 024001 (2015)CrossRefGoogle Scholar
  10. 10.
    Moeskops, P., Viergever, M.A., Mendrik, A.M., de Vries, L.S., Benders, M.J.N.L., Isgum, I.: Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1252–1261 (2016)CrossRefGoogle Scholar
  11. 11.
    Ganaye, P., Sdika, M., Benoit-Cattin, H.: Towards integrating spatial localization in convolutional neural networks for brain image segmentation. In: International Symposium on Biomedical Imaging (ISBI), pp. 621–625. IEEE (2018)Google Scholar
  12. 12.
    Roy, A.G., Conjeti, S., Navab, N., Wachinger, C.: QuickNAT: segmenting MRI neuroanatomy in 20 seconds. CoRR arXiv:1801.04161 (2018)
  13. 13.
    Ambellan, F., Tack, A., Ehlke, M., Zachow, S.: Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the osteoarthritis initiative. Med. Image Anal. 52, 109–118 (2019). OAI-ZIB dataset (supplementary material)CrossRefGoogle Scholar
  14. 14.
    Isensee, F., et al.: nnU-Net: self-adapting framework for U-Net-based medical image segmentation. CoRR arXiv:1809.10486 (2018)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mathias Perslev
    • 1
    Email author
  • Erik Bjørnager Dam
    • 1
    • 2
  • Akshay Pai
    • 1
    • 2
  • Christian Igel
    • 1
  1. 1.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark
  2. 2.Cerebriu A/SCopenhagenDenmark

Personalised recommendations