Multi-modal Segmentation with Missing MR Sequences Using Pre-trained Fusion Networks

  • Karin van GarderenEmail author
  • Marion Smits
  • Stefan Klein
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11795)


Missing data is a common problem in machine learning and in retrospective imaging research it is often encountered in the form of missing imaging modalities. We propose to take into account missing modalities in the design and training of neural networks, to ensure that they are capable of providing the best possible prediction even when multiple images are not available. The proposed network combines three modifications to the standard 3D UNet architecture: a training scheme with dropout of modalities, a multi-pathway architecture with fusion layer in the final stage, and the separate pre-training of these pathways. These modifications are evaluated incrementally in terms of performance on full and missing data, using the BraTS multi-modal segmentation challenge. The final model shows significant improvement with respect to the state of the art on missing data and requires less memory during training.


Convolutional neural network Glioma segmentation Missing data 



This work was supported by the Dutch Cancer Society (project number 11026, GLASS-NL), the Dutch Organization for Scientific Research (NWO) and NVIDIA Corporation (by donating a GPU).


  1. 1.
    Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4, 170117 (2017)CrossRefGoogle Scholar
  2. 2.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). Scholar
  3. 3.
    Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: HeMIS: hetero-modal image segmentation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 469–477. Springer, Cham (2016). Scholar
  4. 4.
    Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H.: No New-Net. ArXiv e-prints, September 2018Google Scholar
  5. 5.
    Jerez, J., et al.: Missing data imputation using statistical and machine learning methods in a real breast cancer problem. Artif. Intell. Med. 50(2), 105–115 (2010)CrossRefGoogle Scholar
  6. 6.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2014)Google Scholar
  7. 7.
    Maaten, L.V.D., Hinton, G., Visualizing data using t-SNE: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  8. 8.
    Menze, B.H., Jakab, A., Bauer, S., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2015)CrossRefGoogle Scholar
  9. 9.
    van Tulder, G., de Bruijne, M.: Representation learning for cross-modality classification. In: Muller, H., et al. (eds.) Medical Computer Vision and Bayesian and Graphical Models for Biomedical Imaging, pp. 126–136. Springer, Cham (2017). Scholar
  10. 10.
    Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Cham (2015). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Karin van Garderen
    • 1
    • 3
    Email author
  • Marion Smits
    • 1
  • Stefan Klein
    • 1
    • 2
  1. 1. Department of Radiology and Nuclear MedicineErasmus MCRotterdamThe Netherlands
  2. 2.Department of Medical InformaticsErasmus MCRotterdamThe Netherlands
  3. 3.Medical DeltaDelftThe Netherlands

Personalised recommendations