HeMIS: Hetero-Modal Image Segmentation

  • Mohammad Havaei
  • Nicolas Guizard
  • Nicolas Chapados
  • Yoshua Bengio
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9901)


We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.


Segmentation Multi-modal Deep learning Convolutional neural networks Data abstraction Data imputation 


  1. 1.
    Brosch, T., Yoo, Y., Tang, L.Y.W., Li, D.K.B., Traboulsee, A., Tam, R.: Deep convolutional encoder networks for multiple sclerosis lesion segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 3–11. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24574-4_1 CrossRefGoogle Scholar
  2. 2.
    Geremia, E., Menze, B.H., Ayache, N.: Spatially adaptive random forests, pp. 1344–1347 (2013)Google Scholar
  3. 3.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT Press, Cambridge (2016)Google Scholar
  4. 4.
    Guizard, N., Coupé, P., Fonov, V.S., Manjón, J.V., Arnold, D.L., Collins, D.L.: Rotation-invariant multi-contrast non-local means for ms lesion segmentation. NeuroImage Clin. 8, 376–389 (2015)CrossRefGoogle Scholar
  5. 5.
    Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural networks. arXiv preprint (2015). arXiv:1505.03540
  6. 6.
    Hofmann, M., Steinke, F., Scheel, V., Charpiat, G., et al.: MRI-based attenuation correction for PET/MRI: a novel approach combining pattern recognition and atlas registration. J. Nucl. Med. 49(11), 1875–1883 (2008)CrossRefGoogle Scholar
  7. 7.
    Hor, S., Moradi, M.: Scandent Tree: a random forest learning method for incomplete multimodal datasets. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 694–701. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_85 CrossRefGoogle Scholar
  8. 8.
    Menze, B., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J.E.A.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE TMI 34(10), 1993–2024 (2015)Google Scholar
  9. 9.
    Souplet, J., Lebrun, C., Ayache, N., Malandain, G.: An automatic segmentation of T2-FLAIR multiple sclerosis lesions, 7 2008Google Scholar
  10. 10.
    Styner, M., Lee, J., Chin, B., Chin, M., Commowick, O., Tran, H., Markovic-Plese, S., Jewells, V., Warfield, S.: 3D segmentation in the clinic: A grand challenge ii: Ms lesion segmentation. MIDAS 2008, 1–6 (2008)Google Scholar
  11. 11.
    Tulder, G., Bruijne, M.: Why does synthesized data improve multi-sequence classification? In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 531–538. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_65 CrossRefGoogle Scholar
  12. 12.
    Tustison, N.J., Shrinidhi, K., Wintermark, M., Durst, C.R., Kandel, B.M., Gee, J.C., Grossman, M.C., Avants, B.B.: Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with ANTsR. Neuroinformatics 13(2), 209–225 (2015)CrossRefGoogle Scholar
  13. 13.
    Van Buuren, S.: Flexible imputation of missing data. CRC Press, Boca Raton (2012)CrossRefzbMATHGoogle Scholar
  14. 14.
    Zhao, L., Wu, W., Corso, J.J.: Semi-automatic brain tumor segmentation by constrained MRFs using structural trajectories. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8151, pp. 567–575. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40760-4_71 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Mohammad Havaei
    • 1
  • Nicolas Guizard
    • 1
  • Nicolas Chapados
    • 1
  • Yoshua Bengio
    • 2
  1. 1.Imagia Inc.MontrealCanada
  2. 2.Université de MontréalMontréalCanada

Personalised recommendations