HeMIS: Hetero-Modal Image Segmentation

  • Mohammad Havaei
  • Nicolas Guizard
  • Nicolas Chapados
  • Yoshua Bengio
Conference paper

DOI: 10.1007/978-3-319-46723-8_54

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9901)
Cite this paper as:
Havaei M., Guizard N., Chapados N., Bengio Y. (2016) HeMIS: Hetero-Modal Image Segmentation. In: Ourselin S., Joskowicz L., Sabuncu M., Unal G., Wells W. (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016. MICCAI 2016. Lecture Notes in Computer Science, vol 9901. Springer, Cham

Abstract

We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.

Keywords

Segmentation Multi-modal Deep learning Convolutional neural networks Data abstraction Data imputation 

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Mohammad Havaei
    • 1
  • Nicolas Guizard
    • 1
  • Nicolas Chapados
    • 1
  • Yoshua Bengio
    • 2
  1. 1.Imagia Inc.MontrealCanada
  2. 2.Université de MontréalMontréalCanada

Personalised recommendations