Deep Multi-Modal Encoder-Decoder Networks for Shape Constrained Segmentation and Joint Representation Learning

  • Nassim BouteldjaEmail author
  • Dorit Merhof
  • Jan Ehrhardt
  • Mattias P. Heinrich
Conference paper
Part of the Informatik aktuell book series (INFORMAT)


Deep learning approaches have been very successful in segmenting cardiac structures from CT and MR volumes. Despite continuous progress, automated segmentation of these structures remains challenging due to highly complex regional characteristics (e.g. homogeneous gray-level transitions) and large anatomical shape variability. To cope with these challenges, the incorporation of shape priors into neural networks for robust segmentation is an important area of current research. We propose a novel approach that leverages shared information across imaging modalities and shape segmentations within a unified multi-modal encoder-decoder network. This jointly end-to-end trainable architecture is advantageous in improving robustness due to strong shape constraints and enables further applications due to smooth transitions in the learned shape space. Despite no skip connections are used and all shape information is encoded in a low-dimensional representation, our approach achieves high-accuracy segmentation and consistent shape interpolation results on the multi-modal whole heart segmentation dataset.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tobon-Gomez C, et al. Benchmark for algorithms segmenting the left atrium from 3D CT and MRI datasets. IEEE Trans Med Imag. 2015;34(7):1460–1473.CrossRefGoogle Scholar
  2. 2.
    Milletari F, et al.; IEEE. V-net: fully convolutional neural networks for volumetric medical image segmentation. Proc 3D Vis. 2016; p. 565–571.Google Scholar
  3. 3.
    Ronneberger O, et al.; Springer. U-net: convolutional networks for biomedical image segmentation. Proc MICCAI. 2015; p. 234–241.Google Scholar
  4. 4.
    Oktay O, et al. Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. CoRR. 2017;abs/1705.08302.Google Scholar
  5. 5.
    Ravishankar H, et al. Learning and incorporating shape models for semantic segmentation. Proc MICCAI. 2017; p. 203–211.Google Scholar
  6. 6.
    Jetley S, Sapienza M, Golodetz S, et al. Straight to shapes: real-time detection of encoded shapes. CoRR. 2016;abs/1611.07932.Google Scholar
  7. 7.
    Bouteldja N, Heinrich MP. Deep 3D encoder-decoder networks with applications to organ segmentation. Stud Conf. 2018; p. 229–232.Google Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  • Nassim Bouteldja
    • 1
    Email author
  • Dorit Merhof
    • 1
  • Jan Ehrhardt
    • 2
  • Mattias P. Heinrich
    • 2
  1. 1.Institute of Imaging and Computer VisionRWTH Aachen UniversityAachenDeutschland
  2. 2.Institute of Medical InformaticsUniversity of LuebeckLübeckDeutschland

Personalised recommendations