A Hybrid Deep Learning Framework for Integrated Segmentation and Registration: Evaluation on Longitudinal White Matter Tract Changes

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11766)


To accurately analyze changes of anatomical structures in longitudinal imaging studies, consistent segmentation across multiple time-points is required. Existing solutions often involve independent registration and segmentation components. Registration between time-points is used either as a prior for segmentation in a subsequent time point or to perform segmentation in a common space. In this work, we propose a novel hybrid convolutional neural network (CNN) that integrates segmentation and registration into a single procedure. We hypothesize that the joint optimization leads to increased performance on both tasks. The hybrid CNN is trained by minimizing an integrated loss function composed of four different terms, measuring segmentation accuracy, similarity between registered images, deformation field smoothness, and segmentation consistency. We applied this method to the segmentation of white matter tracts, describing functionally grouped axonal fibers, using N = 8045 longitudinal brain MRI data of 3249 individuals. The proposed method was compared with two multistage pipelines using two existing segmentation methods combined with a conventional deformable registration algorithm. In addition, we assessed the added value of the joint optimization for segmentation and registration separately. The hybrid CNN yielded significantly higher accuracy, consistency and reproducibility of segmentation than the multistage pipelines, and was orders of magnitude faster. Therefore, we expect it can serve as a novel tool to support clinical and epidemiological analyses on understanding microstructural brain changes over time.


Simultaneous Segmentation Deformable registration Diffusion MRI White matter tract CNN Longitudinal 


  1. 1.
    Balakrishnan, G., et al.: Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imaging 38(8), 1788–1800 (2019)CrossRefGoogle Scholar
  2. 2.
    de Groot, M., et al.: Tract-specific white matter degeneration in aging: the Rotterdam Study. Alzheimer’s Dement. 11(3), 321–330 (2015)CrossRefGoogle Scholar
  3. 3.
    Hofman, A., et al.: The Rotterdam Study: 2016 objectives and design update. Eur. J. Epidemiol. 30(8), 661–708 (2015)CrossRefGoogle Scholar
  4. 4.
    Hu, Y., et al.: Label-driven weakly-supervised learning for multimodal deformable image registration. In: 15th ISBI, pp. 1070–1074. IEEE (2018)Google Scholar
  5. 5.
    Klein, S., et al.: Elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imag. 29(1), 196–205 (2010)CrossRefGoogle Scholar
  6. 6.
    Li, B., de Groot, M., Vernooij, M.W., Ikram, M.A., Niessen, W.J., Bron, E.E.: Reproducible white matter tract segmentation using 3D U-Net on a large-scale DTI dataset. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 205–213. Springer, Cham (2018). Scholar
  7. 7.
    Parisot, S., et al.: Concurrent tumor segmentation and registration with uncertainty-based sparse non-uniform graphs. Med. Image Anal. 18(4), 647–659 (2014)CrossRefGoogle Scholar
  8. 8.
    Pohl, K.M., et al.: An expectation maximization approach for integrated registration, segmentation, and intensity correction (2005)Google Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  10. 10.
    Vlontzos, A., Mikolajczyk, K.: Deep segmentation and registration in x-ray angiography video. arXiv preprint arXiv:1805.06406 (2018)
  11. 11.
    Yendiki, A., et al.: Joint reconstruction of white-matter pathways from longitudinal diffusion MRI data with anatomical priors. Neuroimage 127, 277–286 (2016)CrossRefGoogle Scholar
  12. 12.
    Yezzi, A., et al.: A variational framework for integrating segmentation and registration through active contours. Med. Image Anal. 7(2), 171–185 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Northeastern UniversityShenyangChina
  2. 2.Erasmus MCRotterdamThe Netherlands
  3. 3.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations