Advertisement

Memory-efficient 2.5D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans

  • Alessa HeringEmail author
  • Sven Kuckertz
  • Stefan Heldmann
  • Mattias P. Heinrich
Original Article

Abstract

Purpose 

Despite its potential for improvements through supervision, deep learning-based registration approaches are difficult to train for large deformations in 3D scans due to excessive memory requirements.

Methods 

We propose a new 2.5D convolutional transformer architecture that enables us to learn a memory-efficient weakly supervised deep learning model for multi-modal image registration. Furthermore, we firstly integrate a volume change control term into the loss function of a deep learning-based registration method to penalize occurring foldings inside the deformation field.

Results 

Our approach succeeds at learning large deformations across multi-modal images. We evaluate our approach on 100 pair-wise registrations of CT and MRI whole-heart scans and demonstrate considerably higher Dice Scores (of 0.74) compared to a state-of-the-art unsupervised discrete registration framework (deeds with Dice of 0.71).

Conclusion 

Our proposed memory-efficient registration method performs better than state-of-the-art conventional registration methods. By using a volume change control term in the loss function, the number of occurring foldings can be considerably reduced on new registration cases.

Keywords

Multi-modal registration Convolutional neural networks Weakly supervised learning CT MRI 2.5D 

Notes

Funding

This work was funded in part by the German Research Foundation (DFG) under grant number 320997906.

Compliance with ethical standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

This article does not contain patient data.

References

  1. 1.
    Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2019) VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging.  https://doi.org/10.1109/TMI.2019.2897538
  2. 2.
    de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. Deep learning in medical image analysis and multi-modal learning for clinical decision support. Springer, Cham, pp 204–212CrossRefGoogle Scholar
  3. 3.
    de Vos BD, Berendse FF, Viergever MA, Sokooti H, Staring M, Išgum I (2019) A deep learning framework for unsupervised affine and deformable image registration. Med Image Anal 52:128–143CrossRefGoogle Scholar
  4. 4.
    Fischer B, Modersitzki J (2003) Curvature based image registration. J Math Imaging Vis 18(1):81–85CrossRefGoogle Scholar
  5. 5.
    Haber E, Modersitzki J (2006) Intensity gradient based registration and fusion of multi-modal images. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 726–733Google Scholar
  6. 6.
    Heinrich MP, Jenkinson M, Brady M, Schnabel JA (2013) MRF-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med imaging 32(7):1239–1248CrossRefGoogle Scholar
  7. 7.
    Heinrich MP, Maier O, Handels H (2015) Multi-modal multi-atlas segmentation using discrete optimisation and self-similarities. In VISCERAL Challenge@ ISBI, pp. 27–30Google Scholar
  8. 8.
    Heinrich MP, Okay O, Bouteldja, N (2018) OBELISK-one kernel to solve nearly everything: unified 3D binary convolutions for image analysis. Med Imaging Deep Learn. https://openreview.net/pdf?id=BkZu9wooz
  9. 9.
    Heinrich MP (2018) Intra-operative ultrasound to MRI fusion with a public multi-modal discrete registration tool. MICCAI workshop on simulation, image processing, and ultrasound systems for assisted diagnosis and navigation. Springer, Cham, pp 159–164CrossRefGoogle Scholar
  10. 10.
    Hering A, Kuckertz S, Heldmann S, Heinrich MP (2019) Enhancing label-driven deep deformable image registration with local distance metrics for state-of-the-art cardiac motion tracking. Bildverarbeitung für die Medizin 2019. Springer, Berlin, HeidelbergGoogle Scholar
  11. 11.
    Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Ourselin S (2018) Weakly-supervised convolutional neural networks for multi-modal image registration. Med Image Anal 49:1–13CrossRefGoogle Scholar
  12. 12.
    Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp. 2017–2025Google Scholar
  13. 13.
    Krebs J, Mansi T, Mailhé B, Ayache N, Delingette H (2018) Unsupervised probabilistic deformation modeling for robust diffeomorphic registration. Deep learning in medical image analysis and multi-modal learning for clinical decision support. Springer, Cham, pp 101–109CrossRefGoogle Scholar
  14. 14.
    Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88CrossRefGoogle Scholar
  15. 15.
    Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-Net: learning deformable image registration using shape matching. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 266–274Google Scholar
  16. 16.
    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. International Conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241Google Scholar
  17. 17.
    Roth HR, Lu L, Farag A, Shin HC, Liu J, Turkbey EB, Summers RM (2015) Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 556–564Google Scholar
  18. 18.
    Rühaak J, Heldmann S, Kipshagen T, Fischer B (2013) Highly accurate fast lung CT registration. In Medical imaging 2013: image processing, vol. 8669, p. 86690Y. International Society for Optics and PhotonicsGoogle Scholar
  19. 19.
    Rühaak J, Derksen A, Heldmann S, Hallmann M, Meine H (2015) Accurate CT-MR image registration for deep brain stimulation: a multi-observer evaluation study. In Medical imaging 2015: image processing, vol. 9413, p. 941337. International Society for Optics and PhotonicsGoogle Scholar
  20. 20.
    Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 246–253Google Scholar
  21. 21.
    Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32(7):1153–1190CrossRefGoogle Scholar
  22. 22.
    Xia Y, Xie L, Liu F, Zhu Z, Fishman EK, Yuille AL (2018) Bridging the gap between 2D and 3D organ segmentation. arXivpreprint arXiv:1804.00392
  23. 23.
    Xu Z, Lee CP, Heinrich MP, Modat M, Rueckert D, Ourselin S, Landman B (2016) Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Trans Biomed Eng 63(8):1563–1572CrossRefGoogle Scholar
  24. 24.
    Zhuang X, Shen J (2016) Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal 31:77–87CrossRefGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.Fraunhofer Institute for Digital Medicine MEVISLübeckGermany
  2. 2.Diagnostic Image Analysis GroupRadboudumcNijmegenNetherlands
  3. 3.Institute of Medical InformaticsUniversity of LübeckLübeckGermany

Personalised recommendations