Memory-efficient 2.5D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans
Despite its potential for improvements through supervision, deep learning-based registration approaches are difficult to train for large deformations in 3D scans due to excessive memory requirements.
We propose a new 2.5D convolutional transformer architecture that enables us to learn a memory-efficient weakly supervised deep learning model for multi-modal image registration. Furthermore, we firstly integrate a volume change control term into the loss function of a deep learning-based registration method to penalize occurring foldings inside the deformation field.
Our approach succeeds at learning large deformations across multi-modal images. We evaluate our approach on 100 pair-wise registrations of CT and MRI whole-heart scans and demonstrate considerably higher Dice Scores (of 0.74) compared to a state-of-the-art unsupervised discrete registration framework (deeds with Dice of 0.71).
Our proposed memory-efficient registration method performs better than state-of-the-art conventional registration methods. By using a volume change control term in the loss function, the number of occurring foldings can be considerably reduced on new registration cases.
KeywordsMulti-modal registration Convolutional neural networks Weakly supervised learning CT MRI 2.5D
This work was funded in part by the German Research Foundation (DFG) under grant number 320997906.
Compliance with ethical standards
Conflict of Interest
The authors declare that they have no conflict of interest.
This article does not contain any studies with human participants performed by any of the authors.
This article does not contain patient data.
- 2.de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. Deep learning in medical image analysis and multi-modal learning for clinical decision support. Springer, Cham, pp 204–212CrossRefGoogle Scholar
- 5.Haber E, Modersitzki J (2006) Intensity gradient based registration and fusion of multi-modal images. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 726–733Google Scholar
- 7.Heinrich MP, Maier O, Handels H (2015) Multi-modal multi-atlas segmentation using discrete optimisation and self-similarities. In VISCERAL Challenge@ ISBI, pp. 27–30Google Scholar
- 8.Heinrich MP, Okay O, Bouteldja, N (2018) OBELISK-one kernel to solve nearly everything: unified 3D binary convolutions for image analysis. Med Imaging Deep Learn. https://openreview.net/pdf?id=BkZu9wooz
- 10.Hering A, Kuckertz S, Heldmann S, Heinrich MP (2019) Enhancing label-driven deep deformable image registration with local distance metrics for state-of-the-art cardiac motion tracking. Bildverarbeitung für die Medizin 2019. Springer, Berlin, HeidelbergGoogle Scholar
- 12.Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp. 2017–2025Google Scholar
- 15.Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-Net: learning deformable image registration using shape matching. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 266–274Google Scholar
- 16.Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. International Conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241Google Scholar
- 17.Roth HR, Lu L, Farag A, Shin HC, Liu J, Turkbey EB, Summers RM (2015) Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 556–564Google Scholar
- 18.Rühaak J, Heldmann S, Kipshagen T, Fischer B (2013) Highly accurate fast lung CT registration. In Medical imaging 2013: image processing, vol. 8669, p. 86690Y. International Society for Optics and PhotonicsGoogle Scholar
- 19.Rühaak J, Derksen A, Heldmann S, Hallmann M, Meine H (2015) Accurate CT-MR image registration for deep brain stimulation: a multi-observer evaluation study. In Medical imaging 2015: image processing, vol. 9413, p. 941337. International Society for Optics and PhotonicsGoogle Scholar
- 20.Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 246–253Google Scholar
- 22.Xia Y, Xie L, Liu F, Zhu Z, Fishman EK, Yuille AL (2018) Bridging the gap between 2D and 3D organ segmentation. arXivpreprint arXiv:1804.00392