Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients
- 238 Downloads
Deep convolutional neural networks in their various forms are currently achieving or outperforming state-of-the-art results on several medical imaging tasks. We aim to make these developments available to the so far unsolved task of accurate correspondence finding—especially with regard to image registration.
We propose a two-step hybrid approach to make deep learned features accessible to a discrete optimization-based registration method. In a first step, in order to extract expressive binary local descriptors, we train a deep network architecture on a patch-based landmark retrieval problem as auxiliary task. As second step at runtime within a MRF-regularised dense displacement sampling, their binary nature enables highly efficient similarity computations, thus making them an ideal candidate to replace the so far used handcrafted local feature descriptors during the registration process.
We evaluate our approach on finding correspondences between highly non-rigidly deformed lung CT scans from different breathing states. Although the CNN-based descriptors excell at an auxiliary learning task for finding keypoint correspondences, self-similarity-based descriptors yield more accurate registration results. However, a combination of both approaches turns out to generate the most robust features for registration.
We present a three-dimensional framework for large lung motion estimation based on the combination of CNN-based and handcrafted descriptors efficiently employed in a discrete registration method. Achieving best results by combining learned and handcrafted features encourages further research in this direction.
KeywordsDeep learning Image registration Discrete optimization Hamming distance
Compliance with ethical standards
Conflict of interest
The authors declare that they have no relevant conflict of interest.
This article does not contain any studies with human participants or animals performed by any of the authors.
Statement of informed consent was not applicable since the manuscript does not contain any participants’ data.
- 2.Blendowski M, Heinrich MP (2018) 3D-CNNs for deep binary descriptor learning in medical volume data. In: Maier A, Deserno T, Handels H, Maier-Hein K, Palm C, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, Heidelberg, pp 23–28. https://doi.org/10.1007/978-3-662-56537-7_19
- 3.Brox T, Bregler C, Malik J (2009) Largedisplacement optical flow. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 41–48Google Scholar
- 4.Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: binary robust independent elementary features. In: European conference on computer vision. Springer, pp 778–792Google Scholar
- 6.Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d U-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 424–432Google Scholar
- 7.Conjeti S, Roy AG, Katouzian A, Navab N (2017) Hashing with residual networks for image retrieval. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 541–549Google Scholar
- 8.de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso M et al (eds) Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 204–212. https://doi.org/10.1007/978-3-319-67558-9_24
- 9.Dosovitskiy A, Fischer P, IlgE, Hausser P, Hazirbas C, Golkov V, van der Smagt P,Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766Google Scholar
- 11.Eilertsen G, Forssén PE, Unger J (2017) Briefmatch: dense binary feature matching for real-time optical flow estimation. In: Scandinavian conference on image analysis. Springer, pp 221–233Google Scholar
- 14.He K, Zhang X, Ren S, Sun J (2016) Deepresidual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
- 15.Heinrich MP, Blendowski M,Oktay O (2018) Ternarynet: faster deep model inference without GPUS formedical 3D segmentation using sparse and binary convolutions. arXivpreprint https://arxiv.org/abs/801.09449
- 16.Heinrich MP, Handels H, Simpson IJ (2015) Estimating large lung motion in COPD patients by symmetric regularised correspondence fields. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 338–345Google Scholar
- 18.Heinrich MP, Jenkinson M, Papież BW, Brady M, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 187–194Google Scholar
- 19.Heinrich MP, Oktay O (2017) Briefnet: deep pancreas segmentation using binary sparse convolutions. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 329–337Google Scholar
- 21.Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Patternrecognition 1:3Google Scholar
- 22.Liu H, Wang R, Shan S, Chen X (2016) Deep supervised hashing for fast image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2064–2072Google Scholar
- 23.Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3D vision (3DV). IEEE, pp 565–571Google Scholar
- 26.Muła W, Kurz N, Lemire D (2017) Faster population counts using avx2 instructions. Comput J 61(1):111–120Google Scholar
- 28.Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-net: learning deformable image registration using shape matching. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 266–274Google Scholar
- 29.Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241Google Scholar
- 31.Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE International conference on computer vision (ICCV). IEEE, pp 1385–1392Google Scholar
- 32.Zhang Y, Ozay M, Li S, OkataniT (2017) Truncating wide networks using binary tree architectures. arXivpreprint https://arxiv.org/abs/1704.00509