Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients

  • Max BlendowskiEmail author
  • Mattias P. Heinrich
Original Article



Deep convolutional neural networks in their various forms are currently achieving or outperforming state-of-the-art results on several medical imaging tasks. We aim to make these developments available to the so far unsolved task of accurate correspondence finding—especially with regard to image registration.


We propose a two-step hybrid approach to make deep learned features accessible to a discrete optimization-based registration method. In a first step, in order to extract expressive binary local descriptors, we train a deep network architecture on a patch-based landmark retrieval problem as auxiliary task. As second step at runtime within a MRF-regularised dense displacement sampling, their binary nature enables highly efficient similarity computations, thus making them an ideal candidate to replace the so far used handcrafted local feature descriptors during the registration process.


We evaluate our approach on finding correspondences between highly non-rigidly deformed lung CT scans from different breathing states. Although the CNN-based descriptors excell at an auxiliary learning task for finding keypoint correspondences, self-similarity-based descriptors yield more accurate registration results. However, a combination of both approaches turns out to generate the most robust features for registration.


We present a three-dimensional framework for large lung motion estimation based on the combination of CNN-based and handcrafted descriptors efficiently employed in a discrete registration method. Achieving best results by combining learned and handcrafted features encourages further research in this direction.


Deep learning Image registration Discrete optimization Hamming distance 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no relevant conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Statement of informed consent was not applicable since the manuscript does not contain any participants’ data.


  1. 1.
    Avants BB, Epstein CL, Grossman M, Gee JC (2008) Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal 12(1):26–41CrossRefGoogle Scholar
  2. 2.
    Blendowski M, Heinrich MP (2018) 3D-CNNs for deep binary descriptor learning in medical volume data. In: Maier A, Deserno T, Handels H, Maier-Hein K, Palm C, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, Heidelberg, pp 23–28.
  3. 3.
    Brox T, Bregler C, Malik J (2009) Largedisplacement optical flow. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 41–48Google Scholar
  4. 4.
    Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: binary robust independent elementary features. In: European conference on computer vision. Springer, pp 778–792Google Scholar
  5. 5.
    Castillo R, Castillo E, Guerra R, Johnson VE, McPhail T, Garg AK, Guerrero T (2009) A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Phys Med Biol 54(7):1849CrossRefGoogle Scholar
  6. 6.
    Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d U-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 424–432Google Scholar
  7. 7.
    Conjeti S, Roy AG, Katouzian A, Navab N (2017) Hashing with residual networks for image retrieval. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 541–549Google Scholar
  8. 8.
    de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso M et al (eds) Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 204–212.
  9. 9.
    Dosovitskiy A, Fischer P, IlgE, Hausser P, Hazirbas C, Golkov V, van der Smagt P,Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766Google Scholar
  10. 10.
    Dou Q, Chen H, Yu L, Qin J, Heng PA (2017) Multilevel contextual 3-D CNNS for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng 64(7):1558–1567CrossRefGoogle Scholar
  11. 11.
    Eilertsen G, Forssén PE, Unger J (2017) Briefmatch: dense binary feature matching for real-time optical flow estimation. In: Scandinavian conference on image analysis. Springer, pp 221–233Google Scholar
  12. 12.
    Felzenszwalb PF, Huttenlocher DP (2005) Pictorial structures for object recognition. Int J Comput Vis 61(1):55–79CrossRefGoogle Scholar
  13. 13.
    Glocker B, Komodakis N, Tziritas G, Navab N, Paragios N (2008) Dense image registration through MRFS and efficient linear programming. Med Image Anal 12(6):731–741CrossRefGoogle Scholar
  14. 14.
    He K, Zhang X, Ren S, Sun J (2016) Deepresidual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  15. 15.
    Heinrich MP, Blendowski M,Oktay O (2018) Ternarynet: faster deep model inference without GPUS formedical 3D segmentation using sparse and binary convolutions. arXivpreprint
  16. 16.
    Heinrich MP, Handels H, Simpson IJ (2015) Estimating large lung motion in COPD patients by symmetric regularised correspondence fields. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 338–345Google Scholar
  17. 17.
    Heinrich MP, Jenkinson M, Brady M, Schnabel JA (2013) Mrf-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med Imaging 32(7):1239–1248CrossRefGoogle Scholar
  18. 18.
    Heinrich MP, Jenkinson M, Papież BW, Brady M, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 187–194Google Scholar
  19. 19.
    Heinrich MP, Oktay O (2017) Briefnet: deep pancreas segmentation using binary sparse convolutions. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 329–337Google Scholar
  20. 20.
    Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Wang G, Bandula S, Moore CM, Emberton M et al (2018) Weakly-supervised convolutional neural networks for multimodal image registration. Med Image Anal 49:1–13CrossRefGoogle Scholar
  21. 21.
    Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Patternrecognition 1:3Google Scholar
  22. 22.
    Liu H, Wang R, Shan S, Chen X (2016) Deep supervised hashing for fast image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2064–2072Google Scholar
  23. 23.
    Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3D vision (3DV). IEEE, pp 565–571Google Scholar
  24. 24.
    Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Prog Biomed 98(3):278–284CrossRefGoogle Scholar
  25. 25.
    Muenzing SE, van Ginneken B, Viergever MA, Pluim JP (2014) Dirboost-an algorithm for boosting deformable image registration: application to lung CT intra-subject registration. Med Image Anal 18(3):449–459CrossRefGoogle Scholar
  26. 26.
    Muła W, Kurz N, Lemire D (2017) Faster population counts using avx2 instructions. Comput J 61(1):111–120Google Scholar
  27. 27.
    Reinhardt JM, Ding K, Cao K, Christensen GE, Hoffman EA, Bodas SV (2008) Registration-based estimates of local lung tissue expansion compared to xenon CT measures of specific ventilation. Med Image Anal 12(6):752–763CrossRefGoogle Scholar
  28. 28.
    Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-net: learning deformable image registration using shape matching. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 266–274Google Scholar
  29. 29.
    Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241Google Scholar
  30. 30.
    Rühaak J, Polzin T, Heldmann S, Simpson IJ, Handels H, Modersitzki J, Heinrich MP (2017) Estimation of large motion in lung CT by integrating regularized keypoint correspondences into dense deformable registration. IEEE Trans Med Imaging 36(8):1746–1757CrossRefGoogle Scholar
  31. 31.
    Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE International conference on computer vision (ICCV). IEEE, pp 1385–1392Google Scholar
  32. 32.
    Zhang Y, Ozay M, Li S, OkataniT (2017) Truncating wide networks using binary tree architectures. arXivpreprint

Copyright information

© CARS 2018

Authors and Affiliations

  1. 1.Institute of Medical InformaticsUniversity of LübeckLübeckGermany

Personalised recommendations