Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks

  • Hessam SokootiEmail author
  • Bob de Vos
  • Floris Berendsen
  • Boudewijn P. F. Lelieveldt
  • Ivana Išgum
  • Marius Staring
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10433)


In this paper we propose a method to solve nonrigid image registration through a learning approach, instead of via iterative optimization of a predefined dissimilarity metric. We design a Convolutional Neural Network (CNN) architecture that, in contrast to all other work, directly estimates the displacement vector field (DVF) from a pair of input images. The proposed RegNet is trained using a large set of artificially generated DVFs, does not explicitly define a dissimilarity metric, and integrates image content at multiple scales to equip the network with contextual information. At testing time nonrigid registration is performed in a single shot, in contrast to current iterative methods. We tested RegNet on 3D chest CT follow-up data. The results show that the accuracy of RegNet is on par with a conventional B-spline registration, for anatomy within the capture range. Training RegNet with artificially generated DVFs is therefore a promising approach for obtaining good results on real clinical data, thereby greatly simplifying the training problem. Deformable image registration can therefore be successfully casted as a learning problem.


Image registration Convolutional neural networks Multi-scale analysis Chest CT 



This work is financed by the Netherlands Organization for Scientific Research (NWO), project 13351. Dr. M.E. Bakker and J. Stolk are acknowledged for providing a ground truth for the SPREAD study data used in this paper. The Tesla K40 used for this research was donated by the NVIDIA Corporation.


  1. 1.
    Hu, S., Wei, L., Gao, Y., Guo, Y., Wu, G., Shen, D.: Learning-based deformable image registration for infant MR images in the first year of life. Med. Phys. 44(1), 158–170 (2017)CrossRefGoogle Scholar
  2. 2.
    Miao, S., Wang, Z.J., Liao, R.: A CNN regression approach for real-time 2D/3D registration. IEEE Trans. Med. Imaging 35(5), 1352–1363 (2016)CrossRefGoogle Scholar
  3. 3.
    Liao, R., Miao, S., de Tournemire, P., Grbic, S., Kamen, A., Mansi, T., Comaniciu, D.: An artificial agent for robust image registration. arXiv preprint arXiv:1611.10336 (2016)
  4. 4.
    Yang, X., Kwitt, R., Niethammer, M.: Fast predictive image registration. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 48–57. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_6CrossRefGoogle Scholar
  5. 5.
    Eppenhof, K.A., Pluim, J.P.: Supervised local error estimation for nonlinear image registration using convolutional neural networks. In: SPIE Medical Imaging, p. 101331U. International Society for Optics and Photonics (2017)Google Scholar
  6. 6.
    Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: Flownet: Learning optical flow with convolutional networks. arXiv preprint. arXiv:1504.06852 (2015)
  7. 7.
    Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017)CrossRefGoogle Scholar
  8. 8.
    Stolk, J., Putter, H., Bakker, E.M., Shaker, S.B., Parr, D.G., Piitulainen, E., Russi, E.W., Grebski, E., Dirksen, A., Stockley, R.A., Reiber, J.H.C., Stoel, B.C.: Progression parameters for emphysema: a clinical investigation. Respir. Med. 101(9), 1924–1930 (2007)CrossRefGoogle Scholar
  9. 9.
    Murphy, K., van Ginneken, B., Klein, S., Staring, M., de Hoop, B.J., Viergever, M.A., Pluim, J.P.: Semi-automatic construction of reference standards for evaluation of image registration. Med. Image Anal. 15(1), 71–84 (2011)CrossRefGoogle Scholar
  10. 10.
    Theano Development Team: Theano: a Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688, May 2016Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Hessam Sokooti
    • 1
    Email author
  • Bob de Vos
    • 2
  • Floris Berendsen
    • 1
  • Boudewijn P. F. Lelieveldt
    • 1
    • 3
  • Ivana Išgum
    • 2
  • Marius Staring
    • 1
    • 3
  1. 1.Leiden University Medical CenterLeidenThe Netherlands
  2. 2.Image Sciences InstituteUniversity Medical Center UtrechtUtrechtThe Netherlands
  3. 3.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations