A Deep Metric for Multimodal Registration

  • Martin SimonovskyEmail author
  • Benjamín Gutiérrez-Becker
  • Diana Mateus
  • Nassir Navab
  • Nikos Komodakis
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9902)


Multimodal registration is a challenging problem due the high variability of tissue appearance under different imaging modalities. The crucial component here is the choice of the right similarity measure. We make a step towards a general learning-based solution that can be adapted to specific situations and present a metric based on a convolutional neural network. Our network can be trained from scratch even from a few aligned image pairs. The metric is validated on intersubject deformable registration on a dataset different from the one used for training, demonstrating good generalization. In this task, we outperform mutual information by a significant margin.



We gratefully acknowledge NVIDIA Corporation for the donated GPU used in this research. ALBERTs atlases are copyrighted by Imperial College of Science, Technology and Medicine and Ioannis S. Gousias 2013. B. Gutiérrez-Becker thanks the financial support of CONACYT and the DAAD.


  1. 1.
    IXI - Information eXtraction from images.
  2. 2.
    Cao, T., Jojic, V., Modla, S., Powell, D., Czymmek, K., Niethammer, M.: Robust multimodal dictionary learning. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 259–266. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40811-3_33CrossRefGoogle Scholar
  3. 3.
    Cheng, X., Zhang, L., Zheng, Y.: Deep similarity learning for multimodal medical images. In: Computer Methods in Biomechanics and Biomedical Engineering: Imaging and Visualization, pp. 1–5 (2015)Google Scholar
  4. 4.
    Glocker, B., Sotiras, A., Komodakis, N., Paragios, N.: Deformable medical image registration: setting the state of the art with discrete methods. Ann. Rev. Biomed. Eng. 13, 219–244 (2011)CrossRefGoogle Scholar
  5. 5.
    Gousias, I., Edwards, A., Rutherford, M., Counsell, S., Hajnal, J., Rueckert, D., Hammers, A.: Magnetic resonance imaging of the newborn brain: manual segmentation of labelled atlases in term-born and preterm infants. Neuroimage 62, 1499–1509 (2012). www.brain-development.orgCrossRefGoogle Scholar
  6. 6.
    Guetter, C., Xu, C., Sauer, F., Hornegger, J.: Learning based non-rigid multi-modal image registration using kullback-leibler divergence. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3750, pp. 255–262. Springer, Heidelberg (2005). doi: 10.1007/11566489_32CrossRefGoogle Scholar
  7. 7.
    Lee, D., Hofmann, M., Steinke, F., Altun, Y., Cahill, N., Scholkopf, B.: Learning similarity measure for multi-modal 3D image registration. In: CVPR, pp. 186–193 (2009)Google Scholar
  8. 8.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)Google Scholar
  9. 9.
    Mattes, D., Haynor, D.R., Vesselle, H., Lewellyn, T.K., Eubank, W.: Nonrigid multimodality image registration. In: SPIE, vol. 4322, pp. 1609–1620 (2001)Google Scholar
  10. 10.
    Michel, F., Bronstein, M., Bronstein, A., Paragios, N.: Boosted metric learning for 3D multi-modal deformable registration. In: ISBI, pp. 1209–1214 (2011)Google Scholar
  11. 11.
    Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_83CrossRefGoogle Scholar
  12. 12.
    Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: a survey. TMI 32(7), 1153–1190 (2013)Google Scholar
  13. 13.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: CVPR, pp. 4353–4361 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Martin Simonovsky
    • 1
    Email author
  • Benjamín Gutiérrez-Becker
    • 2
  • Diana Mateus
    • 2
  • Nassir Navab
    • 2
  • Nikos Komodakis
    • 1
  1. 1.ImagineUniversité Paris Est/École des Ponts ParisTechChamps-sur-MarneFrance
  2. 2.Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany

Personalised recommendations