Deep Learning Based Multi-modal Registration for Retinal Imaging

  • Mustafa Arikan
  • Amir Sadeghipour
  • Bianca Gerendas
  • Reinhard Told
  • Ursula Schmidt-ErfurtEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11797)


The precise alignment of retina images from different modalities allows ophthalmologists not only to track morphological/pathological changes over time but also to combine different modalities to approach the diagnosis, prognostication, management and monitoring of a retinal disease. We propose an image registration algorithm to trace changes in the retina structure across modalities using vessel segmentation and automatic landmark detection. The segmentation of the vessels is done using a U-Net and the detection of the vessel junctions is achieved with Mask R-CNN. We evaluated the results of our approach using manual grading by expert readers. In the largest dataset (FA-to-SLO/OCT) containing 1130 pairs we achieve an average error rate of 13.12%. We compared our method with intensity based affine registration methods using original and vessel segmentation images.


  1. 1.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. CoRR abs/1703.06870 (2017).
  2. 2.
    Hervella, Á.S., Rouco, J., Novo, J., Ortega, M.: Multimodal registration of retinal images using domain-specific landmarks and vessel enhancement. CoRR abs/1803.00951 (2018).
  3. 3.
    Khojasteh, P., Aliahmad, B., Kumar, D.K.: Fundus images analysis using deep features for detection of exudates, hemorrhages and microaneurysms. In: BMC Ophthalmology (2018)Google Scholar
  4. 4.
    Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.W.: elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imag. 29(1), 196–205 (2010). Scholar
  5. 5.
    Lam, C., Yi, D., Guo, M., Lindsey, T.: Automated detection of diabetic retinopathy using deep learning. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science 2017, 147–155 (2018)Google Scholar
  6. 6.
    Li, Y., Gregori, G., Knighton, R.W., Lujan, B., Rosenfeld, P.: Registration of OCT fundus images with color fundus photographs based on blood vessel ridges. Opt. Express 19, 7–16 (2011). Scholar
  7. 7.
    Li, Z., Huang, F., Zhang, J., Dashtbozorg, B., Abbasi-Sureshjani, S., Sun, Y., Long, X., Yu, Q., ter Haar Romeny, B., Tan, T.: Multi-modal and multi-vendor retina image registration. Biomed. Opt. Express 9(2), 410–422 (2018)CrossRefGoogle Scholar
  8. 8.
    Liskowski, P., Krawiec, K.: Segmenting retinal blood vessels with deep neural networks. IEEE Trans. Med. Imag. 35, 1–1 (2016)CrossRefGoogle Scholar
  9. 9.
    Mattes, D., Haynor, D.R., Vesselle, H., Lewellyn, T.K., Eubank, W.: Nonrigid multimodality image registration. Proc. SPIE - Int. Soc. Opt. Eng. 4322, 1609–1620 (2001). Scholar
  10. 10.
    Miri, M.S., Abramoff, M., Kwon, Y.H., Garvin, M.K.: Multimodal registration of SD-OCT volumes and fundus photographs using histograms of oriented gradients. Biomed. Opt. Express 7, 5252–5267 (2016)CrossRefGoogle Scholar
  11. 11.
    Novais, E., Baumal, C., Sarraf, D., Freund, K., Duker, J.: Multimodal imaging in retinal disease: a consensus definition. Ophthalmic Surg. Lasers & Imag. Retina 47, 201–205 (2016). Scholar
  12. 12.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)CrossRefGoogle Scholar
  13. 13.
    Palén, A.: Advanced algorithms for manipulating 2D objects on touch screens. Master’s thesis, Tampere University of Technology (2016)Google Scholar
  14. 14.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  15. 15.
    Schlegl, T., et al.: Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology 125(4), 549–558 (2018)CrossRefGoogle Scholar
  16. 16.
    Schmitz-Valckenberg, S., Holz, F., Bird, A., Spaide, R.F.: Fundus autofluorescence imaging: review and perspectives. Retina (Philadelphia, Pa.) 28, 385–409 (2008). Scholar
  17. 17.
    Zana, F., Klein, J.C.: A multimodal registration algorithm of eye fundus images using vessels detection and hough transform. IEEE Trans. Med. Imag. 18(5), 419–428 (1999). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mustafa Arikan
    • 1
    • 2
  • Amir Sadeghipour
    • 1
    • 2
  • Bianca Gerendas
    • 1
    • 2
  • Reinhard Told
    • 2
  • Ursula Schmidt-Erfurt
    • 1
    • 2
    Email author
  1. 1.Christian Doppler Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and OptometryMedical University of ViennaViennaAustria
  2. 2.Department of Ophthalmology and OptometryMedical University of ViennaViennaAustria

Personalised recommendations