Retinal Image Understanding Emerges from Self-Supervised Multimodal Reconstruction

  • Álvaro S. HervellaEmail author
  • José Rouco
  • Jorge Novo
  • Marcos Ortega
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11070)


The successful application of deep learning-based methodologies is conditioned by the availability of sufficient annotated data, which is usually critical in medical applications. This has motivated the proposal of several approaches aiming to complement the training with reconstruction tasks over unlabeled input data, complementary broad labels, augmented datasets or data from other domains. In this work, we explore the use of reconstruction tasks over multiple medical imaging modalities as a more informative self-supervised approach. Experiments are conducted on multimodal reconstruction of retinal angiography from retinography. The results demonstrate that the detection of relevant domain-specific patterns emerges from this self-supervised setting.


Self-supervised Multimodal Retinography Angiography 



This work is supported by I.S. Carlos III, Government of Spain, and the ERDF of the EU through the DTS15/00153 research project, and by the MINECO, Government of Spain, through the DPI2015-69948-R research project. The authors of this work also receive financial support from the ERDF and ESF of the EU, and the Xunta de Galicia through Centro Singular de Investigación de Galicia, accreditation 2016–2019, ref. ED431G/01 and Grupo de Referencia Competitiva, ref. ED431C 2016-047 research projects, and the predoctoral grant contract ref. ED481A-2017/328.


  1. 1.
    Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017)CrossRefGoogle Scholar
  2. 2.
    Karri, S.P.K., Chakraborty, D., Chatterjee, J.: Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration. Biomed. Opt. Express 8(2), 579–592 (2017)CrossRefGoogle Scholar
  3. 3.
    Shin, H., Orton, M.R., Collins, D.J., Doran, S.J., Leach, M.O.: Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4D patient data. IEEE Trans. PAMI 35(8), 1930–1943 (2013)CrossRefGoogle Scholar
  4. 4.
    Rasmus, A., Berglund, M., Honkala, M., Valpola, H., Raiko, T.: Semi-supervised learning with ladder networks. Adv. Neural Inf. Process. Syst. 28, 3546–3554 (2015)Google Scholar
  5. 5.
    Ruder, S.: An overview of multi-task learning in deep neural networks. CoRR abs/1706.05098 (2017)Google Scholar
  6. 6.
    Tan, J.H., Acharya, U.R., Bhandary, S.V., Chua, K.C., Sivaprasad, S.: Segmentation of optic disc, fovea and retinal vasculature using a single convolutional neural network. J. Comput. Sci. 20, 70–79 (2017)Google Scholar
  7. 7.
    Costa, P., Galdran, A., Meyer, M.I., Niemeijer, M., Abràmoff, M., Mendonça, A.M., Campilho, A.: End-to-end adversarial retinal image synthesis. IEEE Trans. Med. Imaging 37(3), 781–791 (2018)CrossRefGoogle Scholar
  8. 8.
    Alipour, S.H.M., Rabbani, H., Akhlaghi, M.R.: Diabetic retinopathy grading by digital curvelet transform. Comput. Math. Methods Med. (2012)Google Scholar
  9. 9.
    Hervella, A.S., Rouco, J., Novo, J., Ortega, M.: Multimodal registration of retinal images using domain-specific landmarks and vessel enhancement. In: International Conference on Knowledge-Based and Intelligent Information and Engineering Systems (KES) (2018)CrossRefGoogle Scholar
  10. 10.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  11. 11.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, December 2015Google Scholar
  13. 13.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR), May 2015Google Scholar
  14. 14.
    Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., van Ginneken, B.: Ridge based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23(4), 501–509 (2004)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Álvaro S. Hervella
    • 1
    • 2
    Email author
  • José Rouco
    • 1
    • 2
  • Jorge Novo
    • 1
    • 2
  • Marcos Ortega
    • 1
    • 2
  1. 1.CITIC-Research Center of Information and Communication TechnologiesUniversity of A CoruñaA CoruñaSpain
  2. 2.Department of Computer ScienceUniversity of A CoruñaA CoruñaSpain

Personalised recommendations