A No-Reference Quality Metric for Retinal Vessel Tree Segmentation

  • Adrian GaldranEmail author
  • Pedro Costa
  • Alessandro Bria
  • Teresa Araújo
  • Ana Maria Mendonça
  • Aurélio Campilho
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11070)


Due to inevitable differences between the data used for training modern CAD systems and the data encountered when they are deployed in clinical scenarios, the ability to automatically assess the quality of predictions when no expert annotation is available can be critical. In this paper, we propose a new method for quality assessment of retinal vessel tree segmentations in the absence of a reference ground-truth. For this, we artificially degrade expert-annotated vessel map segmentations and then train a CNN to predict the similarity between the degraded images and their corresponding ground-truths. This similarity can be interpreted as a proxy to the quality of a segmentation. The proposed model can produce a visually meaningful quality score, effectively predicting the quality of a vessel tree segmentation in the absence of a manually segmented reference. We further demonstrate the usefulness of our approach by applying it to automatically find a threshold for soft probabilistic segmentations on a per-image basis. For an independent state-of-the-art unsupervised vessel segmentation technique, the thresholds selected by our approach lead to statistically significant improvements in F1-score \((+2.67\%)\) and Matthews Correlation Coefficient (+\(3.11\%\)) over the thresholds derived from ROC analysis on the training set. The score is also shown to correlate strongly with F1 and MCC when a reference is available.



This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT - Fundação para a Ciência e a Tecnologia within project CMUP-ERI/TIC/0028/2014. Teresa Araújo is funded by the FCT grant contract SFRH/BD/122365/2016. The Titan Xp used for this research was donated by the NVIDIA Corporation.


  1. 1.
    Azzopardi, G., Strisciuglio, N., Vento, M., Petkov, N.: Trainable COSFIRE filters for vessel delineation with application to retinal images. Med. Image Anal. 19(1), 46–57 (2015)CrossRefGoogle Scholar
  2. 2.
    Decenciére, E., et al.: Feedback on a publicly distributed image database: the messidor database. Image Anal. Stereol. 33(3), 231–234 (2014)CrossRefGoogle Scholar
  3. 3.
    Gegundez-Arias, M.E., Aquino, A., Bravo, J.M., Marin, D.: A function for quality evaluation of retinal vessel segmentations. IEEE Trans. Med. Imaging 31(2), 231–239 (2012)CrossRefGoogle Scholar
  4. 4.
    Ghasemi, A., Zahediasl, S.: Normality tests for statistical analysis: a guide for non-statisticians. Int. J. Endocrinol. Metab. 10(2), 486 (2012)CrossRefGoogle Scholar
  5. 5.
    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], December 2014
  6. 6.
    Melbourne, A., Hawkes, D., Atkinson, D.: Image registration using uncertainty coefficients. In: 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 951–954, June 2009Google Scholar
  7. 7.
    Pluim, J.P.W., Maintz, J.B.A., Viergever, M.A.: Mutual-information-based registration of medical images: a survey. IEEE Trans. Med. Imaging 22(8), 986–1004 (2003)CrossRefGoogle Scholar
  8. 8.
    Prentašić, P., et al.: Diabetic retinopathy image database(DRiDB): a new database for diabetic retinopathy screening programs research. In: 2013 8th International Symposium on Image and Signal Processing and Analysis (ISPA), pp. 711–716, September 2013Google Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  10. 10.
    Spearman, C.: The proof and measurement of association between two things. Am. J. Psychol. 15(1), 72–101 (1904)CrossRefGoogle Scholar
  11. 11.
    Staal, J., Abramoff, M.D., Niemeijer, M., Viergever, M.A.: Ginneken, B.v.: Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23(4), 501–509 (2004)Google Scholar
  12. 12.
    Valindria, V.V., et al.: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE Trans. Med. Imaging 36(8), 1597–1606 (2017)CrossRefGoogle Scholar
  13. 13.
    Wilcoxon, F.: Individual comparisons by ranking methods. Biom. Bull. 1(6), 80–83 (1945)CrossRefGoogle Scholar
  14. 14.
    Yan, Z., Yang, X., Cheng, K.T.T.: A skeletal similarity metric for quality evaluation of retinal vessel segmentation. IEEE Trans. Med. Imaging PP(99), 1 (2017)Google Scholar
  15. 15.
    Zhang, J., et al.: Robust retinal vessel segmentation via locally adaptive derivative frames in orientation scores. IEEE Trans. Med. Imaging 35(12), 2631–2644 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Adrian Galdran
    • 1
    Email author
  • Pedro Costa
    • 1
  • Alessandro Bria
    • 2
  • Teresa Araújo
    • 1
    • 3
  • Ana Maria Mendonça
    • 1
    • 3
  • Aurélio Campilho
    • 1
    • 3
  1. 1.INESC-TEC - Institute for Systems and Computer Engineering, Technology and SciencePortoPortugal
  2. 2.Università degli studi di Cassino e del Lazio MeridionaleCassinoItaly
  3. 3.Faculdade de Engenharia da Universidade do PortoPortoPortugal

Personalised recommendations