Abstract
Non-mydriatic retinal color fundus photography (CFP) is widely available due to the advantage of not requiring pupillary dilation, however, is prone to poor quality due to operators, systemic imperfections, or patient-related causes. Optimal retinal image quality is mandated for accurate medical diagnoses and automated analyses. Herein, we leveraged the Optimal Transport (OT) theory to propose an unpaired image-to-image translation scheme for mapping low-quality retinal CFPs to high-quality counterparts. Furthermore, to improve the flexibility, robustness, and applicability of our image enhancement pipeline in the clinical practice, we generalized a state-of-the-art model-based image reconstruction method, regularization by denoising, by plugging in priors learned by our OT-guided image-to-image translation network. We named it as regularization by enhancing (RE). We validated the integrated framework, OTRE, on three publicly available retinal image datasets by assessing the quality after enhancement and their performance on various downstream tasks, including diabetic retinopathy grading, vessel segmentation, and diabetic lesion segmentation. The experimental results demonstrated the superiority of our proposed framework over some state-of-the-art unsupervised competitors and a state-of-the-art supervised method.
W. Zhu and P. Qiu—The two authors contributed equally to this paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wolf, R.M., Channa, R., Abramoff, M.D., Lehmann, H.P.: Cost-effectiveness of autonomous point-of-care diabetic retinopathy screening for pediatric patients with diabetes. JAMA Ophthalmol. 138(10), 1063–1069 (2020)
Cheung, C.Y., et al.: A deep learning model for detection of Alzheimer’s disease based on retinal photographs: a retrospective, multicentre case-control study. Lancet Digit. Health 4(11), e806–e815 (2022)
Shen, Z., Fu, H., Shen, J., Shao, L.: Modeling and enhancing low-quality retinal fundus images. IEEE Trans. Med. Imaging 40(3), 996–1006 (2021)
Lehtinen, J., et al.: Noise2Noise: learning image restoration without clean data. ICML 80, 2965–2974 (2018)
Krull, A., et al.: Noise2void-learning denoising from single noisy images. In: Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 2129–2137 (2019)
Bousmalis, K., Silberman, N., et al.: Unsupervised pixel-level domain adaptation with generative adversarial networks. In: CVPR (2016)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)
Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: CVPR, pp. 2242–2251 (2017)
Liu, M.Y., Tuzel, O.: Coupled generative adversarial networks. In: Advances in Neural Information Processing Systems (2016)
Wang, W., Wen, F., Yan, Z., Liu, P.: Optimal transport for unsupervised denoising learning. IEEE PAMI 1 (2022)
Romano, Y., Elad, M., Milanfar, P.: The little engine that could: regularization by denoising (RED). SIAM J. Imag. Sci. 10(4), 1804–1844 (2017)
Ryu, E., Liu, J., Wang, S., Chen, X., Wang, Z., Yin, W.: Plug-and-play methods provably converge with properly trained denoisers. PMLR 97, 5546–5557 (2019)
Lucas, A., Iliadis, M., Molina, R., Katsaggelos, A.K.: Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Process. Mag. 35(1), 20–36 (2018)
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: ECA-net: efficient channel attention for deep convolutional neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (2020)
Ledig, C., Theis, L., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2016)
Gulrajani, I., Ahmed, F., Arjovsky, M., et al.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Brunet, D., Vrscay, E.R., Wang, Z.: On the mathematical properties of the structural similarity index. IEEE Trans. Image Process. 21(4), 1488–1499 (2012)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Miyato, T., et al.: Spectral normalization for generative adversarial networks. In: International Conference on Learning Representations (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2016)
Fu, H., et al.: Evaluation of retinal image quality assessment networks in different color-spaces. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 48–56. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_6
Staal, J., et al.: Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23(4), 501–509 (2004)
Porwal, P., et al.: IDRID: a database for diabetic retinopathy screening research. Data 3(3) (2018)
Zhu, W., Qiu, P., Lepore, N., Dumitrascu, O., Wang, Y.: Self-supervised equivariant regularization reconciles multiple instance learning: joint referable diabetic retinopathy classification and lesion segmentation. In: 18th International Symposium on Medical Information Processing and Analysis (SIPAIM) (2022)
Acknowledgement
This work was partially supported by grants from NIH (R21AG065942, R01EY032125, and R01DE030286).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, W. et al. (2023). OTRE: Where Optimal Transport Guided Unpaired Image-to-Image Translation Meets Regularization by Enhancing. In: Frangi, A., de Bruijne, M., Wassermann, D., Navab, N. (eds) Information Processing in Medical Imaging. IPMI 2023. Lecture Notes in Computer Science, vol 13939. Springer, Cham. https://doi.org/10.1007/978-3-031-34048-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-34048-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34047-5
Online ISBN: 978-3-031-34048-2
eBook Packages: Computer ScienceComputer Science (R0)