Abstract
A large-scale labeled dataset is a key factor for the success of supervised deep learning in computer vision. However, a limited number of annotated data is very common, especially in ophthalmic image analysis, since manual annotation is time-consuming and labor-intensive. Self-supervised learning (SSL) methods bring huge opportunities for better utilizing unlabeled data, as they do not need massive annotations. With an attempt to use as many as possible unlabeled ophthalmic images, it is necessary to break the dimension barrier, simultaneously making use of both 2D and 3D images. In this paper, we propose a universal self-supervised Transformer framework, named Uni4Eye, to discover the inherent image property and capture domain-specific feature embedding in ophthalmic images. Uni4Eye can serve as a global feature extractor, which builds its basis on a Masked Image Modeling task with a Vision Transformer (ViT) architecture. We employ a Unified Patch Embedding module to replace the origin patch embedding module in ViT for jointly processing both 2D and 3D input images. Besides, we design a dual-branch multitask decoder module to simultaneously perform two reconstruction tasks on the input image and its gradient map, delivering discriminative representations for better convergence. We evaluate the performance of our pre-trained Uni4Eye encoder by fine-tuning it on six downstream ophthalmic image classification tasks. The superiority of Uni4Eye is successfully established through comparisons to other state-of-the-art SSL pre-training methods.
Z. Cai and L. Lin—Contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Atito, S., Awais, M., Kittler, J.: Sit: self-supervised vision transformer. arXiv preprint arXiv: 2104.03602 (2021)
Bao, H., Dong, L., et al.: BEIT: BERT pre-training of image transformers. In: International Conference on Learning Representations, ICLR (2022)
Cai, Z., Lin, L., He, H., Tang, X.: Corolla: an efficient multi-modality fusion framework with supervised contrastive learning for glaucoma grading. arXiv preprint arXiv: 2201.03795 (2022)
Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local features for medical image segmentation with limited annotations. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 33 (2020)
Chen, L., Bentley, P., et al.: Self-supervised learning for medical image analysis using image context restoration. IEEE Trans. Med. Imaging 58, 101539 (2019). https://doi.org/10.1016/j.media.2019.101539
Chen, S., Ma, K., et al.: Med3D: transfer learning for 3D medical image analysis. arXiv preprint arXiv: 1904.00625 (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv: 2002.05709 (2020)
Cordeiro, F.R., Sachdeva, R., et al.: LongReMix: robust learning with high confidence samples in a noisy label environment. arXiv preprint arXiv: 2103.04173 (2021)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 248–255 (2009)
Donahue, J., Simonyan, K.: Large scale adversarial representation learning. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 32 (2019)
Dosovitskiy, A., Beyer, L., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929 (2021)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018)
Goodfellow, I.J., Pouget-Abadie, J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 27 (2014)
He, H., Lin, L., Cai, Z., Tang, X.: JOINED: prior guided multi-task learning for joint optic disc/cup segmentation and fovea detection. In: International Conference on Medical Imaging with Deep Learning, MIDL (2022)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv preprint arXiv: 2111.06377 (2021)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 9729–9738 (2020)
Huang, Y., Lin, L., Cheng, P., Lyu, J., Tang, X.: Lesion-based contrastive learning for diabetic retinopathy grading from fundus images. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 113–123. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_11
Kanopoulos, N., Vasanthavada, N., Baker, R.L.: Design of an image edge detection filter using the Sobel operator. IEEE J. Solid State Circuits 23(2), 358–367 (1988)
Li, X., Hu, X., et al.: Rotation-oriented collaborative self-supervised learning for retinal disease diagnosis. IEEE Trans. Med. Imaging 40(9), 2284–2294 (2021)
Li, X., Jia, M., Islam, M.T., Yu, L., Xing, L.: Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis. IEEE Trans. Med. Imaging 39(12), 4023–4033 (2020)
Lin, L., et al.: The SUSTech-SYSU dataset for automated exudate detection and diabetic retinopathy grading. Sci. Data 7(1), 1–10 (2020)
Lin, L., et al.: BSDA-Net: a boundary shape and distance aware joint learning framework for segmenting and classifying OCTA images. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 65–75. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_7
Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. arXiv preprint arXiv: 1711.05101 (2017)
Oliver, A., Odena, A., Raffel, C., Cubuk, E.D., Goodfellow, I.J.: Realistic evaluation of deep semi-supervised learning algorithms. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 31 (2019)
Paszke, A., Gross, S., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 32 (2019)
Taleb, A., Loetzsch, W., et al.: 3D self-supervised methods for medical imaging. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 33 (2020)
Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, ICML, pp. 6105–6114 (2019)
Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv: 2112.09133 (2021)
Ye, M., Zhang, X., Yuen, P.C., Chang, S.F.: Unsupervised embedding learning via invariant and spreading instance feature. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 6210–6219 (2019)
Zhou, H.Y., Lu, C., et al.: Preservational learning improves self-supervised medical image models by reconstructing diverse contexts. In: The IEEE International Conference on Computer Vision, ICCV, pp. 3499–3509 (2021)
Zhuang, X., Li, Y., Hu, Y., Ma, K., Yang, Y., Zheng, Y.: Self-supervised feature learning for 3D medical images by playing a Rubik’s cube. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 420–428. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_46
Acknowledgement
This study was supported by the Shenzhen Basic Research Program (JCYJ20190 809120205578); the National Natural Science Foundation of China (62071210); the Shenzhen Science and Technology Program (RCYX20210609103056042); the Shenzhen Basic Research Program (JCYJ20200925153847004).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cai, Z., Lin, L., He, H., Tang, X. (2022). Uni4Eye: Unified 2D and 3D Self-supervised Pre-training via Masked Image Modeling Transformer for Ophthalmic Image Classification. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13438. Springer, Cham. https://doi.org/10.1007/978-3-031-16452-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-16452-1_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16451-4
Online ISBN: 978-3-031-16452-1
eBook Packages: Computer ScienceComputer Science (R0)
