Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)


Anomaly detection in retinal image refers to the identification of abnormality caused by various retinal diseases/lesions, by only leveraging normal images in training phase. Normal images from healthy subjects often have regular structures (e.g., the structured blood vessels in the fundus image, or structured anatomy in optical coherence tomography image). On the contrary, the diseases and lesions often destroy these structures. Motivated by this, we propose to leverage the relation between the image texture and structure to design a deep neural network for anomaly detection. Specifically, we first extract the structure of the retinal images, then we combine both the structure features and the last layer features extracted from original health image to reconstruct the original input healthy image. The image feature provides the texture information and guarantees the uniqueness of the image recovered from the structure. In the end, we further utilize the reconstructed image to extract the structure and measure the difference between structure extracted from original and the reconstructed image. On the one hand, minimizing the reconstruction difference behaves like a regularizer to guarantee that the image is corrected reconstructed. On the other hand, such structure difference can also be used as a metric for normality measurement. The whole network is termed as P-Net because it has a “P” shape. Extensive experiments on RESC dataset and iSee dataset validate the effectiveness of our approach for anomaly detection in retinal images. Further, our method also generalizes well to novel class discovery in retinal images and anomaly detection in real-world images.


Structure-texture relation Anomaly detection Novel class discovery 



The work was supported by National Key R&D Program of China (2018AAA0100704), NSFC #61932020, Guangdong Provincial Key Laboratory (2020B121201001), ShanghaiTech-Megavii Joint Lab, and ShanghaiTech-UnitedImaging Joint Lab.

Supplementary material

504476_1_En_22_MOESM1_ESM.pdf (727 kb)
Supplementary material 1 (pdf 726 KB)


  1. 1.
    Boyd, K.: What is diabetic retinopathy. (2019)
  2. 2.
    Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  3. 3.
    Xing, F., Xie, Y., Su, H., Liu, F., Yang, L.: Deep learning in microscopy image analysis: a survey. IEEE Trans. Neural Netw. Learn. Syst. 29(10), 4550–4568 (2017)CrossRefGoogle Scholar
  4. 4.
    Zhou, S.K., Greenspan, H., Shen, D.: Deep Learning for Medical Image Analysis. Academic Press, Cambridge (2017)Google Scholar
  5. 5.
    Zhou, K., et al.: Multi-cell multi-task convolutional neural networks for diabetic retinopathy grading. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2724–2727. IEEE (2018)Google Scholar
  6. 6.
    Fu, H., Cheng, J., Xu, Y., Wong, D.W.K., Liu, J., Cao, X.: Joint optic disc and cup segmentation based on multi-label deep network and polar transformation. IEEE Trans. Med. Imaging 37(7), 1597–1605 (2018)CrossRefGoogle Scholar
  7. 7.
    Zhang, S., et al.: Attention guided network for retinal image segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 797–805. Springer, Cham (2019). Scholar
  8. 8.
    Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 146–157. Springer, Cham (2017). Scholar
  9. 9.
    Baur, C., Wiestler, B., Albarqouni, S., Navab, N.: Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11383, pp. 161–169. Springer, Cham (2019). Scholar
  10. 10.
    Chen, X., Konukoglu, E.: Unsupervised detection of lesions in brain MRI using constrained adversarial auto-encoders. arXiv preprint arXiv:1806.04972 (2018)
  11. 11.
    Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: f-anogan: fast unsupervised anomaly detection with generative adversarial networks. Med. Image Anal. 54, 30–44 (2019)CrossRefGoogle Scholar
  12. 12.
    Zhou, C., Paffenroth, R.C.: Anomaly detection with robust deep auto encoders. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665–674. ACM (2017)Google Scholar
  13. 13.
    Zimmerer, D., Kohl, S.A., Petersen, J., Isensee, F., Maier-Hein, K.H.: Context-encoding variational auto encoder for unsupervised anomaly detection. arXiv preprint arXiv:1812.05941 (2018)
  14. 14.
    Zhou, K., et al.: Sparse-gan: sparsity-constrained generative adversarial network for anomaly detection in retinal oct image. In: 2020 IEEE 17th International Symposium on Biomedical Imaging(ISBI), pp. 1227–1231. IEEE (2020)Google Scholar
  15. 15.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  16. 16.
    Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 622–637. Springer, Cham (2019). Scholar
  17. 17.
    Puliafito, C.A., et al.: Imaging of macular diseases with optical coherence tomography. Ophthalmology 102(2), 217–229 (1995)CrossRefGoogle Scholar
  18. 18.
    Zinreich, S.J., et al.: Fungal sinusitis: diagnosis with ct and mr imaging. Radiology 169(2), 439–444 (1988)CrossRefGoogle Scholar
  19. 19.
    Hartnett, M.E., Weiter, J.J., Staurenghi, G., Elsner, A.E.: Deep retinal vascular anomalous complexes in advanced age-related macular degeneration. Ophthalmology 103(12), 2042–2053 (1996)CrossRefGoogle Scholar
  20. 20.
    Chai, Z., et al.: Perceptual-assisted adversarial adaptation for choroid segmentation in optical coherence tomography. In: 2020 IEEE 17th International Symposium on Biomedical Imaging(ISBI), pp. 1966–1970. IEEE (2020)Google Scholar
  21. 21.
    Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.C.: Support vector method for novelty detection. In: Advances in Neural Information Processing Systems, pp. 582–588 (2000)Google Scholar
  22. 22.
    Ruff, L., et al.: Deep one-class classification. In: International Conference on Machine Learning, pp. 4393–4402 (2018)Google Scholar
  23. 23.
    Perera, P., Nallapati, R., Xiang, B.: Ocgan: one-class novelty detection using gans with constrained latent representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2906 (2019)Google Scholar
  24. 24.
    Gong, D., et al.: Memorizing normality to detect anomaly: memory-augmented deep auto encoder for unsupervised anomaly detection. arXiv preprint arXiv:1904.02639 (2019)
  25. 25.
    Aujol, J.F., Gilboa, G., Chan, T., Osher, S.: Structure-texture image decomposition–modeling, algorithms, and parameter selection. Int. J. Comput. Vis. 67(1), 111–136 (2006)CrossRefGoogle Scholar
  26. 26.
    Ren, Y., Yu, X., Zhang, R., Li, T.H., Liu, S., Li, G.: Structure flow: image in painting via structure-aware appearance flow. In: The IEEE International Conference on Computer Vision (ICCV), October 2019Google Scholar
  27. 27.
    Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: Edge connect: structure guided image in painting using edge prediction. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019Google Scholar
  28. 28.
    Hu, J., Chen, Y., Yi, Z.: Automated segmentation of macular edema in oct using deep neural networks. Med. Image Anal. 55, 216–227 (2019)CrossRefGoogle Scholar
  29. 29.
    Yan, Y., et al.: Oversampling for imbalanced data via optimal transport. Proc. AAAI Conf. Artif. Intell. 33, 5605–5612 (2019)Google Scholar
  30. 30.
    Hoover, A., Kouznetsova, V., Goldbaum, M.: Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Trans. Med. Imaging 19(3), 203–210 (2000)CrossRefGoogle Scholar
  31. 31.
    Staal, J., Abràmoff, M.D., Niemeijer, M., Viergever, M.A., Van Ginneken, B.: Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23(4), 501–509 (2004)CrossRefGoogle Scholar
  32. 32.
    Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7472–7481 (2018)Google Scholar
  33. 33.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  34. 34.
    Cheng, J., et al.: Speckle reduction in 3d optical coherence tomography of retina by a-scan reconstruction. IEEE Trans. Med. Imaging 35(10), 2270–2279 (2016)CrossRefGoogle Scholar
  35. 35.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)Google Scholar
  36. 36.
    Chen, X., Pawlowski, N., Rajchl, M., Glocker, B., Konukoglu, E.: Deep generative models in the real-world: an open challenge from medical imaging. arXiv preprint arXiv:1806.05452 (2018)
  37. 37.
    Luo, W., Liu, W., Gao, S.: A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 341–349 (2017)Google Scholar
  38. 38.
    Luo, W., et al.: Video anomaly detection with sparse coding inspired deep neural networks. IEEE Trans. Pattern Anal. Mach. Intell. (2019)Google Scholar
  39. 39.
    Porwal, P., et al.: Idrid: diabetic retinopathy-segmentation and grading challenge. Med. Image Anal. 59, 101561 (2019)CrossRefGoogle Scholar
  40. 40.
    Orlando, J.I., et al.: Refuge challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Med. Image Anal. 59, 101570 (2020)CrossRefGoogle Scholar
  41. 41.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)Google Scholar
  42. 42.
    Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: Mvtec ad-a comprehensive real-world dataset for unsupervised anomaly detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9592–9600 (2019)Google Scholar
  43. 43.
    Napoletano, P., Piccoli, F., Schettini, R.: Anomaly detection in nanofibrous materials by cnn-based self-similarity. Sensors 18(1), 209 (2018)CrossRefGoogle Scholar
  44. 44.
    Böttger, T., Ulrich, M.: Real-time texture error detection on textured surfaces with compressed sensing. Pattern Recognit. Image Anal. 26(1), 88–94 (2016). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Information Science and TechnologyShanghaiTech UniversityShanghaiChina
  2. 2.Cixi Institute of Biomedical Engineering, Chinese Academy of SciencesBeijingChina
  3. 3.UBTech ResearchShenzhenChina
  4. 4.Southern University of Science and TechnologyShenzhenChina
  5. 5.Shanghai Engineering Research Center of Intelligent Vision and ImagingShanghaiChina

Personalised recommendations