Unsupervised Feature Learning for Endomicroscopy Image Retrieval

  • Yun Gu
  • Khushi Vyas
  • Jie YangEmail author
  • Guang-Zhong Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)


Learning the visual representation for medical images is a critical task in computer-aided diagnosis. In this paper, we propose Unsupervised Multimodal Graph Mining (UMGM) to learn the discriminative features for probe-based confocal laser endomicroscopy (pCLE) mosaics of breast tissue. We build a multiscale multimodal graph based on both pCLE mosaics and histology images. The positive pairs are mined via cycle consistency and the negative pairs are extracted based on geodetic distance. Given the positive and negative pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images. Experiments on a database with 700 pCLE mosaics demonstrate that the proposed method outperforms previous works on pCLE feature learning. Specially, the top-1 accuracy in an eight-class retrieval task is 0.659 which leads to 10% improvement compared with the state-of-the-art method.


Probe-based laser endomicroscopy Histology Graph mining Feature learning 



This work is partially supported by NSFC, China (No: 61572315,6151101179) and 973 Plan, China (No. 2015CB856004). Yun Gu is also supported by Chinese Scholarship Council (CSC). The tissue specimens were obtained from consented patients using the Imperial tissue bank ethical protocol following the R-12047 project.


  1. 1.
    Jiang, M., Zhang, S., Huang, J., Yang, L., Metaxas, D.N.: Scalable histopathological image analysis via supervised hashing with multiple features. Med. Image Anal. 34, 3–12 (2016)CrossRefGoogle Scholar
  2. 2.
    Zhang, X., Su, H., Yang, L., Zhang, S.: Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval. In: CVPR, pp. 5361–5368 (2015)Google Scholar
  3. 3.
    André, B., Vercauteren, T., Buchner, A.M., Wallace, M.B., Ayache, N.: A smart atlas for endomicroscopy using automated video retrieval. Med. Image Anal. 15(4), 460–476 (2011)CrossRefGoogle Scholar
  4. 4.
    André, B., Vercauteren, T., Perchant, A., Buchner, A.M., Wallace, M.B., Ayache, N.: Endomicroscopic image retrieval and classification using invariant visual features. In: IEEE ISBI 2009, pp. 346–349. IEEE (2009)Google Scholar
  5. 5.
    Gu, Y., Yang, J., Yang, G.Z.: Multi-view multi-modal feature embedding for endomicroscopy mosaic classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 11–19 (2016)Google Scholar
  6. 6.
    Li, D., Hung, W.-C., Huang, J.-B., Wang, S., Ahuja, N., Yang, M.-H.: Unsupervised visual representation learning by graph-based consistent constraints. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 678–694. Springer, Cham (2016). doi: 10.1007/978-3-319-46493-0_41 CrossRefGoogle Scholar
  7. 7.
    Zhai, X., Peng, Y., Xiao, J.: Heterogeneous metric learning with joint graph regularization for cross-media retrieval. In: AAAI 2013, pp. 1198–1204 (2013)Google Scholar
  8. 8.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. JMLR 9, 2579–2605 (2008)zbMATHGoogle Scholar
  9. 9.
    Floyd, R.W.: Algorithm 97: shortest path. Commun. ACM 5(6), 345 (1962)CrossRefGoogle Scholar
  10. 10.
    Mignon, A., Jurie, F.: Cmml: a new metric learning approach for cross modal matching. In: Asian Conference on Computer Vision, p. 14 (2012)Google Scholar
  11. 11.
    Chang, T.P., Leff, D.R., Shousha, S., Hadjiminas, D.J., Ramakrishnan, R., Hughes, M.R., Yang, G.Z., Darzi, A.: Imaging breast cancer morphology using probe-based confocal laser endomicroscopy: towards a real-time intraoperative imaging tool for cavity scanning. Breast Cancer Res. Treat. 153(2), 299–310 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yun Gu
    • 1
    • 2
    • 3
  • Khushi Vyas
    • 3
  • Jie Yang
    • 1
    • 2
    Email author
  • Guang-Zhong Yang
    • 3
  1. 1.School of Biomedical EngineeringShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Institute of Image Processing and Pattern RecognitionShanghai Jiao Tong UniversityShanghaiChina
  3. 3.Hamlyn Centre for Robotic SurgeryImperial College LondonLondonUK

Personalised recommendations