Deep Multiple Instance Hashing for Scalable Medical Image Retrieval

  • Sailesh ConjetiEmail author
  • Magdalini Paschali
  • Amin Katouzian
  • Nassir Navab
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)


In this paper, for the first time, we introduce a multiple instance (MI) deep hashing technique for learning discriminative hash codes with weak bag-level supervision suited for large-scale retrieval. We learn such hash codes by aggregating deeply learnt hierarchical representations across bag members through an MI pool layer. For better trainability and retrieval quality, we propose a two-pronged approach that includes robust optimization and training with an auxiliary single instance hashing arm which is down-regulated gradually. We pose retrieval for tumor assessment as an MI problem because tumors often coexist with benign masses and could exhibit complementary signatures when scanned from different anatomical views. Experimental validations demonstrate improved retrieval performance over the state-of-the-art methods.



The authors would like to warmly thank Dr. Shaoting Zhang for generously sharing the datasets used in this paper. We would also like to thank Abhijit Guha Roy and Andrei Costinescu for their insightful comments about this work.


  1. 1.
    Duijm, L.E.M., Louwman, M.W.J., Groenewoud, J.H., van de Poll-Franse, L.V., Fracheboud, J., Coebergh, J.W.: Inter-observer variability in mammography screening and effect of type and number of readers on screening outcome. BJC 24, 901–907 (2009)CrossRefGoogle Scholar
  2. 2.
    Liu, W., Wang, J., Ji, R., Jiang, Y.G., Chang, S.F.: Supervised hashing with kernels. In: CVPR 2012, pp. 2074–2081. IEEE (2012)Google Scholar
  3. 3.
    Conjeti, S., Guha Roy, A., Katouzian, A., Navab, N.: Hashing with residual networks for image retrieval. In: 20th International Conference on Medical Image Computing and Computer Assisted Intervention, Canada (2017)Google Scholar
  4. 4.
    Zhang, X., Liu, W., Dundar, M., Badve, S., Zhang, S.: Towards large-scale histopathological image analysis: hashing-based image retrieval. In: TMI 2015. IEEE (2015)CrossRefGoogle Scholar
  5. 5.
    Veta, M., Pluim, J.P., van Diest, P.J., Viergever, M.A.: Breast cancer histopathology image analysis: a review. Trans. Biomed. Eng. 61, 1400–1411 (2014). IEEECrossRefGoogle Scholar
  6. 6.
    Gong, Y., Lazebnik, S.: Iterative quantization: a procrustean approach to learning binary codes. In: CVPR 2011, pp. 817–824. IEEE (2011)Google Scholar
  7. 7.
    Yang, Y., Xu, X.-S., Wang, X., Guo, S., Cui, L.: Hashing multi-instance data from bag and instance level. In: Cheng, R., Cui, B., Zhang, Z., Cai, R., Xu, J. (eds.) APWeb 2015. LNCS, vol. 9313, pp. 437–448. Springer, Cham (2015). doi: 10.1007/978-3-319-25255-1_36 CrossRefGoogle Scholar
  8. 8.
    Lai, H., Pan, Y., Liu, Y., Yan, S.: Simultaneous feature learning and hash coding with deep neural networks. In: CVPR 2015, pp. 3270–3278 (2015)Google Scholar
  9. 9.
    Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: AAAI 2016 (2016)Google Scholar
  10. 10.
    Wu, J., Yu, Y., Huang, C., Yu, K.: Multiple instance learning for image classification and auto-annotation. In: CVPR 2015, pp. 3460–3469 (2015)Google Scholar
  11. 11.
    Zhennan, Y., Yiqiang, Z., Zhigang, P., Shu, L., Shinagawa, Y., Shaoting, Z., Metaxas, D.N., Xiang, S.Z.: Multi-instance deep learning: discover discriminative local anatomies for bodypart recognition. Trans. Med. Imaging 35, 1332–1343 (2016). IEEECrossRefGoogle Scholar
  12. 12.
    Jiang, M., Zhang, S., Li, H., Metaxas, D.N.: Computer-aided diagnosis of mammographic masses using scalable image retrieval. TBME 62, 783–792 (2015)Google Scholar
  13. 13.
    Chatfeld, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets (2014). arXiv:1405.3531
  14. 14.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR 2015, pp. 1–9 (2015)Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR 2016, pp. 770–778. IEEE Computer Society (2016)Google Scholar
  16. 16.
    Torralba, A., Fergus, R., Weiss, Y.: Small codes and large image databases for recognition. In: CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  17. 17.
    Huber, P.J.: Robust statistics. In: Lovic, M. (ed.) International Encyclopedia of Statistical Science, pp. 1248–1251. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  18. 18.
    Heath, M., Bowyer, K., Kopans, D., Kegelmeyer Jr., W.P., Moore, R., Chang, K., Munishkumaran, S.: Current status of the digital database for screening mammography. In: Karssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (eds.) Digital Mammography, pp. 457–460. Springer, Dordrecht (1998). doi: 10.1007/978-94-011-5318-8_75 CrossRefGoogle Scholar
  19. 19.
    Badve, S., Bilgin, G., Dundar, M., Grcan, M.N., Jain, R.K., Raykar, V.C., Sertel, O.: Computerized classification of intraductal breast lesions using histopathological images. Biomed. Eng. 58, 1977–1984 (2011). IEEEGoogle Scholar
  20. 20.
    Vedaldi, A., Matconvnet, L.K.: Convolutional neural networks for matlab. In: ACM International Conference on Multimedia 2015, pp. 689–692. ACM (2015)Google Scholar
  21. 21.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. IJCV 42, 145–175 (2001)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Sailesh Conjeti
    • 1
    Email author
  • Magdalini Paschali
    • 1
  • Amin Katouzian
    • 2
  • Nassir Navab
    • 1
    • 3
  1. 1.Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
  2. 2.IBM Almaden Research CenterAlmadenUSA
  3. 3.Computer Aided Medical ProceduresJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations