Deep Hashing with Active Pairwise Supervision

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)


In this paper, we propose a Deep Hashing method with Active Pairwise Supervision (DH-APS). Conventional methods with passive pairwise supervision obtain labeled data for training and require large amount of annotations to reach their full potential, which are not feasible in realistic retrieval tasks. On the contrary, we actively select a small quantity of informative samples for annotation to provide effective pairwise supervision so that discriminative hash codes can be obtained with limited annotation budget. Specifically, we generalize the structural risk minimization principle and obtain three criteria for the pairwise supervision acquisition: uncertainty, representativeness and diversity. Accordingly, samples involved in the following training pairs should be labeled: pairs with most uncertain similarity, pairs that minimize the discrepancy between labeled and unlabeled data, and pairs which are most different from the annotated data, so that the discriminality and generalization ability of the learned hash codes are significantly strengthened. Moreover, our DH-APS can also be employed as a plug-and-play module for semi-supervised hashing methods to further enhance the performance. Experiments demonstrate that the presented DH-APS achieves the accuracy of supervised hashing methods with only \(30\%\) labeled training samples and improves the semi-supervised binary codes by a sizable margin.


Active learning Deep hashing Structural risk minimization 



This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 61822603, Grant U1813218, Grant U1713214, and Grant 61672306, in part by Beijing Natural Science Foundation under Grant No. L172051, in part by Beijing Academy of Artificial Intelligence (BAAI), in part by a grant from the Institute for Guo Qiang, Tsinghua University, in part by the Shenzhen Fundamental Research Fund (Subject Arrangement) under Grant JCYJ20170412170602564, and in part by Tsinghua University Initiative Scientific Research Program.

Supplementary material

504475_1_En_31_MOESM1_ESM.pdf (226 kb)
Supplementary material 1 (pdf 226 KB)


  1. 1.
    Babenko, A., Lempitsky, V.: Aggregating local deep features for image retrieval. In: ICCV, pp. 1269–1277 (2015)Google Scholar
  2. 2.
    Balcan, M.-F., Broder, A., Zhang, T.: Margin based active learning. In: Bshouty, N.H., Gentile, C. (eds.) COLT 2007. LNCS (LNAI), vol. 4539, pp. 35–50. Springer, Heidelberg (2007). Scholar
  3. 3.
    Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian complexities: risk bounds and structural results. JMLR 3(Nov), 463–482 (2002)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Beluch, W.H., Genewein, T., Nürnberger, A., Köhler, J.M.: The power of ensembles for active learning in image classification. In: CVPR, pp. 9368–9377 (2018)Google Scholar
  5. 5.
    Borgwardt, K.M., Gretton, A., Rasch, M.J., Kriegel, H.P., Schölkopf, B., Smola, A.J.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), 49–57 (2006)CrossRefGoogle Scholar
  6. 6.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3(1), 1–122 (2011)zbMATHGoogle Scholar
  7. 7.
    Chattopadhyay, R., Wang, Z., Fan, W., Davidson, I., Panchanathan, S., Ye, J.: Batch mode active sampling based on marginal probability distribution matching. TKDD 7(3), 13 (2013)CrossRefGoogle Scholar
  8. 8.
    Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from national university of Singapore. In: Proceedings of the ACM International Conference on Image and Video Retrieval, p. 48 (2009)Google Scholar
  9. 9.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)Google Scholar
  10. 10.
    Duan, Y., Lu, J., Wang, Z., Feng, J., Zhou, J.: Learning deep binary descriptor with multi-quantization. In: CVPR, pp. 1183–1192 (2017)Google Scholar
  11. 11.
    Duan, Y., Wang, Z., Lu, J., Lin, X., Zhou, J.: GraphBit: bitwise interaction mining via deep reinforcement learning. In: CVPR, pp. 8270–8279 (2018)Google Scholar
  12. 12.
    Erin Liong, V., Lu, J., Wang, G., Moulin, P., Zhou, J.: Deep hashing for compact binary codes learning. In: CVPR, pp. 2475–2483 (2015)Google Scholar
  13. 13.
    Freytag, A., Rodner, E., Denzler, J.: Selecting influential examples: active learning with expected model output changes. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 562–577. Springer, Cham (2014). Scholar
  14. 14.
    Gal, Y., Islam, R., Ghahramani, Z.: Deep Bayesian active learning with image data. In: ICML, pp. 1183–1192 (2017)Google Scholar
  15. 15.
    Ghasedi Dizaji, K., Zheng, F., Sadoughi, N., Yang, Y., Deng, C., Huang, H.: Unsupervised deep generative adversarial hashing network. In: CVPR, pp. 3664–3673 (2018)Google Scholar
  16. 16.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)Google Scholar
  17. 17.
    Gordo, A., Almazán, J., Revaud, J., Larlus, D.: Deep image retrieval: learning global representations for image search. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 241–257. Springer, Cham (2016). Scholar
  18. 18.
    Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. JMLR 13(Mar), 723–773 (2012)Google Scholar
  19. 19.
    Hasan, M., Roy-Chowdhury, A.K.: Context aware active learning of activity recognition models. In: ICCV, pp. 4543–4551 (2015)Google Scholar
  20. 20.
    Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. TPAMI 33(1), 117–128 (2010)CrossRefGoogle Scholar
  21. 21.
    Johnson, J., et al.: Image retrieval using scene graphs. In: CVPR, pp. 3668–3678 (2015)Google Scholar
  22. 22.
    Joshi, A.J., Porikli, F., Papanikolopoulos, N.: Multi-class active learning for image classification. In: CVPR, pp. 2372–2379 (2009)Google Scholar
  23. 23.
    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report (2009)Google Scholar
  24. 24.
    Lai, H., Pan, Y., Liu, Y., Yan, S.: Simultaneous feature learning and hash coding with deep neural networks. In: CVPR, pp. 3270–3278 (2015)Google Scholar
  25. 25.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  26. 26.
    Li, W.J., Wang, S., Kang, W.C.: Feature learning based deep supervised hashing with pairwise labels. In: IJCAI, pp. 1711–1717 (2016)Google Scholar
  27. 27.
    Li, X., Guo, Y.: Multi-level adaptive active learning for scene classification. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 234–249. Springer, Cham (2014). Scholar
  28. 28.
    Liu, B., Ferrari, V.: Active learning for human pose estimation. In: ICCV, pp. 4363–4372 (2017)Google Scholar
  29. 29.
    Liu, H., Wang, R., Shan, S., Chen, X.: Deep supervised hashing for fast image retrieval. In: CVPR, pp. 2064–2072 (2016)Google Scholar
  30. 30.
    Luo, W., Schwing, A., Urtasun, R.: Latent structured active learning. In: NIPS, pp. 728–736 (2013)Google Scholar
  31. 31.
    Mac Aodha, O., Campbell, N.D., Kautz, J., Brostow, G.J.: Hierarchical subquery evaluation for active learning on a graph. In: CVPR, pp. 564–571 (2014)Google Scholar
  32. 32.
    Melville, P., Mooney, R.J.: Diverse ensembles for active learning. In: ICML, p. 74 (2004)Google Scholar
  33. 33.
    Nguyen, H.T., Smeulders, A.: Active learning using pre-clustering. In: ICML, p. 79 (2004)Google Scholar
  34. 34.
    Paul, S., Bappy, J.H., Roy-Chowdhury, A.K.: Non-uniform subset selection for active learning in structured data. In: CVPR, pp. 6846–6855 (2017)Google Scholar
  35. 35.
    Pidhorskyi, S., Jones, Q., Motiian, S., Adjeroh, D., Doretto, G.: Deep supervised hashing with spherical embedding. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11364, pp. 417–434. Springer, Cham (2019). Scholar
  36. 36.
    Qin, D., Gammeter, S., Bossard, L., Quack, T., Van Gool, L.: Hello neighbor: accurate object retrieval with k-reciprocal nearest neighbors. In: CVPR, pp. 777–784 (2011)Google Scholar
  37. 37.
    Rényi, A., et al.: On measures of entropy and information. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics (1961)Google Scholar
  38. 38.
    Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. arXiv preprint arXiv:1708.00489 (2017)
  39. 39.
    Settles, B., Craven, M.: An analysis of active learning strategies for sequence labeling tasks. In: EMNLP, pp. 1070–1079 (2008)Google Scholar
  40. 40.
    Settles, B., Craven, M., Ray, S.: Multiple-instance active learning. In: NIPS, pp. 1289–1296 (2008)Google Scholar
  41. 41.
    Shen, F., Shen, C., Liu, W., Tao Shen, H.: Supervised discrete hashing. In: CVPR, pp. 37–45 (2015)Google Scholar
  42. 42.
    Shen, F., Xu, Y., Liu, L., Yang, Y., Huang, Z., Shen, H.T.: Unsupervised deep hashing with similarity-adaptive and discrete optimization. TPAMI 40(12), 3034–3044 (2018)CrossRefGoogle Scholar
  43. 43.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  44. 44.
    Vasisht, D., Damianou, A., Varma, M., Kapoor, A.: Active learning for sparse Bayesian multilabel classification. In: KDD, pp. 472–481 (2014)Google Scholar
  45. 45.
    Vijayanarasimhan, S., Grauman, K.: Large-scale live active learning: training object detectors with crawled data and crowds. IJCV 108(1–2), 97–114 (2014). Scholar
  46. 46.
    Wang, G., Hu, Q., Cheng, J., Hou, Z.: Semi-supervised generative adversarial hashing for image retrieval. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 491–507. Springer, Cham (2018). Scholar
  47. 47.
    Wang, J., Kumar, S., Chang, S.F.: Semi-supervised hashing for large-scale search. TPAMI 34(12), 2393–2406 (2012)CrossRefGoogle Scholar
  48. 48.
    Wang, Q., Si, L., Zhang, Z., Zhang, N.: Active hashing with joint data example and tag selection. In: SIGIR, pp. 405–414 (2014)Google Scholar
  49. 49.
    Wang, X., Yang, M., Cour, T., Zhu, S., Yu, K., Han, T.X.: Contextual weighting for vocabulary tree based image retrieval. In: ICCV, pp. 209–216 (2011)Google Scholar
  50. 50.
    Yang, H.F., Lin, K., Chen, C.S.: Supervised learning of semantics-preserving hash via deep convolutional neural networks. TPAMI 40(2), 437–451 (2017)CrossRefGoogle Scholar
  51. 51.
    Zhang, J., Peng, Y.: SSDH: semi-supervised deep hashing for large scale image retrieval. TCSVT 29(1), 212–225 (2017)MathSciNetGoogle Scholar
  52. 52.
    Zhang, S., Li, J., Zhang, B.: Pairwise teacher-student network for semi-supervised hashing. In: CVPR, pp. 0–0 (2019)Google Scholar
  53. 53.
    Zhao, F., Huang, Y., Wang, L., Tan, T.: Deep semantic ranking based hashing for multi-label image retrieval. In: CVPR, pp. 1556–1564 (2015)Google Scholar
  54. 54.
    Zhen, Y., Yeung, D.Y.: Active hashing and its application to image and text retrieval. Data Min. Knowl. Disc. 26(2), 255–274 (2013). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of AutomationTsinghua UniversityBeijingChina
  2. 2.State Key Lab of Intelligent Technologies and SystemsBeijingChina
  3. 3.Beijing National Research Center for Information Science and TechnologyBeijingChina
  4. 4.Tsinghua Shenzhen International Graduate SchoolTsinghua UniversityShenzhenChina

Personalised recommendations