Targeted Attack for Deep Hashing Based Retrieval

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12346)


The deep hashing based retrieval method is widely adopted in large-scale image and video retrieval. However, there is little investigation on its security. In this paper, we propose a novel method, dubbed deep hashing targeted attack (DHTA), to study the targeted attack on such retrieval. Specifically, we first formulate the targeted attack as a point-to-set optimization, which minimizes the average distance between the hash code of an adversarial example and those of a set of objects with the target label. Then we design a novel component-voting scheme to obtain an anchor code as the representative of the set of hash codes of objects with the target label, whose optimality guarantee is also theoretically derived. To balance the performance and perceptibility, we propose to minimize the Hamming distance between the hash code of the adversarial example and the anchor code under the \(\ell ^\infty \) restriction on the perturbation. Extensive experiments verify that DHTA is effective in attacking both deep hashing based image retrieval and video retrieval.


Targeted attack Deep hashing Adversarial attack Similarity retrieval 



This work is supported in part by the National Key Research and Development Program of China under Grant 2018YFB1800204, the National Natural Science Foundation of China under Grant 61771273, the R&D Program of Shenzhen under Grant JCYJ20180508152204044, the project “PCL Future Greater-Bay Area Network Facilities for Large-scale Experiments and Applications (LZC0019)”, the Natutal Sciences and Engineering Research Council of Canada under Grant RGPIN203035-16, and the Canada Research Chairs Program. We also thank vivo and Rejoice Sport Tech. co., LTD. for their GPUs.

Supplementary material

500725_1_En_36_MOESM1_ESM.pdf (1.5 mb)
Supplementary material 1 (pdf 1513 KB)


  1. 1.
    Bai, Y., Zeng, Y., Jiang, Y., Wang, Y., Xia, S.T., Guo, W.: Improving query efficiency of black-box adversarial attack. In: ECCV 2020 (2020)Google Scholar
  2. 2.
    Cao, Y., Long, M., Liu, B., Wang, J.: Deep Cauchy Hashing for hamming space retrieval. In: CVPR (2018)Google Scholar
  3. 3.
    Cao, Z., Long, M., Wang, J., Yu, P.S.: HashNet: deep learning to hash by continuation. In: ICCV (2017)Google Scholar
  4. 4.
    Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: IEEE S&P (2017)Google Scholar
  5. 5.
    Carlini, N., Wagner, D.: Audio adversarial examples: targeted attacks on speech-to-text. In: IEEE S&P Workshops (2018)Google Scholar
  6. 6.
    Chen, W., Zhang, Z., Hu, X., Wu, B.: Boosting decision-based black-box adversarial attacks with random sign flip. In: ECCV (2020)Google Scholar
  7. 7.
    Chen, Y., Lai, Z., Ding, Y., Lin, K., Wong, W.K.: Deep supervised hashing with anchor graph. In: CVPR (2019)Google Scholar
  8. 8.
    Chen, Z., Yuan, X., Lu, J., Tian, Q., Zhou, J.: Deep hashing via discrepancy minimization. In: CVPR (2018)Google Scholar
  9. 9.
    Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from national university of Singapore. In: ICMR (2009)Google Scholar
  10. 10.
    Dong, Y., et al.: Boosting adversarial attacks with momentum. In: CVPR (2018)Google Scholar
  11. 11.
    Dong, Y., et al.: Efficient decision-based black-box adversarial attacks on face recognition. In: CVPR (2019)Google Scholar
  12. 12.
    Duan, R., Ma, X., Wang, Y., Bailey, J., Qin, A.K., Yang, Y.: Adversarial camouflage: hiding physical-world attacks with natural styles. In: CVPR (2020)Google Scholar
  13. 13.
    Eykholt, K., et al.: Robust physical-world attacks on deep learning visual classification. In: CVPR (2018)Google Scholar
  14. 14.
    Fan, Y., Wu, B., Li, T., Zhang, Y., Li, M., Li, Z., Yang, Y.: Sparse adversarial attack via perturbation factorization. In: ECCV (2020)Google Scholar
  15. 15.
    Feng, Y., Chen, B., Dai, T., Xia, S.T.: Adversarial attack on deep product quantization network for image retrieval. In: AAAI (2020)Google Scholar
  16. 16.
    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (2015)Google Scholar
  17. 17.
    Gu, Y., Ma, C., Yang, J.: Supervised recurrent hashing for large scale video retrieval. In: ACM MM (2016)Google Scholar
  18. 18.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  19. 19.
    Hu, D., Nie, F., Li, X.: Deep binary reconstruction for cross-modal hashing. IEEE Trans. Multimedia 21(4), 973–985 (2018)CrossRefGoogle Scholar
  20. 20.
    Jhuang, H., Gall, J., Zuffi, S., Schmid, C., Black, M.J.: Towards understanding action recognition. In: ICCV (2013)Google Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NeurIPS, pp. 1097–1105 (2012)Google Scholar
  22. 22.
    Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial examples in the physical world. In: ICLR (2017)Google Scholar
  23. 23.
    Lai, H., Pan, Y., Liu, Y., Yan, S.: Simultaneous feature learning and hash coding with deep neural networks. In: CVPR (2015)Google Scholar
  24. 24.
    Li, C., Deng, C., Li, N., Liu, W., Gao, X., Tao, D.: Self-supervised adversarial hashing networks for cross-modal retrieval. In: CVPR (2018)Google Scholar
  25. 25.
    Li, J., Ji, R., Liu, H., Hong, X., Gao, Y., Tian, Q.: Universal perturbation attack against image retrieval. In: ICCV (2019)Google Scholar
  26. 26.
    Li, P., Wang, M., Cheng, J., Xu, C., Lu, H.: Spectral hashing with semantically consistent graph for image indexing. IEEE Trans. Multimedia 15(1), 141–152 (2012)CrossRefGoogle Scholar
  27. 27.
    Li, S., Chen, Z., Lu, J., Li, X., Zhou, J.: Neighborhood preserving hashing for scalable video retrieval. In: ICCV, pp. 8212–8221 (2019)Google Scholar
  28. 28.
    Liu, H., Wang, R., Shan, S., Chen, X.: Deep supervised hashing for fast image retrieval. In: CVPR (2016)Google Scholar
  29. 29.
    Liu, W., Wang, J., Ji, R., Jiang, Y.G., Chang, S.F.: Supervised hashing with kernels. In: CVPR (2012)Google Scholar
  30. 30.
    Ma, X., et al.: Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recogn. (2019)Google Scholar
  31. 31.
    Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: CVPR (2016)Google Scholar
  32. 32.
    Qin, Y., Carlini, N., Cottrell, G., Goodfellow, I., Raffel, C.: Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In: ICML (2019)Google Scholar
  33. 33.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Shen, F., Shen, C., Liu, W., Tao Shen, H.: Supervised discrete hashing. In: CVPR (2015)Google Scholar
  35. 35.
    Shen, F., Xu, Y., Liu, L., Yang, Y., Huang, Z., Shen, H.T.: Unsupervised deep hashing with similarity-adaptive and discrete optimization. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 3034–3044 (2018)CrossRefGoogle Scholar
  36. 36.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  37. 37.
    Song, J., Zhang, H., Li, X., Gao, L., Wang, M., Hong, R.: Self-supervised video hashing with hierarchical binary auto-encoder. IEEE Trans. Image Process. 27(7), 3210–3221 (2018)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Soomro, K., Zamir, A.R., Shah, M.: Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  39. 39.
    Szegedy, C., et al.: Intriguing properties of neural networks. In: ICLR (2014)Google Scholar
  40. 40.
    Tolias, G., Radenovic, F., Chum, O.: Targeted mismatch adversarial attack: query with a flower to retrieve the tower. In: ICCV (2019)Google Scholar
  41. 41.
    Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., McDaniel, P.: Ensemble adversarial training: attacks and defenses. In: ICLR (2018)Google Scholar
  42. 42.
    Wang, J., Zhang, T., Sebe, N., Shen, H.T., et al.: A survey on learning to hash. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 769–790 (2017)CrossRefGoogle Scholar
  43. 43.
    Wang, J., Liu, W., Kumar, S., Chang, S.F.: Learning to hash for indexing big data-a survey. Proc. IEEE 104(1), 34–57 (2015)CrossRefGoogle Scholar
  44. 44.
    Wu, D., Lin, Z., Li, B., Ye, M., Wang, W.: Deep supervised hashing for multi-label and large-scale image retrieval. In: ICMR (2017)Google Scholar
  45. 45.
    Wu, D., Wang, Y., Xia, S.T., Bailey, J., Ma, X.: Skip connections matter: on the transferability of adversarial examples generated with ResNets. In: ICLR (2020)Google Scholar
  46. 46.
    Wu, G., et al.: Unsupervised deep video hashing via balanced code for large-scale video retrieval. IEEE Trans. Image Process. 28(4), 1993–2007 (2018)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Xia, R., Pan, Y., Lai, H., Liu, C., Yan, S.: Supervised hashing for image retrieval via image representation learning. In: AAAI (2014)Google Scholar
  48. 48.
    Xu, Y., et al.: Exact adversarial attack to image captioning via structured output learning with latent variables. In: CVPR (2019) Google Scholar
  49. 49.
    Yan, X., Zhang, L., Li, W.J.: Semi-supervised deep hashing with a bipartite graph. In: IJCAI (2017)Google Scholar
  50. 50.
    Yang, E., Liu, T., Deng, C., Liu, W., Tao, D.: DistillHash: unsupervised deep hashing by distilling data pairs. In: CVPR (2019)Google Scholar
  51. 51.
    Yang, E., Liu, T., Deng, C., Tao, D.: Adversarial examples for hamming space search. IEEE Trans. Cybern. 50(4), 1473–1484 (2018)CrossRefGoogle Scholar
  52. 52.
    Yao, Z., Gholami, A., Xu, P., Keutzer, K., Mahoney, M.W.: Trust region based adversarial attack on neural networks. In: CVPR (2019)Google Scholar
  53. 53.
    Zhao, F., Huang, Y., Wang, L., Tan, T.: Deep semantic ranking based hashing for multi-label image retrieval. In: CVPR (2015)Google Scholar
  54. 54.
    Zuva, K., Zuva, T.: Evaluation of information retrieval systems. Int. J. Comput. Sci. Inf. Technol. 4(3), 35 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Tsinghua Shenzhen International Graduate SchoolTsinghua UniversityShenzhenChina
  2. 2.Peng Cheng LaboratoryPCL Research Center of Networks and CommunicationsShenzhenChina
  3. AI LabShenzhenChina
  4. 4.Department of Electrical and Computer EngineeringUniversity of WaterlooWaterlooCanada

Personalised recommendations