Adversarial Ranking Attack and Defense

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12359)


Deep Neural Network (DNN) classifiers are vulnerable to adversarial attack, where an imperceptible perturbation could result in misclassification. However, the vulnerability of DNN-based image ranking systems remains under-explored. In this paper, we propose two attacks against deep ranking systems, i.e., Candidate Attack and Query Attack, that can raise or lower the rank of chosen candidates by adversarial perturbations. Specifically, the expected ranking order is first represented as a set of inequalities, and then a triplet-like objective function is designed to obtain the optimal perturbation. Conversely, a defense method is also proposed to improve the ranking system robustness, which can mitigate all the proposed attacks simultaneously. Our adversarial ranking attacks and defense are evaluated on datasets including MNIST, Fashion-MNIST, and Stanford-Online-Products. Experimental results demonstrate that a typical deep ranking system can be effectively compromised by our attacks. Meanwhile, the system robustness can be moderately improved with our defense. Furthermore, the transferable and universal properties of our adversary illustrate the possibility of realistic black-box attack.



This work was supported partly by National Key R&D Program of China Grant 2018AAA0101400, NSFC Grants 61629301, 61773312, 61976171, and 61672402, China Postdoctoral Science Foundation Grant 2019M653642, Young Elite Scientists Sponsorship Program by CAST Grant 2018QNRC001, and Natural Science Foundation of Shaanxi Grant 2020JQ-069.

Supplementary material

504468_1_En_46_MOESM1_ESM.pdf (24.7 mb)
Supplementary material 1 (pdf 25247 KB)


  1. 1.
    Athalye, A., Carlini, N.: On the robustness of the CVPR 2018 white-box adversarial example defenses. arXiv preprint arXiv:1804.03286 (2018)
  2. 2.
    Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 (2018)
  3. 3.
    Bui, T., Ribeiro, L., Ponti, M., Collomosse, J.: Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. CVIU 164, 27–37 (2017)Google Scholar
  4. 4.
    Carlini, N., Wagner, D.: Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv:1607.04311 (2016)
  5. 5.
    Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. IEEE (2017)Google Scholar
  6. 6.
    Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. JMLR 11, 1109–1135 (2010)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Chen, J., Jordan, M.I.: Boundary attack++: Query-efficient decision-based adversarial attack. arXiv preprint arXiv:1904.02144 (2019)
  8. 8.
    Chen, P.Y., Sharma, Y., Zhang, H., Yi, J., Hsieh, C.J.: EAD: elastic-net attacks to deep neural networks via adversarial examples. In: AAAI (2018)Google Scholar
  9. 9.
    Croce, F., Hein, M.: Sparse and imperceivable adversarial attacks. In: ICCV, pp. 4724–4732 (2019)Google Scholar
  10. 10.
    Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., Li, J.: Boosting adversarial attacks with momentum. In: CVPR (June 2018)Google Scholar
  11. 11.
    Dong, Y., Pang, T., Su, H., Zhu, J.: Evading defenses to transferable adversarial examples by translation-invariant attacks. In: CVPR, pp. 4312–4321 (2019)Google Scholar
  12. 12.
    Dong, Y., et al.: Efficient decision-based black-box adversarial attacks on face recognition. In: CVPR, pp. 7714–7722 (2019)Google Scholar
  13. 13.
    Dong, Y., Su, H., Zhu, J., Bao, F.: Towards interpretable deep neural networks by leveraging adversarial examples. arXiv preprint arXiv:1708.05493 (2017)
  14. 14.
    Dubey, A., van der Maaten, L., Yalniz, Z., Li, Y., Mahajan, D.: Defense against adversarial images using web-scale nearest-neighbor search. In: CVPR, pp. 8767–8776 (2019)Google Scholar
  15. 15.
    Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: VSE++: Improved visual-semantic embeddings, vol. 2, no. 7, p. 8. arXiv preprint arXiv:1707.05612 (2017)
  16. 16.
    Ganeshan, A., Babu, R.V.: FDA: feature disruptive attack. In: ICCV, pp. 8069–8079 (2019)Google Scholar
  17. 17.
    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  18. 18.
    Gopinath, D., Katz, G., Pasareanu, C.S., Barrett, C.: DeepSafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 (2017)
  19. 19.
    Goren, G., Kurland, O., Tennenholtz, M., Raiber, F.: Ranking robustness under adversarial document manipulations. In: ACM SIGIR, pp. 395–404. ACM (2018)Google Scholar
  20. 20.
    Guo, C., Rana, M., Cisse, M., Van Der Maaten, L.: Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117 (2017)
  21. 21.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (June 2016)Google Scholar
  22. 22.
    He, W., Wei, J., Chen, X., Carlini, N., Song, D.: Adversarial example defense: ensembles of weak defenses are not strong. In: 11th USENIX Workshop on Offensive Technologies, WOOT 2017 (2017)Google Scholar
  23. 23.
    He, X., He, Z., Du, X., Chua, T.S.: Adversarial personalized ranking for recommendation. In: ACM SIGIR, pp. 355–364. ACM (2018)Google Scholar
  24. 24.
    Huang, Q., et al.: Intermediate level adversarial attack for enhanced transferability. arXiv preprint arXiv:1811.08458 (2018)
  25. 25.
    Huang, R., Xu, B., Schuurmans, D., Szepesvári, C.: Learning with a strong adversary. CoRR abs/1511.03034 (2015).
  26. 26.
    Jacob, P., Picard, D., Histace, A., Klein, E.: Metric learning with horde: high-order regularizer for deep embeddings. In: ICCV, pp. 6539–6548 (2019)Google Scholar
  27. 27.
    Joachims, T.: Optimizing search engines using clickthrough data. In: ACM SIGKDD, pp. 133–142. ACM (2002)Google Scholar
  28. 28.
    Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). Scholar
  29. 29.
    Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 (2014)
  30. 30.
    Komkov, S., Petiushko, A.: AdvHat: Real-world adversarial attack on ArcFace Face ID system. arXiv preprint arXiv:1908.08705 (2019)
  31. 31.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NeurIPS, pp. 1097–1105 (2012)Google Scholar
  32. 32.
    Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016)
  33. 33.
    Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 (2016)
  34. 34.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  35. 35.
    Lee, K.-H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 212–228. Springer, Cham (2018). Scholar
  36. 36.
    Li, J., Ji, R., Liu, H., Hong, X., Gao, Y., Tian, Q.: Universal perturbation attack against image retrieval. In: ICCV, pp. 4899–4908 (2019)Google Scholar
  37. 37.
    Liu, H., et al.: Universal adversarial perturbation via prior driven uncertainty approximation. In: ICCV, pp. 2941–2949 (2019)Google Scholar
  38. 38.
    Liu, T.Y., et al.: Learning to rank for information retrieval. Found. Trends® Inf. Retr. 3(3), 225–331 (2009)CrossRefGoogle Scholar
  39. 39.
    Liu, X., Cheng, M., Zhang, H., Hsieh, C.-J.: Towards robust neural networks via random self-ensemble. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 381–397. Springer, Cham (2018). Scholar
  40. 40.
    Liu, X., Li, Y., Wu, C., Hsieh, C.J.: Adv-BNN: Improved adversarial defense through robust Bayesian neural network. arXiv preprint arXiv:1810.01279 (2018)
  41. 41.
    Liu, Y., Chen, X., Liu, C., Song, D.: Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770 (2016)
  42. 42.
    Liu, Z., Zhao, Z., Larson, M.: Who’s afraid of adversarial queries?: the impact of image modifications on content-based image retrieval. In: ICMR, pp. 306–314. ACM (2019)Google Scholar
  43. 43.
    Lu, J., Issaranon, T., Forsyth, D.: SafetyNet: detecting and rejecting adversarial examples robustly. In: ICCV, pp. 446–454 (2017)Google Scholar
  44. 44.
    Luo, Y., Boix, X., Roig, G., Poggio, T., Zhao, Q.: Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292 (2015)
  45. 45.
    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
  46. 46.
    Mao, C., Zhong, Z., Yang, J., Vondrick, C., Ray, B.: Metric learning for adversarial robustness. In: NeurIPS, pp. 478–489 (2019)Google Scholar
  47. 47.
    Meng, D., Chen, H.: MagNet: a two-pronged defense against adversarial examples. In: ACM SIGSAC, pp. 135–147. ACM (2017)Google Scholar
  48. 48.
    Metzen, J.H., Genewein, T., Fischer, V., Bischoff, B.: On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267 (2017)
  49. 49.
    Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: CVPR, pp. 1765–1773 (2017)Google Scholar
  50. 50.
    Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: CVPR, pp. 2574–2582 (2016)Google Scholar
  51. 51.
    Mummadi, C.K., Brox, T., Metzen, J.H.: Defending against universal perturbations with shared adversarial training. In: ICCV, pp. 4928–4937 (2019)Google Scholar
  52. 52.
    Niu, Z., Zhou, M., Wang, L., Gao, X., Hua, G.: Hierarchical multimodal LSTM for dense visual-semantic embedding. In: ICCV, pp. 1881–1889 (2017)Google Scholar
  53. 53.
    Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: CVPR, pp. 4004–4012 (2016)Google Scholar
  54. 54.
    Papernot, N., McDaniel, P.: On the effectiveness of defensive distillation. arXiv preprint arXiv:1607.05113 (2016)
  55. 55.
    Papernot, N., McDaniel, P., Goodfellow, I.: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 (2016)
  56. 56.
    Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519. ACM (2017)Google Scholar
  57. 57.
    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387. IEEE (2016)Google Scholar
  58. 58.
    Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 582–597. IEEE (2016)Google Scholar
  59. 59.
    Paszke, A., et al.: Automatic differentiation in PyTorch (2017)Google Scholar
  60. 60.
    Prakash, A., Moran, N., Garber, S., DiLillo, A., Storer, J.: Deflecting adversarial attacks with pixel deflection. In: CVPR, pp. 8571–8580 (2018)Google Scholar
  61. 61.
    Sabour, S., Cao, Y., Faghri, F., Fleet, D.J.: Adversarial manipulation of deep representations. arXiv preprint arXiv:1511.05122 (2015)
  62. 62.
    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR, pp. 815–823 (2015)Google Scholar
  63. 63.
    Shaham, U., Yamada, Y., Negahban, S.: Understanding adversarial training: increasing local stability of supervised models through robust optimization. Neurocomputing 307, 195–204 (2018)CrossRefGoogle Scholar
  64. 64.
    Sharif, M., Bhagavatula, S., Bauer, L., Reiter, M.K.: Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In: ACM SIGSAC, pp. 1528–1540. ACM (2016)Google Scholar
  65. 65.
    Shi, Y., Wang, S., Han, Y.: Curls & Whey: boosting black-box adversarial attacks. arXiv preprint arXiv:1904.01160 (2019)
  66. 66.
    Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput. 23, 828–841 (2019)CrossRefGoogle Scholar
  67. 67.
    Szegedy, C., et al.: Going deeper with convolutions. In: CVPR, pp. 1–9 (2015)Google Scholar
  68. 68.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR, pp. 2818–2826 (2016)Google Scholar
  69. 69.
    Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
  70. 70.
    Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: CVPR, pp. 1386–1393 (2014)Google Scholar
  71. 71.
    Wang, J., Zhang, H.: Bilateral adversarial training: towards fast training of more robust models against adversarial attacks. In: ICCV, pp. 6629–6638 (2019)Google Scholar
  72. 72.
    Wang, Z., Zheng, S., Song, M., Wang, Q., Rahimpour, A., Qi, H.: advPattern: physical-world attacks on deep person re-identification via adversarially transformable patterns. In: ICCV, pp. 8341–8350 (2019)Google Scholar
  73. 73.
    Wu, L., Zhu, Z., Tai, C., et al.: Understanding and enhancing the transferability of adversarial examples. arXiv preprint arXiv:1802.09707 (2018)
  74. 74.
    Xiao, C., Zhu, J.Y., Li, B., He, W., Liu, M., Song, D.: Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612 (2018)
  75. 75.
    Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
  76. 76.
    Xie, C., et al.: Improving transferability of adversarial examples with input diversity. In: CVPR, pp. 2730–2739 (2019)Google Scholar
  77. 77.
    Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: attacks and defenses for deep learning. IEEE TNNLS 30, 2805–2824 (2019)MathSciNetGoogle Scholar
  78. 78.
    Zhong, Y., Deng, W.: Adversarial learning with margin-based triplet embedding regularization. In: ICCV, pp. 6549–6558 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Xi’an Jiaotong UniversityXi’anChina
  2. 2.Alibaba DAMO MIILHangzhouChina
  3. 3.HERE TechnologiesChicagoUSA
  4. 4.Wormpex AI ResearchBellevueUSA

Personalised recommendations