New Threats Against Object Detector with Non-local Block

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)


The introduction of non-local blocks to the traditional CNN architecture enhances its performance for various computer vision tasks by improving its capabilities of capturing long-range dependencies. However, the usage of non-local blocks may also introduce new threats to computer vision systems. Therefore, it is important to study the threats caused by non-local blocks before directly applying them on commercial systems. In this paper, two new threats named disappearing attack and appearing attack against object detectors with a non-local block are investigated. The former aims at misleading an object detector with a non-local block such that it is unable to detect a target object category while the latter aims at misleading the object detector such that it detects a predefined object category, which is not present in images. Different from the existing attacks against object detectors, these threats are able to be performed in long range cases. This means that the target object and the universal adversarial patches learned from the proposed algorithms can have long distance between them. To examine the threats, digital and physical experiments are conducted on Faster R-CNN with a non-local block and 6331 images from 56 videos. The experiments show that the universal patches are able to mislead the detector with greater probabilities. To explain the threats from non-local blocks, the reception fields of CNN models with and without non-local blocks are studied empirically and theoretically.


Non-local block Adversarial examples Object detection 



This work is partially supported by the Ministry of Education, Singapore through Academic Research Fund Tier 1, RG30/17.

Supplementary material

504476_1_En_29_MOESM1_ESM.pdf (2 mb)
Supplementary material 1 (pdf 2015 KB)


  1. 1.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 60–65. IEEE (2005)Google Scholar
  2. 2.
    Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: GCNet: non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)Google Scholar
  3. 3.
    Chen, L., Song, H., Li, Q., Cui, Y., Yang, J., Hu, X.T.: Liver segmentation in CT images using a non-local fully convolutional neural network. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 639–642. IEEE (2019)Google Scholar
  4. 4.
    Chen, S.T., Cornelius, C., Martin, J., Chau, D.H.: ShapeShifter: robust physical adversarial attack on faster R-CNN object detector. In: ECML/PKDD (2018)Google Scholar
  5. 5.
    Chi, L., Tian, G., Mu, Y., Xie, L., Tian, Q.: Fast non-local neural networks with spectral residual learning. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2142–2151 (2019)Google Scholar
  6. 6.
    Fu, C., et al.: Non-local recurrent neural memory for supervised sequence modeling. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6311–6320 (2019)Google Scholar
  7. 7.
    Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (2015).
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  9. 9.
    Hu, G., Cui, B., Yu, S.: Skeleton-based action recognition with synchronous local and non-local spatio-temporal learning and frequency attention. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 1216–1221. IEEE (2019)Google Scholar
  10. 10.
    Huang, Y., Kong, A.W.K., Lam, K.Y.: Adversarial signboard against object detector. In: Proceedings of the British Machine Vision Conference (BMVC) (2019)Google Scholar
  11. 11.
    Huang, Y., Kong, A.W.-K., Lam, K.-Y.: Attacking object detectors without changing the target object. In: Nayak, A.C., Sharma, A. (eds.) PRICAI 2019. LNCS (LNAI), vol. 11672, pp. 3–15. Springer, Cham (2019). Scholar
  12. 12.
    Lee, J., Kim, J.: Improving video captioning with non-local neural networks. In: 2018 IEEE International Conference on Consumer Electronics - Asia (ICCE-Asia), pp. 206–212 (2018)Google Scholar
  13. 13.
    Levi, H., Ullman, S.: Efficient coarse-to-fine non-local module for the detection of small objects. arXiv preprint arXiv:1811.12152 (2018)
  14. 14.
    Li, G., He, X., Zhang, W., Chang, H., Dong, L., Lin, L.: Non-locally enhanced encoder-decoder network for single image de-raining. arXiv preprint arXiv:1808.01491 (2018)
  15. 15.
    Li, Y., Tang, S., Ye, Y., Ma, J.: Spatial-aware non-local attention for fashion landmark detection. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 820–825. IEEE (2019)Google Scholar
  16. 16.
    Liao, X., He, L., Yang, Z., Zhang, C.: Video-based person re-identification via 3D convolutional networks and non-local attention. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 620–634. Springer, Cham (2019). Scholar
  17. 17.
    Lu, J., Sibai, H., Fabry, E.: Adversarial examples that fool detectors. CoRR abs/1712.02494 (2017)Google Scholar
  18. 18.
    Lu, J., Sibai, H., Fabry, E., Forsyth, D.A.: No need to worry about adversarial examples in object detection in autonomous vehicles. CoRR abs/1707.03501 (2017)Google Scholar
  19. 19.
    Lu, N., Yu, W., Qi, X., Chen, Y., Gong, P., Xiao, R.: MASTER: multi-aspect non-local network for scene text recognition. arXiv preprint arXiv:1910.02562 (2019)
  20. 20.
    Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 4898–4906 (2016)Google Scholar
  21. 21.
    Ma, Y., Liu, X., Bai, S., Wang, L., He, D., Liu, A.: Coarse-to-fine image inpainting via region-wise convolutions and non-local correlation. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3123–3129. AAAI Press (2019)Google Scholar
  22. 22.
    Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2574–2582 (2016)Google Scholar
  23. 23.
    Papernot, N., McDaniel, P.D., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387 (2016)Google Scholar
  24. 24.
    Shi, L., Zhang, Y., Cheng, J., Lu, H.: Non-local graph convolutional networks for skeleton-based action recognition. arXiv preprint arXiv:1805.07694 (2018)
  25. 25.
    Shokri, M., Harati, A., Taba, K.: Salient object detection in video using deep non-local neural networks. arXiv preprint arXiv:1810.07097 (2018)
  26. 26.
    Song, D., et al.: Physical adversarial examples for object detectors. In: 12th USENIX Workshop on Offensive Technologies (WOOT 2018) (2018)Google Scholar
  27. 27.
    Szegedy, C., et al.: Intriguing properties of neural networks. In: International Conference on Learning Representations (2014).
  28. 28.
    Tang, Y., Zhang, X., Wang, J., Chen, S., Ma, L., Jiang, Y.-G.: Non-local NetVLAD encoding for video classification. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11132, pp. 219–228. Springer, Cham (2019). Scholar
  29. 29.
    Tu, Z., Ma, Y., Li, C., Tang, J., Luo, B.: Edge-guided non-local fully convolutional network for salient object detection. arXiv preprint arXiv:1908.02460 (2019)
  30. 30.
    Wang, S., Hou, X., Zhao, X.: Automatic building extraction from high-resolution aerial imagery via fully convolutional encoder-decoder network with non-local block. IEEE Access 8, 7313–7322 (2020)CrossRefGoogle Scholar
  31. 31.
    Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)Google Scholar
  32. 32.
    Wang, Y., Seo, J., Jeon, T.: NL-LinkNet: toward lighter but more accurate road extraction with non-local operations. arXiv preprint arXiv:1908.08223 (2019)
  33. 33.
    Xia, B.N., Gong, Y., Zhang, Y., Poellabauer, C.: Second-order non-local attention networks for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3760–3769 (2019)Google Scholar
  34. 34.
    Xie, C., Wang, J., Zhang, Z., Zhou, Y., Xie, L., Yuille, A.L.: Adversarial examples for semantic segmentation and object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1378–1387 (2017)Google Scholar
  35. 35.
    Xu, X., Wang, J.: Extended non-local feature for visual saliency detection in low contrast images. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11132, pp. 580–592. Springer, Cham (2019). Scholar
  36. 36.
    Yue, K., Sun, M., Yuan, Y., Zhou, F., Ding, E., Xu, F.: Compact generalized non-local network. In: Advances in Neural Information Processing Systems, pp. 6510–6519 (2018)Google Scholar
  37. 37.
    Zajac, M., Zołna, K., Rostamzadeh, N., Pinheiro, P.O.: Adversarial framing for image and video classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 10077–10078 (2019)Google Scholar
  38. 38.
    Zhang, Y., Li, K., Li, K., Zhong, B., Fu, Y.: Residual non-local attention networks for image restoration. arXiv preprint arXiv:1903.10082 (2019)
  39. 39.
    Zhu, Z., Xu, M., Bai, S., Huang, T., Bai, X.: Asymmetric non-local neural networks for semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 593–602 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Nanyang Technological UniversitySingaporeSingapore

Personalised recommendations