Abstract
Owing to the extensive application of infrared object detectors in the safety-critical tasks, it is necessary to evaluate their robustness against adversarial examples in the real world. However, current few physical infrared attacks are complicated to implement in practical application because of their complex transformation from the digital world to physical world. To address this issue, in this paper, we propose a physically feasible infrared attack method called “infrared adversarial patches”. Considering the imaging mechanism of infrared cameras by capturing objects’ thermal radiation, infrared adversarial patches conduct attacks by attaching a patch of thermal insulation materials on the target object to manipulate its thermal distribution. To enhance adversarial attacks, we present a novel aggregation regularization to guide the simultaneous learning for the patch’s shape and location on the target object. Thus, a simple gradient-based optimization can be adapted to solve for them. We verify infrared adversarial patches in different object detection tasks with various object detectors. Experimental results show that our method achieves more than 90% Attack Success Rate (ASR) versus the pedestrian detector and vehicle detector in the physical environment, where the objects are captured in different angles, distances, postures, and scenes. More importantly, infrared adversarial patch is easy to implement, and it only needs 0.5 h to be manufactured in the physical world, which verifies its effectiveness and efficiency. Another advantage of our infrared adversarial patches is the ability to extend to attack the visible object detector in the physical world. As a consequence, we can simultaneously perform the infrared and visible physical attacks by a unified adversarial patch, which shows the good generalization.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availibility
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Notes
We don’t compare with adversarial bulbs (Zhu et al., 2021) because they don’t report corresponding physical results under different angles and distances, and reproducing the physical attacks is difficult owing to its poor implementation.
We choose 0.5 as the threshold in the former experiments. When performing comparison under the physical world, because (Zhu et al., 2022) uses 0.7 as the threshold for the object detection model, to ensure the fair comparison, we change the threshold 0.5 to 0.7 for all three compared physical attacks.
We generated 750 adversarial examples and mixed them with 250 clean examples at a classical ratio 3:1 to finetune the trained detection model.
References
Abdelfattah, M., Yuan, K., Wang, Z. J., & Ward, R. (2021). Adversarial attacks on camera-lidar models for 3D car detection. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (vol. 2021, pp. 2189–2194).
Brown, T. B., Mané, D., Roy, A., Abadi, M., & Gilmer, J. (2017). Adversarial patch. arXiv preprint arXiv:1712.09665
Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60(2), 223–311.
Carlini, N., & Wagner, D. (2017). Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (sp) (pp. 39–57). IEEE
Chen, Z., Li, B., Wu, S., Xu, J., Ding, S., & Zhang, W. (2022). Shape matters: Deformable patch attack. In Proceedings of the 17th European conference on computer vision (ECCV), Part IV, Tel Aviv, Israel (pp. 529–548). Springer.
Edwards, D. M., & Rawat, D. B. (2020). Study of adversarial machine learning with infrared examples for surveillance applications. Electronics, 9(8), 1284.
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1625–1634).
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572
Goswami, G., Agarwal, A., Ratha, N., Singh, R., & Vatsa, M. (2019). Detecting and mitigating adversarial perturbations for robust face recognition. International Journal of Computer Vision, 127, 719–742.
Hayes, J. (2018). On visible adversarial perturbations & digital watermarking. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 1597–1604).
Holland, P. W., & Leinhardt, S. (1971). Transitivity in structural models of small groups. Comparative Group Studies, 2(2), 107–124.
Jia, J., Cao, X., Wang, B., & Gong, N. Z. (2019). Certified robustness for top-k predictions against adversarial perturbations via randomized smoothing. arXiv preprint arXiv:1912.09899
Jia, X., Zhu, C., Li, M., Tang, W., & Zhou, W. (2021). LLVIP: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE international conference on computer vision (pp. 3496–350).
Kazemi, E., Kerdreux, T., & Wang, L. (2023). Minimally distorted structured adversarial attacks. International Journal of Computer Vision, 131(1), 160–176.
Komkov, S., & Petiushko, A. (2021). Advhat: Real-world adversarial attack on arcface face id system. In 2020 25th international conference on pattern recognition (ICPR) (pp. 819–826). IEEE.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25
Naseer, M., Khan, S., & Porikli, F. (2019). Local gradients smoothing: Defense against localized adversarial attacks. In 2019 IEEE winter conference on applications of computer vision (WACV) (pp. 1300–1307). IEEE.
Osahor, U. M., & Nasrabadi, N. M. (2019). Deep adversarial attack on target detection systems. In Artificial intelligence and machine learning for multi-domain operations applications, (vol. 11006, p. 110061Q). International Society for Optics and Photonics.
Pautov, M., Melnikov, G., Kaziakhmedov, E., Kireev, K., & Petiushko, A. (2019). On adversarial patches: Real-world attack on arcface-100 face recognition system. In International multi-conference on engineering, computer and information sciences (pp. 0391–0396).
Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1), 145–151.
Qin, Z., Fan, Y., Zha, H., & Baoyuan, W. (2021). Random noise defense against query-based black-box attacks. Advances in Neural Information Processing Systems, 34, 7650–7663.
Razakarivony, S., & Jurie, F. (2016). Vehicle detection in aerial imagery: A small target detection benchmark. Journal of Visual Communication and Image Representation, 34, 187–203.
Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE CVPR (pp. 4510–4520).
Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In IEEE CVPR (pp. 815–823).
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016). Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of ACM SIGSAC conference on computer and communications security (pp. 1528–1540).
Song, D., Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Tramer, F., Prakash, A., & Kohno, T. (2018). Physical adversarial examples for object detectors. In 12th USENIX workshop on offensive technologies (WOOT 18)
Suard, F., Rakotomamonjy, A., Bensrhair, A., & Broggi, A. (2006). Pedestrian detection using infrared images and histograms of oriented gradients. In IEEE intelligent vehicles symposium (pp. 206–212).
Tu, J., Li, H., Yan, X., Ren, M., Chen, Y., Liang, M., Bitar, E., Yumer, E., & Urtasun, R. (2021). Exploring adversarial robustness of multi-sensor perception systems in self driving. arXiv preprint arXiv:2101.06784
Vollmer, M. (2021). Infrared thermal imaging. In Computer vision: A reference guide (pp. 666–670).
Wang, S., Wu, T., Chakrabarti, A., & Vorobeychik, Y. (2022). Adversarial robustness of deep sensor fusion models. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 2387–2396).
Watts, D. J., & Strogatz, S. H. (1998). Collective dynamics of ‘small-world’ networks. Nature, 393(6684), 440–442.
Wei, H., Wang, Z., Jia, X., Zheng, Y., Tang, H., Satoh, S., & Wang, Z. (2022). Hotcold block: Fooling thermal infrared detectors with a novel wearable design. arXiv preprint arXiv:2212.05709
Wei, X., Guo, Y., & Jie, Yu. (2022). Adversarial sticker: A stealthy attack method in the physical world. IEEE TPAMI, 45(3), 2711–2715.
Wei, X., Guo, Y., Yu, J., & Zhang, B. (2022). Simultaneously optimizing perturbations and positions for black-box adversarial patch attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence
Wei, X., Pu, B., Lu, J., & Wu, B. (2022). Physically adversarial attacks and defenses in computer vision: A survey. arXiv preprint: arXiv:2211.01671
Wei, X., Yan, H., & Li, B. (2022). Sparse black-box video attack with reinforcement learning. International Journal of Computer Vision, 130(6), 1459–1473.
Wei, X., Yu, J., & Huang, Y. (2023). Physically adversarial infrared patches with learnable shapes and locations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12334–12342).
Weng, Q. (2009). Thermal infrared remote sensing for urban climate and environmental studies: Methods, applications, and trends. ISPRS Journal of Photogrammetry and Remote Sensing, 64(4), 335–344.
Xiao, C., Li, B., Zhu, J.-Y., He, W., Liu, M., & Song, D. (2018). Generating adversarial examples with adversarial networks. arXiv:1801.02610
Xu, W., Evans, D., & Qi, Y. (2017). Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155
Zhang, H., & Wang, J. (2019). Towards adversarially robust object detection In Proceedings of the IEEE international conference on computer vision (pp. 421–430).
Zhu, X., Hu, Z., Huang, S., Li, J., & Hu, X. (2022). Infrared invisible clothing: Hiding from infrared detectors at multiple angles in real world. arXiv preprint arXiv:2205.05909
Zhu, X., Li, X., Li, J., Wang, Z., & Xiaolin, H. (2021). Fooling thermal infrared pedestrian detectors in real world using small bulbs. In Proceedings of the AAAI conference on artificial intelligence, (vol. 35, pp. 3616–3624).
Zhu, Y., Ma, C., & Jun, D. (2019). Rotated cascade r-cnn: A shape robust detector with coordinate regression. Pattern Recognition, 96, 106964.
Funding
This work was supported by the Project of the National Natural Science Foundation of China (No. 62076018), and the Fundamental Research Funds for the Central Universities.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Shin’ichi Satoh.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wei, X., Yu, J. & Huang, Y. Infrared Adversarial Patches with Learnable Shapes and Locations in the Physical World. Int J Comput Vis (2023). https://doi.org/10.1007/s11263-023-01963-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11263-023-01963-y