Skip to main content
SpringerLink
Log in

\(\ell _\infty \)-Robustness and Beyond: Unleashing Efficient Adversarial Training

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13671))

Included in the following conference series:

Abstract

Neural networks are vulnerable to adversarial attacks: adding well-crafted, imperceptible perturbations to their input can modify their output. Adversarial training is one of the most effective approaches in training robust models against such attacks. However, it is much slower than vanilla training of neural networks since it needs to construct adversarial examples for the entire training data at every iteration, hampering its effectiveness. Recently, Fast Adversarial Training (FAT) was proposed that can obtain robust models efficiently. However, the reasons behind its success are not fully understood, and more importantly, it can only train robust models for \(\ell _\infty \)-bounded attacks as it uses FGSM during training. In this paper, by leveraging the theory of coreset selection, we show how selecting a small subset of training data provides a general, more principled approach toward reducing the time complexity of robust training. Unlike existing methods, our approach can be adapted to a wide variety of training objectives, including TRADES, \(\ell _p\)-PGD, and Perceptual Adversarial Training (PAT). Our experimental results indicate that our approach speeds up adversarial training by 2–3 times while experiencing a slight reduction in the clean and robust accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that adversarial training in the literature generally refers to a particular approach proposed by Madry et al. [25]. For the purposes of this paper, we refer to any method that builds adversarial attacks around the training data and incorporates them into the training of the neural network as adversarial training. Using this taxonomy, methods such as TRADES [44], \(\ell _p\)-PGD [25] or Perceptual Adversarial Training (PAT) [22] are all considered different versions of adversarial training.

  2. 2.

    Our implementation can be found in this repository.

References

  1. Adadi, A.: A survey on data-efficient algorithms in big data era. J. Big Data 8(1), 1–54 (2021). https://doi.org/10.1186/s40537-021-00419-9

    Article  Google Scholar 

  2. Andriushchenko, M., Flammarion, N.: Understanding and improving fast adversarial training. In: Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS) (2020)

    Google Scholar 

  3. Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., Roli, F.: Evasion attacks against machine learning at test time. In: Blockeel, H., Kersting, K., Nijssen, S., Železný, F. (eds.) ECML PKDD 2013. LNCS (LNAI), vol. 8190, pp. 387–402. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40994-3_25

    Chapter  Google Scholar 

  4. Campbell, T., Broderick, T.: Bayesian coreset construction via greedy iterative geodesic ascent. In: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 697–705 (2018)

    Google Scholar 

  5. Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 2206–2216 (2020)

    Google Scholar 

  6. Danskin, J.M.: The Theory of Max-min and its Application to Weapons Allocation Problems, vol. 5. Springer Science & Business Media (1967). https://doi.org/10.1007/978-3-642-46092-0

  7. Elenberg, E.R., Khanna, R., Dimakis, A.G., Negahban, S.N.: Restricted strong convexity implies weak submodularity. CoRR abs/1612.00804 (2016)

    Google Scholar 

  8. Eykholt, K., et al.: Robust physical-world attacks on deep learning visual classification. In: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1625–1634 (2018)

    Google Scholar 

  9. Feldman, D.: Introduction to core-sets: an updated survey. CoRR abs/2011.09384 (2020)

    Google Scholar 

  10. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  11. Har-Peled, S., Mazumdar, S.: On coresets for k-means and k-median clustering. In: Proceedings of the 36th Annual ACM Symposium on Theory of Computing (STOC), pp. 291–300 (2004)

    Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  13. Kang, D., Sun, Y., Hendrycks, D., Brown, T., Steinhardt, J.: Testing robustness against unforeseen adversaries. CoRR abs/1908.08016 (2019)

    Google Scholar 

  14. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107–8116 (2020)

    Google Scholar 

  15. Katharopoulos, A., Fleuret, F.: Not all samples are created equal: Deep learning with importance sampling. In: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 2530–2539 (2018)

    Google Scholar 

  16. Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., De, A., Iyer, R.K.: GRAD-MATCH: gradient matching based data subset selection for efficient deep model training. In: Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 5464–5474 (2021)

    Google Scholar 

  17. Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., Iyer, R.K.: GLISTER: generalization based data subset selection for efficient and robust learning. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 8110–8118 (2021)

    Google Scholar 

  18. Kolter, Z., Madry, A.: Adversarial robustness: theory and practice. In: Tutorial in the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS) (2018). https://adversarial-ml-tutorial.org/

  19. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto (2009)

    Google Scholar 

  20. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems 25: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 1106–1114 (2012)

    Google Scholar 

  21. Laidlaw, C., Feizi, S.: Functional adversarial attacks. In: Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 10408–10418 (2019)

    Google Scholar 

  22. Laidlaw, C., Singla, S., Feizi, S.: Perceptual adversarial robustness: defense against unseen threat models. In: Proceedings of the 9th International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

  23. Liu, Y., Ma, X., Bailey, J., Lu, F.: Reflection backdoor: a natural backdoor attack on deep neural networks. In: Proceedings of the 16th European Conference on Computer Vision (ECCV), pp. 182–199 (2020). https://doi.org/10.1007/978-3-030-58607-2_11

  24. Ma, X., et al.: Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recogn. 110, 107332 (2021)

    Google Scholar 

  25. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  26. Minoux, M.: Accelerated greedy algorithms for maximizing submodular set functions. In: Optimization Techniques, pp. 234–243. Springer (1978). https://doi.org/10.1007/BFb0006528

  27. Mirzasoleiman, B., Bilmes, J.A., Leskovec, J.: Coresets for data-efficient training of machine learning models. In: Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 6950–6960 (2020)

    Google Scholar 

  28. Mirzasoleiman, B., Cao, K., Leskovec, J.: Coresets for robust training of deep neural networks against noisy labels. In: Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS) (2020)

    Google Scholar 

  29. Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions - I. Math. Program. 14(1), 265–294 (1978). https://doi.org/10.1007/BF01588971

    Article  MathSciNet  MATH  Google Scholar 

  30. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)

    Google Scholar 

  31. Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, vol. 1, pp. 40–44 (1993)

    Google Scholar 

  32. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  33. Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green AI. Commun. ACM, 63(12), 54–63 (2020)

    Google Scholar 

  34. Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. In: Korhonen, A., Traum, D.R., Màrquez, L. (eds.) Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pp. 3645–3650 (2019)

    Google Scholar 

  35. Szegedy, C., et al.: Intriguing properties of neural networks. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR) (2014)

    Google Scholar 

  36. Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I.J., Boneh, D., McDaniel, P.D.: Ensemble adversarial training: Attacks and defenses. In: Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  37. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy. In: Proceedings of the 7th International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  38. Vahdat, A., Kautz, J.: NVAE: a deep hierarchical variational autoencoder. In: Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS) (2020)

    Google Scholar 

  39. Wei, K., Iyer, R., Bilmes, J.: Submodularity in data subset selection and active learning. In: Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1954–1963 (2015)

    Google Scholar 

  40. Wolsey, L.A.: An analysis of the greedy algorithm for the submodular set covering problem. Combinatorica 2(4), 385–393 (1982). https://doi.org/10.1007/BF02579435

    Article  MathSciNet  MATH  Google Scholar 

  41. Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: revisiting adversarial training. In: Proceedings of the 8th International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  42. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  43. Xiao, C., Zhu, J., Li, B., He, W., Liu, M., Song, D.: Spatially transformed adversarial examples. In: Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  44. Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 7472–7482 (2019)

    Google Scholar 

  45. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018)

    Google Scholar 

Download references

Acknowledgements

This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200.

Author information

Authors and Affiliations

  1. School of Computing and Information Systems, The University of Melbourne, Parkville, VIC, Australia

    Hadi M. Dolatabadi, Sarah Erfani & Christopher Leckie

Authors
  1. Hadi M. Dolatabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Erfani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher Leckie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hadi M. Dolatabadi .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 860 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dolatabadi, H.M., Erfani, S., Leckie, C. (2022). \(\ell _\infty \)-Robustness and Beyond: Unleashing Efficient Adversarial Training. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13671. Springer, Cham. https://doi.org/10.1007/978-3-031-20083-0_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20083-0_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20082-3

  • Online ISBN: 978-3-031-20083-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation