Skip to main content
SpringerLink
Log in

Distortion diminishing with vulnerability filters pruning

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Overparameterization of convolutional neural networks allows model compression, and model pruning algorithms have attracted much interest due to their practical acceleration effects. The pruning algorithm should remove as many redundant structures as possible while maintaining the original accuracy to maximize the acceleration effect. However, prior works utilize benign samples to evaluate the loss of accuracy but ignore the vulnerability to malicious adversarial samples, which brings potential security risks. To tackle this limitation, we propose the distortion diminishing with vulnerability filters pruning (DD-VFP) method that simultaneously improves the compressed model’s classification accuracy and adversarial defense capabilities. Specifically, we define adversarial vulnerability for filters by computing the distortion of their latent features. We then propose a new pruning framework, vulnerability filter pruning (VFP), to remove filters with higher adversarial vulnerability during training. We propose a new loss for adversarial training, distortion diminishing (DD) loss, which directly suppresses the model’s dependence on non-robust features by reducing the latent feature distortion under adversarial perturbation. Experiments on multiple benchmark datasets prove the effectiveness of our method, which not only achieves state-of-the-art adversarial defense capabilities but also removes more model parameters. An interesting phenomenon is that although the DD-VFP method slightly loses accuracy after pruning, the trade-off between accuracy and adversarial defense capabilities is significantly reduced, which proves that our method successfully removes non-robust features.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing, pp. 1106–1114 (2012)

  2. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) Proc. of ICLR (2015)

  3. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  4. Girshick, R.: Fast r-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)

  5. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

  6. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

  7. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

  8. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  9. Li, H., Shen, C.: Interactive color image segmentation with linear programming. Mach. Vis. Appl. 21(4), 403–412 (2010)

    Article  Google Scholar 

  10. Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. ArXiv preprint (2015)

  11. Luo, J.-H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)

  12. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R.: Intriguing properties of neural networks. In: Bengio, Y., LeCun, Y. (eds.) Proc. of ICLR (2014)

  13. Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., Roli, F.: Evasion attacks against machine learning at test time. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 387–402. Springer, Berlin (2013)

  14. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Bengio, Y., LeCun, Y. (eds.) Proc. of ICLR (2015)

  15. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: Proc. of ICLR (2018)

  16. Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International Conference on Machine Learning, pp. 7472–7482. PMLR (2019)

  17. Xie, C., Yuille, A.L.: Intriguing properties of adversarial training at scale. In: Proc. of ICLR (2020)

  18. Wu, B., Chen, J., Cai, D., He, X., Gu, Q.: Do wider neural networks really help adversarial robustness? In: Advances in Neural Information Processing Systems, vol. 34 (2021)

  19. Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

  20. Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In: Proc. of ICLR (2019)

  21. Wang, H., Wu, X., Huang, Z., Xing, E.P.: High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684–8694 (2020)

  22. Liu, Z., Liu, Q., Liu, T., Xu, N., Lin, X., Wang, Y., Wen, W.: Feature distillation: DNN-oriented jpeg compression against adversarial examples. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 860–868. IEEE (2019)

  23. Raff, E., Sylvester, J., Forsyth, S., McLean, M.: Barrage of random transforms for adversarially robust defense. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6528–6537 (2019)

  24. Akhtar, N., Liu, J., Mian, A.: Defense against universal adversarial perturbations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3389–3398 (2018)

  25. Liu, J., Zhang, W., Zhang, Y., Hou, D., Liu, Y., Zha, H., Yu, N.: Detection based defense against adversarial examples from the steganalysis point of view. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4825–4834 (2019)

  26. Raghunathan, A., Steinhardt, J., Liang, P.: Certified defenses against adversarial examples. In: Proc. of ICLR (2018)

  27. Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing. In: International Conference on Machine Learning, pp. 1310–1320. PMLR (2019)

  28. Tao, L., Feng, L., Yi, J., Huang, S.-J., Chen, S.: Better safe than sorry: Preventing delusive adversaries with adversarial training. In: Advances in Neural Information Processing Systems, vol. 34 (2021)

  29. Geiping, J., Fowl, L.H., Somepalli, G., Goldblum, M., Moeller, M., Goldstein, T.: What doesn’t kill you makes you robust (er): how to adversarially train against data poisoning (2021)

  30. Yang, H., Zhang, J., Dong, H., Inkawhich, N., Gardner, A., Touchet, A., Wilkes, W., Berry, H., Li, H.: Dverge: diversifying vulnerabilities for enhanced robust generation of ensembles. Adv. Neural Inf. Process. Syst. 33, 5505–5515 (2020)

    Google Scholar 

  31. Pang, T., Xu, K., Du, C., Chen, N., Zhu, J.: Improving adversarial robustness via promoting ensemble diversity. In: International Conference on Machine Learning, pp. 4970–4979. PMLR (2019)

  32. Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I.J., Boneh, D., McDaniel, P.D.: Ensemble adversarial training: attacks and defenses. In: Proc. of ICLR (2018)

  33. Pang, T., Yang, X., Dong, Y., Xu, K., Zhu, J., Su, H.: Boosting adversarial training with hypersphere embedding. Adv. Neural Inf. Process. Syst. 33, 7779–7792 (2020)

    Google Scholar 

  34. Dong, Y., Deng, Z., Pang, T., Zhu, J., Su, H.: Adversarial distributional training for robust deep learning. Adv. Neural Inf. Process. Syst. 33, 8270–8283 (2020)

    Google Scholar 

  35. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

  36. He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. In: Lang, J. (ed.) Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13–19, 2018, Stockholm, Sweden, pp. 2234–2240 (2018)

  37. He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)

  38. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. In: Proc. of ICLR (2017)

  39. Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. In: Proc. of ICLR (2017)

  40. He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017)

  41. Dhillon, G.S., Azizzadenesheli, K., Lipton, Z.C., Bernstein, J., Kossaifi, J., Khanna, A., Anandkumar, A.: Stochastic activation pruning for robust adversarial defense. In: Proc. of ICLR (2018)

  42. Ye, S., Xu, K., Liu, S., Cheng, H., Lambrechts, J.-H., Zhang, H., Zhou, A., Ma, K., Wang, Y., Lin, X.: Adversarial robustness vs. model compression, or both? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 111–120 (2019)

  43. Madaan, D., Shin, J., Hwang, S.J.: Adversarial neural pruning with latent vulnerability suppression. In: International Conference on Machine Learning, pp. 6575–6585. PMLR (2020)

  44. Xie, H., Qian, L., Xiang, X., Liu, N.: Blind adversarial pruning: towards the comprehensive robust models with gradually pruning against blind adversarial attacks. In: 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2021)

  45. Tao, L., Feng, L., Yi, J., Chen, S.: With false friends like these, who can notice mistakes? In: Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22–March 1, 2022, pp. 8458–8466 (2022)

  46. Pang, T., Yang, X., Dong, Y., Su, H., Zhu, J.: Bag of tricks for adversarial training. In: Proc. of ICLR (2021)

  47. Li, C., Wang, G., Wang, B., Liang, X., Li, Z., Chang, X.: Dynamic slimmable network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8607–8617 (2021)

  48. Hou, Z., Qin, M., Sun, F., Ma, X., Yuan, K., Xu, Y., Chen, Y.-K., Jin, R., Xie, Y., Kung, S.-Y.: Chex: channel exploration for CNN model compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12287–12298 (2022)

  49. Dong, Y., Fu, Q.-A., Yang, X., Pang, T., Su, H., Xiao, Z., Zhu, J.: Benchmarking adversarial robustness on image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 321–331 (2020)

  50. Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. In: Proc. of ICLR (2019)

  51. Gao, R., Cai, T., Li, H., Hsieh, C.-J., Wang, L., Lee, J.D.: Convergence of adversarial training in overparametrized neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

  52. Rice, L., Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: International Conference on Machine Learning, pp. 8093–8104. PMLR (2020)

  53. Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: revisiting adversarial training. In: Proc. of ICLR (2020)

  54. Li, T., Wu, Y., Chen, S., Fang, K., Huang, X.: Subspace adversarial training. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18–24, 2022, pp. 13399–13408 (2022)

Download references

Funding

This work was supported by the National Natural Science Foundation of China (Grant No.61375021), NUAA Fundamental Research Funds for the Central Universities (Grant No.3082020NZ2020017) and Natural Science Foundation of Jiangsu Province (Grant BK20222012).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Hengyi Huang, Pingfan Wu, Shifeng Xia & Ningzhong Liu

  2. MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, 211106, China

    Hengyi Huang, Pingfan Wu, Shifeng Xia & Ningzhong Liu

Authors
  1. Hengyi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pingfan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shifeng Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ningzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ningzhong Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, H., Wu, P., Xia, S. et al. Distortion diminishing with vulnerability filters pruning. Machine Vision and Applications 34, 123 (2023). https://doi.org/10.1007/s00138-023-01468-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01468-1

Keywords

Search

Navigation