Skip to main content

Adversarial Attacks and Defenses in Images, Graphs and Text: A Review

Abstract

Deep neural networks (DNN) have achieved unprecedented success in numerous machine learning tasks in various domains. However, the existence of adversarial examples raises our concerns in adopting deep learning to safety-critical applications. As a result, we have witnessed increasing interests in studying attack and defense mechanisms for DNN models on different data types, such as images, graphs and text. Thus, it is necessary to provide a systematic and comprehensive overview of the main threats of attacks and the success of corresponding countermeasures. In this survey, we review the state of the art algorithms for generating adversarial examples and the countermeasures against adversarial examples, for three most popular data types, including images, graphs and text.

References

  1. [1]

    A. Krizhevsky, I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutiona neura networks In Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, pp. 1097–1105, 2012.

    Google Scholar 

  2. [2]

    K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE Las Vegas USA pp. 770–778, 2016 DOI 10.1109/CVPR.2016.90

    Google Scholar 

  3. [3]

    G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. R. Mohamed, N. Jaitly A. Senior, V. Vanhoucke P. Nguyen, T. N. Sainath, B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, vol. 29, no. 6, pp 82–97, 2012 DOI: 10.1109/MSP2012. 2205597.

    Google Scholar 

  4. [4]

    S. Hochreiter, J. Schmidhuber. Long short-term memory. Neural Computation, vo. 9, no 8, pp 1735–1780, 1997 DOI: 10.1162/neco.1997.9.8.1735.

    Google Scholar 

  5. [5]

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Dressche J. Schrittwieser I. Antonoglou V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N Kalchbrenner I. Sutskever T. Lillicrap M. Leach, K. Kavukcuoglu, T. Graepel, D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, vol. 529, no. 7587, pp. 484–489, 2016. DOI: 10.1038/nature16961.

    Google Scholar 

  6. [6]

    D. Cireçan, U. Meier, J. Masci, J. Schmidhuber. Multi-column deep neural network for traffic sign classification. Neural Networks, vol. 32, p. 333–338, 012. DO 10.1016/j.neunet.2012.02.023.

    Google Scholar 

  7. [7]

    T. N Kipf, M. Weling. Sem-supervsed cassification with graph convolutional networks. ArXiv: 1609.02907, 2016.

    Google Scholar 

  8. [8]

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus Intriguing properties of neural networks. ArXiv: 1312.6199, 2013.

    Google Scholar 

  9. [9]

    I. J. Goodfellow, J. Shlens, C. Szegedy. Explaining and harnessing adversarial examples. ArXiv: 1412.6572, 2014.

    Google Scholar 

  10. [10]

    D. Zügner, A. Akbarnejad, S. Günnemann. Adversarial attacks on neural networks for graph data}. In Proceedings of the 24th ACM SIGKDD International Conference ton Knowledge Discovery & Data Mining, ACM, London, UK, p. 2847–2856, 018. OI:.1145/3219819. 3220078.

  11. [11]

    J. Ebrahimi, A. Y. Rao, D. Lowd, D. J. Dou. HotFlip: White-box adversaral exampes for text cassification ArXiv: 1712.06751, 2017.

    Google Scholar 

  12. [12]

    N. Papernot, P. McDaniel, X. Wu, S. Jha, A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of IEEE Symposium on Security and Privacy, IEEE San Jose USA pp. 582–597, 2016. DOI: 10.1109/SP.2016.41.

    Google Scholar 

  13. [13]

    A. Athalye, N. Carlini, D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. ArXiv: 1802.00420, 2018.

    Google Scholar 

  14. [14]

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv: 1706.06083, 2017.

    Google Scholar 

  15. [15]

    A. Kurakn, I. Goodfelow, S. Bengo. Adversaral examples in the physical world. ArXiv: 1607.02533, 2016.

    Google Scholar 

  16. [16]

    N. Carlini, D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ACM, Dallas, USA, pp. 3–14, 2017. DOI: 10.1145/3128572.3140444.

    Google Scholar 

  17. [17]

    W. L. Xu, D. Evans, Y. J. Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. ArXiv: 1704.01155, 2017.

    Google Scholar 

  18. [18]

    A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, A. Madry. Adversarial examples are not bugs, they are features. ArXiv: 1905.02175, 2019.

    Google Scholar 

  19. [19]

    B. Bggio, B. Neson P. Lakov. Posonng atacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, Omnipress, Edinburgh, UK, 2012.

    Google Scholar 

  20. [20]

    K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. W. Xiao, A. Prakash, T. Kohno, D. Song. Robust physical-world attacks on deep learning models. ArXiv: 1707.08945, 2017.

    Google Scholar 

  21. [21]

    F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. Ensemble adversarial training: Attacks and defenses. ArXiv: 1705.07204, 2017.

    Google Scholar 

  22. [22]

    B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, F. Roli. Evasion attacks against machine learning at test time. In Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases, Springer Prague Czech Repubic, pp. 387–402, 2013. DOI: 10.1007/978-3-642-40994-325.

    Google Scholar 

  23. [23]

    M. Barreno, B. Nelson, A. D. Joseph, J. D. Tygar. The security of machne earning Machine Learning, vo. 81, no. 2, pp. 121–148, 2010. DOI: 10.1007/s10994-010-5188-5.

    MathSciNet  Google Scholar 

  24. [24]

    N. Dalvi, P. Domingos, Mausam, S. Sanghai, D. Verma. Adversarial cassification. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM eattle, USA, pp. 99–108, 2004. DOI: 10.1145/1014052.1014066.

    Google Scholar 

  25. [25]

    D. Tspras, S. Santurkar, L. Engstrom, A. Turner, A. Madry. Robustness may be at odds with accuracy. ArXiv: 1805.12152, 2018.

    Google Scholar 

  26. [26]

    D. Su, H. Zhang, H. G. Chen, J. F. Yi, P. Y. Chen, Y. P. Gao. Is robustness the cost of accuracy? - A comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 644–661, 2018. DOI: 10.1007/978-3-030-01258-839.

    Google Scholar 

  27. [27]

    D. Stutz, M. Hein, B. Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the 32nd IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Piscataway, USA, pp. 6976–6987, 2019.

    Google Scholar 

  28. [28]

    H. Y. Zhang Y. D. Yu, J. T. Jiao E. P. Xing, L. El Ghaoui, M. I. Jordan. Theoretically principled trade-off between robustness and accuracy, ArXv: 190108573, 2019.

    Google Scholar 

  29. [29]

    J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, F. F. Li. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE Miam, USA pp. 248–255, 2009. DOI: 10.1109/CVPR.2009.5206848.

  30. [30]

    D. C. Liu, J. Nocedal. On the limted memory BFGS method for large scale optimization. Mathematical Programming, vol. 45, no. 1-3, p. 503–528, 1989. DO: 10.1007/BF01589116.

    MathSciNet  MATH  Google Scholar 

  31. [31]

    A. Kurakin, I. Goodfellow, S. Bengio. Adversarial machine learning at scale. ArXiv: 1611.01236, 2016.

    Google Scholar 

  32. [32]

    S. M. Moosavi-Dezfooli, A. Fawzi, P. Frossard. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 2574–2582, 2016. DOI: 10.1109/CVPR.2016.282.

    Google Scholar 

  33. [33]

    N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, A. Swami. The limitations of deep learning in adversarial settings. In Proceedings of IEEE European Symposium on Security and Privacy, IEEE Saarbrucken Germany, p. 372–387, 2016. DO: 0.1109/EuroSP. 2016.36.

    Google Scholar 

  34. [34]

    N. Carlini, D. Wagner. Towards evaluating the robustness of neural networks. In Proceedings of IEEE Symposium on Security and Privacy, EEE, San Jose USA pp. 39–57, 2017. DOI: 10.1109/SP.2017.49.

    Google Scholar 

  35. [35]

    N. Carlini, G. Katz, C. Barrett, D. L. Dill. Provably minimally-distorted adversarial examples. ArXiv: 1709.10207, 2017.

    Google Scholar 

  36. [36]

    G. Kaz, C. Barett, D. L. Dill, K. Juian M. J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Proceedings of the 29th International Conference on Computer Aided Verification, Springer, Heidelberg, Germany pp. 97–117, 2017. DOI: 10.1007/978-3-319-63387-9_5.

    Google Scholar 

  37. [37]

    V. Tjeng, K. Xiao, R. Tedrake. Evaluating robustness of neural networks with mixed integer programming. ArXiv: 1711.07356, 2017.

    Google Scholar 

  38. [38]

    K. Y. Xiao, V. Tjeng, N. M. Shafiullah, A. Madry. Training for faster adversarial robustness verification via inducing ReLU stability. ArXiv: 1809.03008, 2018.

    Google Scholar 

  39. [39]

    J. W. Su, D. V. Vargas, K. Sakurai. One pixel attack for fooling deep neura networks IEEE Transactions on Evolutionary Computation, vol 23, no. 5, pp 828–841, 2019. DOI: 10.1109/TEVC.2019.2890858.

    Google Scholar 

  40. [40]

    P. Y. Chen, Y. Sharma, H. Zhang, J. F. Yi, C. J. Hsieh. EAD: Elasticnet attacks to deep neural networks via adversarial exampes In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018.

    Google Scholar 

  41. [41]

    Y. Sharma, P. Y. Chen. Attacking the madry defense model with L1-based dversarial xamples. ArXiv: 1710.10733, 2017.

  42. [42]

    S. M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, P. Frossard. Universal adversaral perturbatons In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 86–94, 2017. DOI: 10.1109/CVPR.2017.17.

    Google Scholar 

  43. [43]

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. A. Ma, Z. H. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, F. F. Li. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, vol. 115, no. 3, p. 211–252, 15. DOI: 10.1007/s11263-015-0816-y.

  44. [44]

    C. W. Xiao, J. Y. Zhu, B. Li, W. He, M. Y. Liu, D. Song. Spatially ransformed dversarial xamples. ArXiv: 1801.02612, 2018.

    Google Scholar 

  45. [45]

    Y. Song, R. Shu, N. Kushman, S. Ermon. Constructing unrestricted adversarial examples with generative models. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Mntréal, Canada, pp. 8312–8323, 2018.

  46. [46]

    A. Odena, C. Olah, J. Shlens. Conditional image synthesis with auxiliary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 2642–2651, 2017.

    Google Scholar 

  47. [47]

    A. Athalye, L. Engstrom, A. Ilyas, K. Kwok. Synthesizing robust adversarial examples. ArXiv: 1707.07397, 2017.

    Google Scholar 

  48. [48]

    N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, A. Swami. Practical black-box attacks against machine learning. In Proceedings of ACM on Asia Conference on Computer and Communications Security, ACM, Abu Dhab, Unted Arab Emrates pp. 506–519 2017 DOI: 10.1145/3052973.3053009.

    Google Scholar 

  49. [49]

    Y. P. Dong, F. Z. Liao, T. Y. Pang, H. Su, J. Zhu, X. L. Hu, J. G. Li. Boosting adversaral attacks with momentum. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake Cty USA pp. 9185–9193 2018 DOI: 101109/ CVPR.2018.00957.

    Google Scholar 

  50. [50]

    P. Y. Chen, H. Zhang, Y. Sharma, J. F. Yi, C. J. Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ACM Dallas, USA, pp. 15–26, 2017. DOI: 10.1145/3128572.3140448.

    Google Scholar 

  51. [51]

    A. Ilyas, L. Engstrom, A. Athalye, J. Lin. Black-box adversarial attacks with limited queries and information. ArXiv: 1804.08598, 2018.

    Google Scholar 

  52. [52]

    D. Werstra T. Schau, T. Glasmachers, Y. Sun, J. Peters, J. Schmdhuber. Natural evouton strateges Natural evolution strategies. Journal of Machine Learning Research, vol. 15, no. 1, pp. 949–980, 2014.

    Google Scholar 

  53. [53]

    M. Alzantot, Y. Sharma, S. Chakraborty, M. Srivastava. Genattack: Practical black-box attacks with gradient-free optimization. ArXiv: 1805.11090, 2018.

    Google Scholar 

  54. [54]

    C. W. Xiao, B. Li, J. Y. Zhu, W. He, M. Y. Liu, D. Song. Generating adversarial examples with adversarial networks. ArXiv: 1801.02610, 2018.

    Google Scholar 

  55. [55]

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 2672–2680, 2014.

    Google Scholar 

  56. [56]

    D. Deb, J. B. Zhang, A. K. Jain. Advfaces: Adversarial face synthesis. ArXiv: 1908.05008, 2019.

    Google Scholar 

  57. [57]

    G. Cauwenberghs, T. Poggio. Incremental and decre- mental support vector machine learning. In Proceedings of the 13th International Conference on Neural Information Processing Systems, MIT Press, Denver, USA, pp. 388–394, 2000.

    Google Scholar 

  58. [58]

    P. W. Koh, P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 1885–1894, 2017.

    Google Scholar 

  59. [59]

    A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, T. Goldstein. Poison frogs! Targeted clean- label poisoning attacks on neural networks. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montreal, Canada, pp. 6103–6113, 2018.

    Google Scholar 

  60. [60]

    G. Hinton, O. Vinyals, J. Dean. Distilling the knowledge in a neural network. ArXiv: 1503.02531, 2015.

    Google Scholar 

  61. [61]

    J. Buckman, A. Roy, C. Raffel, I. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

    Google Scholar 

  62. [62]

    C. Guo, M. Rana, M. Cisse, L. van der Maaten. Countering adversarial images using input transformations. ArXiv: 1711.00117, 2017.

    Google Scholar 

  63. [63]

    V. K. Ha, J. C. Ren, X. Y. Xu, S. Zhao, G. Xie, V. M. Vargas. Deep learning based single image super-resolution: A survey. In Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, Springer, Xi'an, China, pp. 106–119, 2018. DOI: 10.1007/978-3-030-00563-4_11.

    Google Scholar 

  64. [64]

    G. S. Dhillon, K. Azizzadenesheli, Z. C. Lipton, J. Bernstein, J. Kossaifi, A. Khanna, A. Anandkumar. Stochastic activation pruning for robust adversarial defense. ArXiv: 1803.01442, 2018.

    Google Scholar 

  65. [65]

    C. H. Xie, J. Y. Wang, Z. S. Zhang, Z. Ren, A. Yuille. Mitigating adversarial effects through randomization. ArXiv: 1711.01991, 2017.

    Google Scholar 

  66. [66]

    Y. Song, T. Kim, S. Nowozin, S. Ermon, N. Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. ArXiv: 1710.10766, 2017.

    Google Scholar 

  67. [67]

    P. Samangouei, M. Kabkab, R. Chellappa. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. ArXiv: 1805.06605, 2018.

    Google Scholar 

  68. [68]

    A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu. Conditional image generation with PixelCNN decoders. In Proceedings of the 30th Conference on Neural Information Processing Systems, Curran Associates Inc., Barcelona, Spain, pp. 4790–4798, 2016.

    Google Scholar 

  69. [69]

    M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, N. Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 854–863, 2017.

    Google Scholar 

  70. [70]

    T. Miyato, S. I. Maeda, M. Koyama, K. Nakae, S. Ishii. Distributional smoothing with virtual adversarial training. ArXiv: 1507.00677, 2015.

    Google Scholar 

  71. [71]

    S. X. Gu, L. Rigazio. Towards deep neural network architectures robust to adversarial examples. ArXiv: 1412.5068, 2014.

    Google Scholar 

  72. [72]

    S. Rifai, P. Vincent, X. Muller, X. Glorot, Y. Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, Omnipress, Bellevue, USA, pp. 833–840, 2011.

    Google Scholar 

  73. [73]

    S. Ioffe, C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv: 1502.03167, 2015.

    Google Scholar 

  74. [74]

    A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, T. Goldstein. Adversarial training for free. ArXiv: 1904.12843, 2019.

    Google Scholar 

  75. [75]

    D. H. Zhang, T. Y. Zhang, Y. P. Lu, Z. X. Zhu, B. Dong. You only propagate once: Accelerating adversarial training via maximal principle. ArXiv: 1905.00877, 2019.

    Google Scholar 

  76. [76]

    L. S. Pontryagin. Mathematical Theory of Optimal Processes, London, UK: Routledge, 2018.

    Google Scholar 

  77. [77]

    A. Raghunathan, J. Steinhardt, P. Liang. Certified defenses against adversarial examples. ArXiv: 1801.09344, 2018.

    Google Scholar 

  78. [78]

    E. Wong, J. Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. ArXiv: 1711.00851, 2017.

    Google Scholar 

  79. [79]

    M. Hein, M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, USA, pp. 2266–2276, 2017.

    Google Scholar 

  80. [80]

    L. Vandenberghe, S. Boyd. Semidefinite programming. Semidefinite programming. SIAM Review, vol. 38, no. 1, pp. 49–95, 1996. DOI: 10.1137/1038003.

    MathSciNet  MATH  Google Scholar 

  81. [81]

    A. Raghunathan, J. Steinhardt, P. S. Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montreal, Canada, pp. 10877–10887, 2018.

    Google Scholar 

  82. [82]

    E. Wong, F. Schmidt, J. H. Metzen, J. Z. Kolter. Scaling provable adversarial defenses. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montreal, Canada, pp. 8400–8409, 2018.

    Google Scholar 

  83. [83]

    A. Sinha, H. Namkoong, J. Duchi. Certifying some distributional robustness with principled adversarial training. ArXiv: 1710.10571, 2017.

    Google Scholar 

  84. [84]

    K. Grosse, P. Manoharan, N. Papernot, M. Backes, P. McDaniel. On the (statistical) detection of adversarial examples. ArXiv: 1702.06280, 2017.

    Google Scholar 

  85. [85]

    Z. T. Gong, W. L. Wang, W. S. Ku. Adversarial and clean data are not twins. ArXiv: 1704.04960, 2017.

    Google Scholar 

  86. [86]

    J. H. Metzen, T. Genewein, V. Fischer, B. Bischoff. On detecting adversarial perturbations. ArXiv: 1702.04267, 2017.

    Google Scholar 

  87. [87]

    D. Hendrycks, K. Gimpel. Early methods for detecting adversarial images. ArXiv: 1608.00530, 2016.

    Google Scholar 

  88. [88]

    A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scholkopf, A. Smola. A kernel two-sample test. A kernel two-sample test. Journal of Machine Learning Research, vol. 13, pp. 723–773, 2012.

    MATH  Google Scholar 

  89. [89]

    R. Feinman, R. R. Curtin, S. Shintre, A. B. Gardner. Detecting adversarial samples from artifacts. ArXiv: 1703.00410, 2017.

    Google Scholar 

  90. [90]

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.

    MathSciNet  MATH  Google Scholar 

  91. [91]

    Y. Sharma, P. Y. Chen. Bypassing feature squeezing by increasing adversary strength. ArXiv: 1803.09868, 2018.

    Google Scholar 

  92. [92]

    A. Fawzi, S. M. Moosavi-Dezfooli, P. Frossard. Robustness of classifiers: From adversarial to random noise. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 1632–1640, 2016.

    Google Scholar 

  93. [93]

    S. M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, P. Frossard, S. Soatto. Analysis of universal adversarial perturbations. ArXiv: 1705.09554, 2017.

    Google Scholar 

  94. [94]

    A. Fawzi, O. Fawzi, P. Frossard. Analysis of classifiers' robustness to adversarial perturbations. Machine Learning, vol. 107, no. 3, pp. 481–508, 2018. DOI: 10.1007/ s10994-017-5663-3.

    MathSciNet  MATH  Google Scholar 

  95. [95]

    A. Shafahi, W. R. Huang, C. Studer, S. Feizi, T. Goldstein. Are adversarial examples inevitable. ArXiv: 1809.02104, 2018.

    Google Scholar 

  96. [96]

    L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, A. Madry. Adversarially robust generalization requires more data. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montreal, Canada, pp. 5014–5026, 2018.

    Google Scholar 

  97. [97]

    H. J. Dai, H. Li, T. Tian, X. Huang, L. Wang, J. Zhu, L. Song. Adversarial attack on graph structured data. ArXiv: 1806.02371, 2018.

    Google Scholar 

  98. [98]

    Y. Ma, S. H. Wang, T. Derr, L. F. Wu, J. L. Tang. Attacking graph convolutional networks via rewiring. ArXiv: 1906.03750, 2019.

    Google Scholar 

  99. [99]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. Playing Atari with deep reinforcement learning. ArXiv: 1312.5602, 2013.

    Google Scholar 

  100. [100]

    D. Zuugner, S. Gunnemann. Adversarial attacks on graph neural networks via meta learning. ArXiv: 1902.08412, 2019.

    Google Scholar 

  101. [101]

    C. Finn, P. Abbeel, S. Levine. Model-agnostic metalearning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, JMLR.org, Sydney, Australia, pp. 1126–1135, 2017.

    Google Scholar 

  102. [102]

    A. Bojchevski, S. Gunnemann. Adversarial attacks on node embeddings via graph poisoning. ArXiv: 1809.01093, 2018.

    Google Scholar 

  103. [103]

    B. Perozzi, R. Al-Rfou, S. Skiena. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, USA, pp. 701–710, 2014. DOI: 10.1145/2623330.2623732.

    Google Scholar 

  104. [104]

    F. L. Feng, X. N. He, J. Tang, T. S. Chua. Graph adversarial training: Dynamically regularizing based on graph structure. ArXiv: 1902.08226, 2019.

    Google Scholar 

  105. [105]

    K. D. Xu, H. G. Chen, S. J. Liu, P. Y. Chen, T. W. Weng, M. Y. Hong, X. Lin. Topology attack and defense for graph neural networks: An optimization perspective. ArXiv: 1906.04214, 2019.

    Google Scholar 

  106. [106]

    N. Carlini, D. Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 1–7, 2018. DOI: 10.1109/SPW.2018.00009.

    Google Scholar 

  107. [107]

    A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, A. Y. Ng. Deep speech: Scaling up end-to-end speech recognition. ArXiv: 1412.5567, 2014.

    Google Scholar 

  108. [108]

    T. Miyato, A. M. Dai, I. Goodfellow. Adversarial training methods for semi-supervised text classification. ArXiv: 1605.07725, 2016.

    Google Scholar 

  109. [109]

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, pp. 3111–3119, 2013.

    Google Scholar 

  110. [110]

    B. Liang, H. C. Li, M. Q. Su, P. Bian, X. R. Li, W. C. Shi. Deep text classification can be fooled. ArXiv: 1704.08006, 2017.

    Google Scholar 

  111. [111]

    J. Gao, J. Lanchantin, M. L. Soffa, Y. J. Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 50–56, 2018. DOI: 10.1109/SPW.2018.00016.

    Google Scholar 

  112. [112]

    J. F. Li, S. L. Ji, T. Y. Du, B. Li, T. Wang. Textbugger: Generating adversarial text against real-world applications. ArXiv: 1812.05271, 2018.

    Google Scholar 

  113. [113]

    S. Samanta, S. Mehta. Towards crafting text adversarial samples. ArXiv: 1707.02812, 2017.

    Google Scholar 

  114. [114]

    M. Iyyer, J. Wieting, K. Gimpel, L. Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. ArXiv: 1804.06059, 2018.

    Google Scholar 

  115. [115]

    Q. Lei, L. F. Wu, P. Y. Chen, A. G. Dimakis, I. S. Dhillon, M. Witbrock. Discrete attacks and submodular optimization with applications to text classification. ArXiv: 1812.00151, 2018.

    Google Scholar 

  116. [116]

    R. Jia, P. Liang. Adversarial examples for evaluating reading comprehension systems. ArXiv: 1707.07328, 2017.

    Google Scholar 

  117. [117]

    Y. Belinkov, Y. Bisk. Synthetic and natural noise both break neural machine translation. ArXiv: 1711.02173, 2017.

    Google Scholar 

  118. [118]

    M. H. Cheng, J. F. Yi, H. Zhang, P. Y. Chen, C. J. Hsieh. Seq2Sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. ArXiv: 1803.01128, 2018.

    Google Scholar 

  119. [119]

    T. Niu, M. Bansal. Adversarial over-sensitivity and overstability strategies for dialogue models. ArXiv: 1809.02079, 2018.

  120. [120]

    T. X. He, J. Glass. Detecting egregious responses in neural sequence-to-sequence models. ArXiv: 1809.04113, 2018.

    Google Scholar 

  121. [121]

    H. C. Liu, T. Derr, Z. T. Liu, J. L. Tang. Say what I want: Towards the dark side of neural dialogue models. ArXiv: 1909.06044, 2019.

    Google Scholar 

  122. [122]

    M. Sharif, S. Bhagavatula, L. Bauer, M. K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of- the-art face recognition. In Proceedings of the ACM SIG- SAC Conference on Computer and Communications Security, ACM, Vienna, Austria, pp. 1528–1540, 2016. DOI: 10.1145/2976749.2978392.

    Google Scholar 

  123. [123]

    O. M. Parkhi, A. Vedaldi, A. Zisserman. Deep face recognition. Machine Learning, 2015.

    Google Scholar 

  124. [124]

    C. H. Xie, J. Y. Wang, Z. S. Zhang, Y. Y. Zhou, L. X. Xie, A. Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 1378–1387, 2017. DOI: 10.1109/ICCV.2017. 153.

    Google Scholar 

  125. [125]

    J. H. Metzen, M. C. Kumar, T. Brox, V. Fischer. Universal adversarial perturbations against semantic image segmentation. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 2774–2783, 2017. DOI: 10.1109/ICCV.2017.300.

    Google Scholar 

  126. [126]

    S. S. Li, A. Neupane, S. Paul, C. Y. Song, S. V. Krishnamurthy, A. K. R. Chowdhury, A. Swami. Adversarial perturbations against real-time video classification systems. ArXiv: 1807.00458, 2018.

    Google Scholar 

  127. [127]

    J. Kos, I. Fischer, D. Song. Adversarial examples for generative models. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 36–42, DOI: 10.1109/SPW.2018.00014.

  128. [128]

    D. P. Kingma, M. Welling. Auto-encoding variational Bayes. ArXiv: 1312.6114, 2013.

    Google Scholar 

  129. [129]

    A. B. L. Larsen, S. K. Sønderby, H. Larochelle, O. Winther. Autoencoding beyond pixels using a learned similarity metric. ArXiv: 1512.09300, 2015.

    Google Scholar 

  130. [130]

    K. Grosse, N. Papernot, P. Manoharan, M. Backes, P. McDaniel. Adversarial perturbations against deep neural networks for malware classification. ArXiv: 1606.04435, 2016.

    Google Scholar 

  131. [131]

    D. Arp, M. Spreitzenbarth, H. Gascon, K. Rieck. DREBIN: Effective and explainable detection of android malware in your pocket. In Proceedings of Symposium Network Distributed System Security, Internet Society, San Diego, USA, 2014.

    Google Scholar 

  132. [132]

    W. W. Hu, Y. Tan. Generating adversarial malware examples for black-box attacks based on GAN. ArXiv: 1702.05983, 2017.

    Google Scholar 

  133. [133]

    H. S. Anderson, J. Woodbridge, B. Filar. DeepDGA: Ad- versarially-tuned domain generation and detection. In Proceedings of ACM Workshop on Artificial Intelligence and Security, ACM, Vienna, Austria, pp. 13–21, 2016. DOI: 10.1145/2996758.2996767.

    Google Scholar 

  134. [134]

    T. Chugh, A. K. Jain. Fingerprint presentation attack detection: Generalization and efficiency. ArXiv: 1812.11574, 2018.

    Google Scholar 

  135. [135]

    T. Chugh, K. Cao, A. K. Jain. Fingerprint spoof buster: Use of minutiae-centered patches. IEEE Transactions on Information Forensics and Security, vol. 13, no. 9, pp. 2190–2202, 2018. DOI: 10.1109/TIFS.2018.2812193.

    Google Scholar 

  136. [136]

    S. Huang, N. Papernot, I. Goodfellow, Y. Duan, P. Abbeel. Adversarial attacks on neural network policies. ArXiv: 1702.02284, 2017.

    Google Scholar 

  137. [137]

    J. Schulman, S. Levine, P. Moritz, M. I. Jordan, P. Abbeel. Trust region policy optimization. In Proceedings of the 31st International Conference on Machine Learning, JMLR, Lille, France, pp. 1889–1897, 2015.

    Google Scholar 

  138. [138]

    V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International conference on Machine Learning, PMLR, New York, USA, pp. 1928–1937, 2016.

    Google Scholar 

Download references

Acknowledgements

This work was supported by National Science Foundation (NSF), USA (Nos. IIS-1845081 and CNS-1815636).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Han Xu.

Additional information

Recommended by Associate Editor Hong Qiao

Han Xu is a second year Ph. D. student of computer science in DSE Lab, Michigan State University, USA. He is under supervision by Dr. Ji-Liang Tang.

His research interests include deep learning safety and robustness, especially studying the problems related to adversarial examples.

Yao Ma received the B. Sc. degree in applied mathematics at Zhejiang University, China in 2015, the M. Sc. degree in statistics, probabilities and operation research at Eindhoven University of Technology, the Netherlands in 2016. He is now a Ph. D. degree candidate of Department of Computer Science and Engineering, Michigan State University, USA. His Ph. D. advisor is Dr. Jiliang Tang.

His research interests include graph neural networks and their related safety issues.

Hao-Chen Liu is currently a Ph. D. student at the Department of Computer Science and Engineering at Michigan State University, under the supervision of Dr. Jiliang Tang. He is a member of Data Science and Engineering (DSE) Lab.

His research interests include natural language processing problems, especially in the robustness, fairness of dialogue systems.

Debayan Deb is a Ph. D. degree candidate in the Biometrics Lab, Michigan State University, USA under the supervision of Dr. Anil K. Jain. Before joining the Biometrics Lab of MSU, he graduated from Michigan State University with a Bachelor Degree of Computer Science and Engineering.

His research interests include face recognition and computer vision tasks.

Hui Liu is a research associate at Michigan State University. Before joining MSU, she received her Ph. D. degree of Electrical Engineering in Southern Methodist University, USA under the supervision by Dr. Dinesh Rajen.

Her research interests include signal processing, wireless communication, and deep learning related topics.

Ji-Liang Tang is an assistant professor in the computer science and engineering department at Michigan State University since Fall 2016. Before that, he was a research scientist in Yahoo Research and received his Ph. D. degree from Arizona State University in 2015. He was the recipients of 2019 NSF Career Award, the 2015 KDD Best Dissertation runner up and 6 Best Paper Awards (or runner-ups) including WSDM 2018 and KDD 2016. He serves as conference organizers (e.g., KDD, WSDM and SDM) and journal editors (e.g., TKDD). He has published his research in highly ranked journals and top conference proceedings, which received thousands of citations and extensive media coverage.

His research interests include social computing, data mining and machine learning and their applications in education.

Anil K. Jain (Ph. D., 1973, Ohio State University; B. Tech., IIT Kanpur) is a University Distinguished Professor at Michigan State University where he conducts research in pattern recognition, machine learning, computer vision, and biometrics recognition. He was a member of the United States Defense Science Board and Forensics Science Standards Board. His prizes include Guggenheim, Humboldt, Fulbright, and King-Sun Fu Prize. For advancing pattern recognition, Jain was awarded Doctor Honoris Causa by Universidad Autónoma de Madrid. He was Editor-in-Chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence and is a Fellow of ACM, IEEE, AAAS, and SPIE. Jain has been assigned 8 U.S. and Korean patents and is active in technology transfer for which he was elected to the National Academy of Inventors. Jain is a member of the U.S. National Academy of Engineering (NAE), foreign member of the Indian National Academy of Engineering (INAE), a member of The World Academy of Science (TWAS) and a foreign member of the Chinese Academy of Sciences (CAS).

His research interests include pattern recognition, machine learning, computer vision, and biometrics recognition.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

To view a copy of this licence, visit http://creativecomm-ons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xu, H., Ma, Y., Liu, HC. et al. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. Int. J. Autom. Comput. 17, 151–178 (2020). https://doi.org/10.1007/s11633-019-1211-x

Download citation

Keywords

  • Adversarial example
  • model safety
  • robustness
  • defenses
  • deep learning