Hardening against adversarial examples with the smooth gradient method
- 152 Downloads
Commonly used methods in deep learning do not utilise transformations of the residual gradient available at the inputs to update the representation in the dataset. It has been shown that this residual gradient, which can be interpreted as the first-order gradient of the input sensitivity at a particular point, may be used to improve generalisation in feed-forward neural networks, including fully connected and convolutional layers. We explore how these input gradients are related to input perturbations used to generate adversarial examples and how the networks that are trained with this technique are more robust to attacks generated with the fast gradient sign method.
The equipment for these experiments was funded by a Grant from NVIDIA Corporation. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GTX Titan X GPUs used for this research.
Compliance with ethical standards
Conflicts of interest
George D. Magoulas has received research grants from NVIDIA Corporation. Alan Mosca owns stock in Alphabet, Facebook, NVIDIA and Twitter.
This article does not contain any studies with human participants or animals performed by any of the authors.
- Anastasiadis AD, Magoulas GD, Vrahatis MN (2003) An efficient improvement of the rprop algorithm. In: Proceedings of the First International Workshop on Artificial Neural Networks in Pattern Recognition (IAPR 2003), University of Florence, Italy, p 197Google Scholar
- Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv:1412.6572
- Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: International joint conference on neural networks, 1989, IJCNN, IEEE, pp 593–605Google Scholar
- Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv:1503.02531
- Igel C, Hüsken M (2000) Improving the Rprop learning algorithm. In: Proceedings of the second international ICSC symposium on neural computation (NC 2000), vol 2000. Citeseer, pp 115–121Google Scholar
- Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
- Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
- Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. ThesisGoogle Scholar
- LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. Handb Brain Theory Neural Netw 3361:310Google Scholar
- Lecun Y, Cortes C (1998) The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/. Accessed 3 Sep 2016
- Mosca A, Magoulas GD (2015) Adapting resilient propagation for deep learning. In: UK workshop on computational intelligenceGoogle Scholar
- Mosca A, Magoulas G (2016) Deep incremental boosting. In: Benzmuller C, Sutcliffe G, Rojas R (eds) GCAI 2016. 2nd global conference on artificial intelligence, EPiC series in computing, vol 41. EasyChair, pp 293–302Google Scholar
- Mosca A, Magoulas GD (2017a) Learning input features representations in deep learning. In: Advances in computational intelligence systems. Springer International Publishing, pp 433–445Google Scholar
- Mosca A, Magoulas GD (2017b) Training convolutional networks with weight-wise adaptive learning rates. In: ESANN 2017 proceedings, european symposium on artificial neural networks, computational intelligence and machine learning. Bruges (Belgium), 26–28 April 2017. http://i6doc.com
- Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2016a) Practical black-box attacks against deep learning systems using adversarial examples. arXiv:1602.02697
- Papernot N, McDaniel P, Wu X, Jha S, Swami A (2016b) Distillation as a defense to adversarial perturbations against deep neural networks. In: Security and privacy (SP), 2016 IEEE Symposium on, IEEE, pp 582–597Google Scholar
- Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: the rprop algorithm. In: Proceeding of the IEEE international conference on neural networks, IEEE, pp 586–591Google Scholar
- Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. http://research.microsoft.com/apps/pubs/default.aspx?id=68920
- Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv:1412.6806
- Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013) Intriguing properties of neural networks. arXiv:1312.6199