Abstract
Commonly used methods in deep learning do not utilise transformations of the residual gradient available at the inputs to update the representation in the dataset. It has been shown that this residual gradient, which can be interpreted as the first-order gradient of the input sensitivity at a particular point, may be used to improve generalisation in feed-forward neural networks, including fully connected and convolutional layers. We explore how these input gradients are related to input perturbations used to generate adversarial examples and how the networks that are trained with this technique are more robust to attacks generated with the fast gradient sign method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In some cases, activation functions with a single non-differentiable point with no discontinuity have been shown to still work (Hahnloser et al. 2000).
In certain cases, a single network spent upwards of 24 hrs to train on a single GPU.
\(50\%\) grey indicates no update, whilst white indicates an update value of \(+\,1\) and black indicates an update value of \(-\,1\). For CIFAR-10, the three separate colour channels have been combined into a colour image.
References
Anastasiadis AD, Magoulas GD, Vrahatis MN (2003) An efficient improvement of the rprop algorithm. In: Proceedings of the First International Workshop on Artificial Neural Networks in Pattern Recognition (IAPR 2003), University of Florence, Italy, p 197
Ciresan DC, Meier U, Gambardella LM, Schmidhuber J (2010) Deep, big, simple neural nets for handwritten digit recognition. Neural Comput 22(12):3207–3220
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(59):1–35
Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv:1412.6572
Hahnloser RH, Sarpeshkar R, Mahowald MA, Douglas RJ, Seung HS (2000) Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789):947
Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: International joint conference on neural networks, 1989, IJCNN, IEEE, pp 593–605
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv:1503.02531
Igel C, Hüsken M (2000) Improving the Rprop learning algorithm. In: Proceedings of the second international ICSC symposium on neural computation (NC 2000), vol 2000. Citeseer, pp 115–121
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Thesis
LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. Handb Brain Theory Neural Netw 3361:310
Lecun Y, Cortes C (1998) The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/. Accessed 3 Sep 2016
Miikkulainen R, Dyer MG (1991) Natural language processing with modular PDP networks and distributed lexicon. Cognit Sci 15(3):343–399
Mosca A, Magoulas GD (2015) Adapting resilient propagation for deep learning. In: UK workshop on computational intelligence
Mosca A, Magoulas G (2016) Deep incremental boosting. In: Benzmuller C, Sutcliffe G, Rojas R (eds) GCAI 2016. 2nd global conference on artificial intelligence, EPiC series in computing, vol 41. EasyChair, pp 293–302
Mosca A, Magoulas GD (2017a) Learning input features representations in deep learning. In: Advances in computational intelligence systems. Springer International Publishing, pp 433–445
Mosca A, Magoulas GD (2017b) Training convolutional networks with weight-wise adaptive learning rates. In: ESANN 2017 proceedings, european symposium on artificial neural networks, computational intelligence and machine learning. Bruges (Belgium), 26–28 April 2017. http://i6doc.com
Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2016a) Practical black-box attacks against deep learning systems using adversarial examples. arXiv:1602.02697
Papernot N, McDaniel P, Wu X, Jha S, Swami A (2016b) Distillation as a defense to adversarial perturbations against deep neural networks. In: Security and privacy (SP), 2016 IEEE Symposium on, IEEE, pp 582–597
Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: the rprop algorithm. In: Proceeding of the IEEE international conference on neural networks, IEEE, pp 586–591
Rumelhart DE, Hinton GE, Williams RJ (1988) Learning representations by back-propagating errors. Cognit Modeling 5(3):1
Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis. http://research.microsoft.com/apps/pubs/default.aspx?id=68920
Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv:1412.6806
Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013) Intriguing properties of neural networks. arXiv:1312.6199
Acknowledgements
The equipment for these experiments was funded by a Grant from NVIDIA Corporation. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GTX Titan X GPUs used for this research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
George D. Magoulas has received research grants from NVIDIA Corporation. Alan Mosca owns stock in Alphabet, Facebook, NVIDIA and Twitter.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by P. Angelov, F. Chao.
Rights and permissions
About this article
Cite this article
Mosca, A., Magoulas, G.D. Hardening against adversarial examples with the smooth gradient method. Soft Comput 22, 3203–3213 (2018). https://doi.org/10.1007/s00500-017-2998-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-017-2998-4