Soft Computing

, Volume 22, Issue 10, pp 3203–3213 | Cite as

Hardening against adversarial examples with the smooth gradient method

  • Alan MoscaEmail author
  • George D. Magoulas


Commonly used methods in deep learning do not utilise transformations of the residual gradient available at the inputs to update the representation in the dataset. It has been shown that this residual gradient, which can be interpreted as the first-order gradient of the input sensitivity at a particular point, may be used to improve generalisation in feed-forward neural networks, including fully connected and convolutional layers. We explore how these input gradients are related to input perturbations used to generate adversarial examples and how the networks that are trained with this technique are more robust to attacks generated with the fast gradient sign method.



The equipment for these experiments was funded by a Grant from NVIDIA Corporation. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GTX Titan X GPUs used for this research.

Compliance with ethical standards

Conflicts of interest

George D. Magoulas has received research grants from NVIDIA Corporation. Alan Mosca owns stock in Alphabet, Facebook, NVIDIA and Twitter.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. Anastasiadis AD, Magoulas GD, Vrahatis MN (2003) An efficient improvement of the rprop algorithm. In: Proceedings of the First International Workshop on Artificial Neural Networks in Pattern Recognition (IAPR 2003), University of Florence, Italy, p 197Google Scholar
  2. Ciresan DC, Meier U, Gambardella LM, Schmidhuber J (2010) Deep, big, simple neural nets for handwritten digit recognition. Neural Comput 22(12):3207–3220CrossRefGoogle Scholar
  3. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(59):1–35MathSciNetzbMATHGoogle Scholar
  4. Goodfellow IJ, Shlens J, Szegedy C (2014) Explaining and harnessing adversarial examples. arXiv:1412.6572
  5. Hahnloser RH, Sarpeshkar R, Mahowald MA, Douglas RJ, Seung HS (2000) Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789):947CrossRefGoogle Scholar
  6. Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: International joint conference on neural networks, 1989, IJCNN, IEEE, pp 593–605Google Scholar
  7. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv:1503.02531
  8. Igel C, Hüsken M (2000) Improving the Rprop learning algorithm. In: Proceedings of the second international ICSC symposium on neural computation (NC 2000), vol 2000. Citeseer, pp 115–121Google Scholar
  9. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
  10. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
  11. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. ThesisGoogle Scholar
  12. LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. Handb Brain Theory Neural Netw 3361:310Google Scholar
  13. Lecun Y, Cortes C (1998) The MNIST database of handwritten digits. Accessed 3 Sep 2016
  14. Miikkulainen R, Dyer MG (1991) Natural language processing with modular PDP networks and distributed lexicon. Cognit Sci 15(3):343–399CrossRefGoogle Scholar
  15. Mosca A, Magoulas GD (2015) Adapting resilient propagation for deep learning. In: UK workshop on computational intelligenceGoogle Scholar
  16. Mosca A, Magoulas G (2016) Deep incremental boosting. In: Benzmuller C, Sutcliffe G, Rojas R (eds) GCAI 2016. 2nd global conference on artificial intelligence, EPiC series in computing, vol 41. EasyChair, pp 293–302Google Scholar
  17. Mosca A, Magoulas GD (2017a) Learning input features representations in deep learning. In: Advances in computational intelligence systems. Springer International Publishing, pp 433–445Google Scholar
  18. Mosca A, Magoulas GD (2017b) Training convolutional networks with weight-wise adaptive learning rates. In: ESANN 2017 proceedings, european symposium on artificial neural networks, computational intelligence and machine learning. Bruges (Belgium), 26–28 April 2017.
  19. Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2016a) Practical black-box attacks against deep learning systems using adversarial examples. arXiv:1602.02697
  20. Papernot N, McDaniel P, Wu X, Jha S, Swami A (2016b) Distillation as a defense to adversarial perturbations against deep neural networks. In: Security and privacy (SP), 2016 IEEE Symposium on, IEEE, pp 582–597Google Scholar
  21. Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: the rprop algorithm. In: Proceeding of the IEEE international conference on neural networks, IEEE, pp 586–591Google Scholar
  22. Rumelhart DE, Hinton GE, Williams RJ (1988) Learning representations by back-propagating errors. Cognit Modeling 5(3):1zbMATHGoogle Scholar
  23. Simard PY, Steinkraus D, Platt JC (2003) Best practices for convolutional neural networks applied to visual document analysis.
  24. Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv:1412.6806
  25. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013) Intriguing properties of neural networks. arXiv:1312.6199

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science and Information SystemsBirkbeck, University of LondonLondonUK

Personalised recommendations