Backpropagation Issues with Deep Feedforward Neural Networks

Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 37)

Abstract

Backpropagation is currently the most widely applied neural network architecture. However, for some cases this architecture is less efficient while dealing with deep neural networks [8, 9] as the learning process becomes slower and the sensitivity of the neural network increases. This paper presents an experimental study of different backpropagation architectures in term of deepness of the neural network with different learning rate and activation functions in order to determine the relation between those elements and their impact on the convergence of the backpropagation.

Keywords

Vanishing gradient descend problem Neural networks Deep learning 

References

  1. 1.
    Hecht-Nielsen, R.: Theory of the Backpropagation Network. Department of Electrical and Computer Engineering University of California at San Diego La Jolla (1992)Google Scholar
  2. 2.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015).  https://doi.org/10.1016/j.neunet.2014.09.003. arXiv:1404.7828 CrossRefGoogle Scholar
  3. 3.
    Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: Kremer, S.C., Kolen, J.F. (eds.) A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press (2001)Google Scholar
  4. 4.
    LeCun, Y., Cortes, C., Burges, C.J.C.: MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris Burges. Accessed 17 Aug 2013Google Scholar
  5. 5.
    Sigillito, V.G., et al.: Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Tech. Digest 10(3), 262–266 (1989)Google Scholar
  6. 6.
    Fisher, R.A.: The use of multiple measurements in taxonomic problems. Ann. Eugenics Part II 7, 179–188 (1936)CrossRefGoogle Scholar
  7. 7.
    Nielsen, M.: Chapter 5: Why are deep neural networks hard to train. In: Neural Networks and Deep Learning (2017)Google Scholar
  8. 8.
    Ghanou, Y., Bencheikh, G.: Architecture optimization and training for the multilayer perceptron using ant system. IAENG Int. J. Comput. Sci. 43(1), 20–26 (2016)Google Scholar
  9. 9.
    Ettaouil, M., Ghanou, Y.: Neural architectures optimization and genetic algorithms. WSEAS Trans. Comput. 8(3), 526–537 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.High School of TechnologyUniversity Moulay IsmailMeknesMorocco

Personalised recommendations