Neural Networks Saturation Reduction

  • Janusz Kolbusz
  • Pawel RozyckiEmail author
  • Oleksandr Lysenko
  • Bogdan M. Wilamowski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10841)


The saturation of particular neuron and a whole neural network is one of the reasons for problems with training effectiveness. The paper shows neural network saturation analysis, proposes a method for detection of saturated neurons and its reduction to achieve better training performance. The proposed approach has been confirmed by several experiments.


Network training improvement Saturation Reduction Activation functions 


  1. 1.
    Rozycki, P., Kolbusz, J., Wilamowski, B.M.: Dedicated deep neural network architectures and methods for their training. In: IEEE 19th International Conference on Intelligent Engineering Systems (INES 2015) Bratislava, 3–5 September 2015, pp. 73–78 (2015)Google Scholar
  2. 2.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Mohamed, A., Dahl, G.E., Hinton, G.E.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20, 14–22 (2012)CrossRefGoogle Scholar
  4. 4.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  5. 5.
    Simonyan K., Zisserman A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  6. 6.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  7. 7.
    Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)CrossRefGoogle Scholar
  8. 8.
    Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)CrossRefGoogle Scholar
  9. 9.
    Hunter, D., Hao, Y., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures—A comparative study. IEEE Trans. Industr. Inf. 8, 228–240 (2012)CrossRefGoogle Scholar
  10. 10.
    Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzz. Knowl. Based Syst. 06, 107 (1998)zbMATHCrossRefGoogle Scholar
  11. 11.
    Larochelle, H., et al.: Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10(Jan), 1–40 (2009)zbMATHGoogle Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV (2015)Google Scholar
  13. 13.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is dificult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)CrossRefGoogle Scholar
  14. 14.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International Conference on Articial Intelligence and Statistics, pp. 249–256 (2010)Google Scholar
  15. 15.
    Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv preprint arXiv:1409.5185 (2014)
  16. 16.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  17. 17.
    Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
  18. 18.
    Kolbusz J., Różycki P., Wilamowski B.M.: The study of architecture MLP with linear neurons in order to eliminate the “vanishing gradient” problem. In: Artificial Intelligence and Soft Computing, ICAISC 2017, pp. 97–106 (2017)Google Scholar
  19. 19.
    Rakitianskaia, A., Engelbrecht, A.: Measuring saturation in neural networks. In: 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, pp. 1423–1430 (2015)Google Scholar
  20. 20.
    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 9–48. Springer, Heidelberg (2012). Scholar
  21. 21.
    Rakitianskaia, A., Engelbrecht, A.: Training high-dimensional neural networks with cooperative particle swarm optimiser. In: 2014 International Joint Conference on Neural Networks (IJCNN), Beijing, pp. 4011–4018 (2014)Google Scholar
  22. 22.
    Wilamowski, B.M., Yu, H.: Improved computation for levenberg Marquardt training. IEEE Trans. Neural Netw. 21(6), 930–937 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Janusz Kolbusz
    • 1
  • Pawel Rozycki
    • 1
    Email author
  • Oleksandr Lysenko
    • 2
  • Bogdan M. Wilamowski
    • 3
  1. 1.University of Information Technology and Management in RzeszowRzeszówPoland
  2. 2.National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”KievUkraine
  3. 3.Auburn UniversityAuburnUSA

Personalised recommendations