Advertisement

Estimation of Deep Neural Networks Capabilities Using Polynomial Approach

  • Pawel RozyckiEmail author
  • Janusz Kolbusz
  • Roman Korostenskyi
  • Bogdan M. Wilamowski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9692)

Abstract

Currently very popular trend in artificial intelligence is the use of deep neural networks. The power of such networks are very large, but the main difficulty is learning these networks. The article presents a analysis of deep neural network nonlinearity with polynomial approximation of neuron activation functions. It is shown that nonlinearity grows exponentially with the depth of the neural network. The effectiveness of the approach is demonstrated by several experiments.

Keywords

Deep neural networks Activation function Nonlinearity 

References

  1. 1.
    Deng, L., Li, X.: Machine learning paradigms in speech recognition: An overview. IEEE Trans. Audio Speech Lang. Process. 21(5), 1060–1089 (2013)CrossRefGoogle Scholar
  2. 2.
    Lee, H., Grosse, R., Ranganath, R., Ng, A.: Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. Assoc. Comput. Mach. (ACM) 54(10), 95–103 (2011)Google Scholar
  3. 3.
    Wilamowski, B.M.: Neural network architectures and learning algorithms - how not to be frustrated with neural networks. IEEE Ind. Electron. Mag. 3(4), 56–63 (2009)CrossRefGoogle Scholar
  4. 4.
    Hunter, D., Yu, H., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures–A comparative study. IEEE Trans. Industr. Inf. 8, 228–240 (2012)CrossRefGoogle Scholar
  5. 5.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Unc. Fuzz. Knowl. Based Syst. 06, 107 (1998)CrossRefzbMATHGoogle Scholar
  7. 7.
    Erhan, D., Bengio, Y., Courvelle, A., Manzagol, P., Vencent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 9, 201–208 (2010)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Schmidhuber, J.: Deep learning in neural networks: An overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  9. 9.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  10. 10.
    Olas, T., Mleczko, W.K., Nowicki, R.K., Wyrzykowski, R., Krzyzak, A.: Adaptation of RBM learning for intel MIC architecture. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 90–101. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  11. 11.
    Wilamowski, B.M., Hunter, D., Malinowski, A.: Solving parity-N problems with feedforward neural networks. In: Proceedings of the 2003 IEEE IJCNN, pp. 2546–2551. IEEE Press (2003)Google Scholar
  12. 12.
    Xu, Y.: An experimental study on speech enhancement based on deep neural networks. IEEE Signal Process. Lett. 21(1), 65–68 (2011)CrossRefGoogle Scholar
  13. 13.
    Gao, J., et al.: Modeling interestingness with deep neural networks. U.S. Patent No. 20,150,363,688, 17 December 2015Google Scholar
  14. 14.
    Erhan, D., et al.: Scalable object detection using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  15. 15.
    Zhang, X., et al.: Improving deep neural network acoustic models using generalized maxout networks. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2014)Google Scholar
  16. 16.
    Montufar, G.F., et al.: On the number of linear regions of deep neural networks. In: Advances in Neural Information Processing Systems (2014)Google Scholar
  17. 17.
    Oquab, M., et al.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  18. 18.
    Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems (2013)Google Scholar
  19. 19.
    Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)CrossRefGoogle Scholar
  20. 20.
    Wilamowski, B.M., Yu, H.: NNT - Neural Networks Trainer. http://www.eng.auburn.edu/wilambm/nnt/. Accessed 15 January 2016
  21. 21.
    Alvarez-Sanchez, J.R.: Injecting knowledge into the solution of the two-spiral problem. Neural Comput. Appl. 8, 265–272 (1999)CrossRefGoogle Scholar
  22. 22.
    Różycki, P., Kolbusz, J., Bartczak, T., Wilamowski, B.M.: Using parity-N problems as a way to compare abilities of shallow, very shallow and very deep architectures. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 112–122. Springer, Heidelberg (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Pawel Rozycki
    • 1
    Email author
  • Janusz Kolbusz
    • 1
  • Roman Korostenskyi
    • 1
  • Bogdan M. Wilamowski
    • 2
  1. 1.University of Information Technology and Management in RzeszowRzeszowPoland
  2. 2.Auburn UniversityAuburnUSA

Personalised recommendations