Boundedness of Weight Elimination for BP Neural Networks

  • Jian Wang
  • Jacek M. Zurada
  • Yanjiang Wang
  • Jing Wang
  • Guofang Xie
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8467)

Abstract

Weight elimination can be usefully interpreted as an assumption about the prior distribution of the weights trained in the backpropagation neural networks (BPNN). Weight elimination based on different scaling of weight parameters is of a general form, with the weight decay and subset selection methods as special cases. The applications of this method have been well developed, however, only few references provides more comprehensive theoretical analysis. To address this issue, we investigate the uniform boundedness of the trained weights based on a descriptive proof.

Keywords

backpropagation neural networks weight decay weight elimination boundedness 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Werbos, P.J.: Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. thesis. Harvard University, Cambridge, MA (1974)Google Scholar
  2. 2.
    Parker, D.B.: Learning-logic, invention report. Stanford University, Stanford (1982)Google Scholar
  3. 3.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagation errors. Nature 323, 533–536 (1986)CrossRefGoogle Scholar
  4. 4.
    Cherkassky, V., Mulier, F.M.: Learning from data: Concepts, Theory, and Methods. IEEE Press (2007)Google Scholar
  5. 5.
    Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Computation 7, 219–269 (1995)CrossRefGoogle Scholar
  6. 6.
    Friedman, J.: An Overview of Predictive Learning and Function Approximation. In: From Statistics to Neural Networks, vol. 136, pp. 1–61. Springer, Heidelberg (1994)CrossRefGoogle Scholar
  7. 7.
    Weigend, A.S., Rumelhart, D.E., Huberman, B.A.: Generalization by weight-elimination applied to currency exchange rate prediction. In: IJCNN 1991-Seattle International Joint Conference on Neural Networks, vol. 1, pp. 837–841 (1991)Google Scholar
  8. 8.
    Bebis, G., Georgiopoulos, M., Kaspalris, T.: Coupling weight elimination and genetic algorithms. In: IEEE International Conference on Neural Networks, vol. 2, pp. 1115–1120 (1996)Google Scholar
  9. 9.
    Gupta, A., Lam, M.: The weight decay backpropagation for generalizations with missing values. Annals of Operations Research 78, 165–187 (1998)CrossRefMATHGoogle Scholar
  10. 10.
    Leung, C.-S., Tsoi, A.-C., Chan, L.-W.: Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks. IEEE Transactions on Neural Networks 12, 1314–1332 (2001)CrossRefGoogle Scholar
  11. 11.
    Kuo, R.J., Wu, P., Wang, C.P.: An intelligent sales forecasting system through integration of artificial neural networks and fuzzy neural networks with fuzzy weight elimination. Neural Networks 15, 909–925 (2002)CrossRefGoogle Scholar
  12. 12.
    Ennett, C.M., Frize, M.: Weight-elimination neural networks applied to coronary surgery mortality prediction. IEEE Transactions on Information Technology in Biomedicine 7, 86–92 (2003)CrossRefGoogle Scholar
  13. 13.
    May, P., Zhou, E., Lee, C.W.: A Comprehensive Evaluation of Weight Growth and Weight Elimination Methods Using the Tangent Plane Algorithm. International Journal of Advanced Computer Sciences and Applications 4, 149–156 (2013)Google Scholar
  14. 14.
    Zhang, H., Wu, W., Liu, F., Yao, M.: Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks. IEEE Transactions on Neural Networks 20, 1050–1054 (2009)CrossRefGoogle Scholar
  15. 15.
    Shao, H., Zheng, G.: Boundedness and convergence of online gradient method with penalty and momentum. Neurocomputing 74, 765–770 (2011)CrossRefGoogle Scholar
  16. 16.
    Wang, J., Wu, W., Zurada, J.M.: Boundedness and convergence of MPN for cyclic and almost cyclic learning with penalty. In: The 2011 International Joint Conference on Neural Networks (IJCNN), pp. 125–132 (2011)Google Scholar
  17. 17.
    Wang, J., Wu, W., Zurada, J.M.: Computational properties and convergence analysis of BPNN for cyclic and almost cyclic learning with penalty. Neural Networks 33, 127–135 (2012)CrossRefMATHGoogle Scholar
  18. 18.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Jian Wang
    • 1
    • 2
    • 3
  • Jacek M. Zurada
    • 2
    • 4
  • Yanjiang Wang
    • 3
  • Jing Wang
    • 1
  • Guofang Xie
    • 5
  1. 1.Dalian University of TechnologyDalianChina
  2. 2.University of LouisvilleLouisvilleUSA
  3. 3.China University of PetroleumQingdaoChina
  4. 4.Information Technology InstituteUniversity of Social SciencesLodzPoland
  5. 5.Freelance Translator and Amateur MathematicianChina

Personalised recommendations