Advertisement

On H ∞  Filtering in Feedforward Neural Networks Training and Pruning

  • He-Sheng Tang
  • Song-Tao Xue
  • Rong Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3971)

Abstract

An efficient training and pruning method based on H∞ filtering algorithm is proposed for Feedforward neural networks (FNNs). A FNNs’ weight importance measure linking up prediction error sensitivity obtained from H ∞  filtering training and a weight salience based pruning technique are derived. The results of extensive experimentation indicate that the proposed method provides better pruning results during the training process of the network without losing its generalization capacity, also provides a robust global optimization training algorithm for given arbitrary network structures.

Keywords

Extend Kalman Filter Recursive Less Square Pruning Method Generalization Capacity Recursive Less Square Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Rumelhart, D., Hinton, G., Williams, G.: Learning Internal Representations by Error Propagation. In: Parallel Distributed Processing, vol. 1, pp. 318–362. MIT Press, Cambridge (1986)Google Scholar
  2. 2.
    Robbins, H., Monro, S.: A Stochastic Approximation Method. Ann. Math. Stat. 22, 400–407 (1951)CrossRefMathSciNetMATHGoogle Scholar
  3. 3.
    Bojarczak, O.S.P., Stodolski, M.: Fast Second-order Learning Algorithm for Feed-forward Multilayer Neural Networks and Its Application. Neural Networks 9(9), 1583–1596 (1996)CrossRefGoogle Scholar
  4. 4.
    Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, Inc., Englewood Cliffs (1999)MATHGoogle Scholar
  5. 5.
    Anderson, B.D.O., Moore, J.B.: Optimal Filtering. Prentice-Hall, Englewood Cliffs (1979)MATHGoogle Scholar
  6. 6.
    Singhal, S., Wu, L.: Training Multiplayer Perceptrons with the Extended Kalman Al-gorithm. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems, vol. 1, pp. 133–140. Morgan Kaufmann, San Mateo (1989)Google Scholar
  7. 7.
    Fukuda, W.K.T., Tzafestas, S.G.: Learning Algorithms of Layered Neural Networks via Extended Kalman Filters. Int. J. Syst. Sci. 22(4), 753–768 (1991)CrossRefMathSciNetMATHGoogle Scholar
  8. 8.
    Iiguni, Y., Sakai, H., Tokumaru, H.: A Real-time Learning Algorithm for a Multilay-ered Neural Network Based on the Extended Kalman Filter. IEEE Trans. Signal Processing 40(4), 959–966 (1992)CrossRefGoogle Scholar
  9. 9.
    Chen, S., Cowan, C.F.N., Billings, S.A., Grant, P.M.: Parallel Recursive Prediction Error Algorithm for Training Layered Neural Network. Int. J. Contr. 51(6), 1215–1228 (1990)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Kollias, S., Anastassiou, D.: An Adaptive Least Squares Algorithm for the Efficient Training of Artificial Neural Networks. IEEE Trans. Circuits Syst. 36(8), 1092–1101 (1989)CrossRefGoogle Scholar
  11. 11.
    Leung, C.S., Wong, K.W., Sum, J., Chan., L.W.: On-line Training and Pruning for RLS Algorithms. Electron. Lett. 32(23), 2152–2153 (1996)CrossRefGoogle Scholar
  12. 12.
    Tang, H., Sato, T.: Structural Damage Detection Using the Neural Network and H∞ Algorithm. In: Kundu, T. (ed.) Health Monitoring and Smart Nondestructive Evaluation of Structural and Biological Systems III, Proceedings of SPIE, San Diego, CA, vol. 5394, pp. 454–463 (2004)Google Scholar
  13. 13.
    Abrahart, R.J., See, L., Kneal, P.E.: Investigating the Role of Saliency Analysis with Neural Network Rainfall-runoff Model. Computer & Geosciences 27(8), 921–928 (2001)CrossRefGoogle Scholar
  14. 14.
    Makarynskyy, O., Pires-Silva, A.A., Makarynska, D., Ventura-Soares, C.: Artificial Neural Networks in Wave Predictions at the West Coast of Portual. Computers & Geo-sciences 31(4), 415–424 (2005)Google Scholar
  15. 15.
    Akaike, H.: A New Look at the Statistical Model Identification. IEEE Trans. Automat. Control AC 19(6), 716–723 (1974)CrossRefMathSciNetMATHGoogle Scholar
  16. 16.
    Stone, M.: An Asymptotic Equivalence of Choice of Model by Cross-validation and Akaike’s criterion. Roy. Stat. Soc. Ser. 39(1), 44–47 (1977)MATHGoogle Scholar
  17. 17.
    Krogh, A., Hertz, J.: A Simple Weight Decay Can Improve Generalization. In: Touretzky, D.S. (ed.) Advances in Neural Information-Processing Systems, vol. 4, pp. 950–957. Morgan Kaufmann, San Mateo (1992)Google Scholar
  18. 18.
    William, P.M.: Bayesian Regularization and Pruning Use a Laplace Prior. Neural Comput. 7(1), 117–143 (1995)CrossRefGoogle Scholar
  19. 19.
    LeCun, Y., Denker, J.S., Solla, S.A.: Optimal Brain Damage. In: Touretzky, D.S. (ed.) Advances in neural information processing, vol. 1, pp. 396–404. Morgan Kaufman, San Mateo (1990)Google Scholar
  20. 20.
    Hassibi, B., Stork, D.G.: Second-order Derivatives for Network Pruning: Optimal Brain Surgeon. In: Hanson, S.J., Cowan, J.D., Lee Giles, C. (eds.) Advances in neu-ral information processing, vol. 4, pp. 164–171. Morgan Kaufman, San Francisco (1993)Google Scholar
  21. 21.
    Sum, J., Leung, C., Young, G.H., Kan, W.: On the Kalman Filtering Method in Neural Network Training and Pruning. IEEE Trans. on Neural Networks 10(1), 161–166 (1999)CrossRefGoogle Scholar
  22. 22.
    Leung, C., Wong, K., Sum, P., Chan, L.: A Pruning Method for the Recursive Least Squared Algorithm. Neural Networks 14(2), 147–174 (2001)CrossRefGoogle Scholar
  23. 23.
    Prechelt, L.: A quantitative Study of Experimental Evaluations of Neural Network Learning Algorithms: Current Research Practice. Neural Networks 9(3), 457–462 (1996)CrossRefGoogle Scholar
  24. 24.
    Sato, T., Qi, K.: Adaptive H∞ fillter: Its Application to Structural Identification. Journal of Engineering Mechanics 124(11), 1233–1240 (1998)CrossRefGoogle Scholar
  25. 25.
    Hassibi, B., Sayed, A.H., Kailath, T.: Indefinite-Quadratic Estimation and Control: A Unified Approach to H2 and H∞Theories. SIAM, Philadelphia (1998)Google Scholar
  26. 26.
    Didinsky, G., Pan, Z., Basar, T.: Parameter Identification of Uncertain Plants Using H∞ Methods. Automatica 31(9), 1227–1250 (1995)CrossRefMathSciNetMATHGoogle Scholar
  27. 27.
    Wen, Y.-K.: Method for Random Vibration of Hysteretic Systems. Journal of Engineering Mechanics 102(EM2), 249–263 (1976)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • He-Sheng Tang
    • 1
  • Song-Tao Xue
    • 2
  • Rong Chen
    • 1
  1. 1.Research Institute of Structural Engineering and Disaster ReductionTongji UniversityShanghaiChina
  2. 2.Department of Architecture, School of Science and EngineeringKinki UniversityHigashi Osaka CityJapan

Personalised recommendations