A Fast and Numerically Robust Neural Network Training Algorithm

  • Youmin Zhang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4029)


A fast and numerically robust algorithm for training feedforward neural networks (FNNs) using a recursive prediction error (RPE) method, along with an adaptively-adjustable time-varying forgetting factor technique, is presented first. Then a U-D factorization-based RPE (UD-RPE) algorithm is proposed to further improve the training rate and accuracy of the FNNs. In comparison with the backpropagation (BP) and existing RPE based training algorithms, the proposed UD-RPE can provide substantially better results in fewer iterations with fewer hidden nodes and improve significantly convergence rate and numerical stability. In addition, it is less sensitive to start-up parameters, such as initial weights and covariance matrix. It has also good model prediction ability and need less training time. The effectiveness of the proposed algorithm is demonstrated via two nonlinear systems modeling and identification examples.


Root Mean Square Error Hide Node Training Algorithm Multiple Input Multiple Output Multiple Input Multiple Output 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bierman, G.J.: Factorization Methods for Discrete Sequential Estimation. Academic Press, New York (1977)zbMATHGoogle Scholar
  2. 2.
    Chen, S., Billings, S.A.: Neural networks for nonlinear dynamical system modeling and identification. Int. J. Control 56(2), 319–346 (1992)zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Chen, S., Billings, S.A., Grant, P.M.: Non-linear system identification using neural networks. Int. J. Control 51(6), 1191–1214 (1990)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Chen, S., Cowan, C.F.N., Grant, P.M.: Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans. Neural Networks 2(2), 302–309 (1991)CrossRefGoogle Scholar
  5. 5.
    Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice-Hall, Upper Saddle River (1999)zbMATHGoogle Scholar
  6. 6.
    Ljung, L., Soderstrom, T.: Theory and Practice of Recursive Identification. MIT Press, Cambridge (1983)zbMATHGoogle Scholar
  7. 7.
    Narendra, K.S., Parthasarathy, K.: Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Networks 1(1), 4–27 (1990)CrossRefGoogle Scholar
  8. 8.
    Singhal, S., Wu, L.: Training feed-forward networks with the extended Kalman algorithm. In: Proc. Int. Conf. on ASSP, pp. 1187–1190 (1989)Google Scholar
  9. 9.
    Zhang, Y.M., Li, X.R.: A fast U-D factorization-based learning algorithm with applications to nonlinear system modeling and identification. IEEE Trans. Neural Networks 10(4), 930–938 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Youmin Zhang
    • 1
  1. 1.Department of Computer Science and EngineeringAalborg University EsbjergEsbjergDenmark

Personalised recommendations