Neural Network Training with Second Order Algorithms
Second order algorithms are very efficient for neural network training because of their fast convergence. In traditional Implementations of second order algorithms [Hagan and Menhaj 1994], Jacobian matrix is calculated and stored, which may cause memory limitation problems when training large-sized patterns. In this paper, the proposed computation is introduced to solve the memory limitation problem in second order algorithms. The proposed method calculates gradient vector and Hessian matrix directly, without Jacobian matrix storage and multiplication. Memory cost for training is significantly reduced by replacing matrix operations with vector operations. At the same time, training speed is also improved due to the memory reduction. The proposed implementation of second order algorithms can be applied to train basically an unlimited number of patterns.
KeywordsJacobian Matrix Gradient Vector Training Pattern Levenberg Marquardt Neural Network Training
Unable to display preview. Download preview PDF.
- [Cao et al. 2006]
- [Hagan and Menhaj 1994]
- [Hohil et al. 1999]
- [Rumelhart et al. 1986]
- [Wilamowski 2009]
- [Wilamowski et al. 2008]
- [Wilamowski et al. 2010]Yu, H., Wilamowski, B.M.: Neural network learning without backpropagation. IEEE Trans. on Neural Networks 21(11) (2010)Google Scholar
- [Wilamowski and Yu 2010]
- [Yu and Wilamowski 2009]Yu, H., Wilamowski, B.M.: Efficient and reliable training of neural networks. In: Proc. 2nd IEEE Human System Interaction Conf. HSI 2009, Catania, Italy, pp. 109–115 (2009)Google Scholar
- [Yu et al. 2009]Yu, H., Wilamowski, B.M.: C++ implementation of neural networks trainer. In: Proc. of 13th Int. Conf. on Intelligent Engineering Systems, INES 2009, Barbados (2009)Google Scholar