Abstract
Probabilistic convergence results of online gradient descent algorithm have been obtained by many authors for the training of recurrent neural networks with innitely many training samples. This paper proves deterministic convergence of o2ine gradient descent algorithm for a recurrent neural network with nite number of training samples. Our results can be hopefully extended to more complicated recurrent neural networks, and serve as a complementary result to the existing probability convergence results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Gori, M., Maggini, M.: Optimal convergence of on-line backpropagation. IEEE Transaction on Neural Networks 7, 251–254 (1996)
Haykin, S.: Neural Networks: A Comprehensive Foundation, 2nd edn. Prentice-Hall, Englewood Cliffs (2001)
Kuan, C.M., Hornik, K., White, H.: A Convergence Resuluts for Learning in Recurrent Neural Networks. Neural Computation 6, 420–440 (1994)
Ortega, J., Rheinboldt, W.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New York (1970)
Williams, R.J., Zisper, J.: A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural computation 1, 270–280 (1989)
Williams, R.J., Zisper, J.: Experimental Analysis of the Real-Time Recurrent Learning Algorithm. Connection Science 1, 81–111 (1989)
Wu, W.: Computation of Neural Networks. Higher Education Press, Beijing (2003)
Yuan, Y.X., Sun, W.Y.: Optimization Theory and Methods. Science Press, Beijing (2001)
Zhang, X.S.: Neural Networks in Optimization. Kluwer Academic Publishers, Boston (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Xu, D., Li, Z., Wu, W., Ding, X., Qu, D. (2007). Convergence of Gradient Descent Algorithm for a Recurrent Neuron. In: Liu, D., Fei, S., Hou, Z., Zhang, H., Sun, C. (eds) Advances in Neural Networks – ISNN 2007. ISNN 2007. Lecture Notes in Computer Science, vol 4493. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72395-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-540-72395-0_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72394-3
Online ISBN: 978-3-540-72395-0
eBook Packages: Computer ScienceComputer Science (R0)