Abstract
We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Amari S. Neural Computation 10(2) 251 (1998).
Yang HY, Amari S. Advances in Neural Information Processing Systems vol 10, ed Jordan MI, Kearns MJ and Solla SA (Cambridge, MA: MIT Press, 1998).
Yang HY, Amari S. Natural Gradient Descent for Training Multi-Layer Perceptrons. Submitted to IEEE Transactions on Neural Networks (1998).
Saad D, Solla SA. Phys. Rev. Lett. 74, 4337 (1995); Phys. Rev. E 52 4225 (1995).
Barber D, Saad D, Sollich P. Europhysics Letters 34 151 (1996).
Saad D, Rattray M. Phys. Rev. Lett. 79 2578 (1997).
West AHL, Saad D. Phys. Rev. E 56 3426 (1997).
Leen TK, Schottky B, Saad D. Advances in Neural Information Processing Systems vol 10, ed Jordan MI, Kearns MJ and Solla SA (Cambridge, MA: MIT Press, 1998).
Cybenko G. Math. Control Signals and Systems 2 303 (1989).
Rattray M, Saad D, Amari S. Natural gradient descent for on-line learning (in preparation, 1998).
Orr GB, Leen TK. Advances in Neural Information Processing Systems vol 9, ed Mozer MC, Jordan MI and Petsche T (Cambridge, MA: MIT Press, 1997) p 606.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag London
About this paper
Cite this paper
Rattray, M., Saad, D. (1998). Transients and Asymptotics of Natural Gradient Learning. In: Niklasson, L., Bodén, M., Ziemke, T. (eds) ICANN 98. ICANN 1998. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1599-1_21
Download citation
DOI: https://doi.org/10.1007/978-1-4471-1599-1_21
Published:
Publisher Name: Springer, London
Print ISBN: 978-3-540-76263-8
Online ISBN: 978-1-4471-1599-1
eBook Packages: Springer Book Archive