Abstract
We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton’s method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Fabian V. Ann. Math. Statist. 39 1327 (1968).
Amari S. Neural Computation 10(2) 251 (1998).
Orr GB, Leen TK. Advances in Neural Information Processing Systems vol 9, ed Mozer MC, Jordan MI and Petsche T (Cambridge, MA: MIT Press, 1997) p 606
Orr GB. Ph.D. Dissertation, Oregon Graduate Institute of Science & Technology (1995).
Saad D, Solla SA. Phys. Rev. Lett. 74, 4337 (1995); Phys. Rev. E 52 4225 (1995).
Leen TK, Schottky B, Saad D. Advances in Neural Information Processing Systems vol 10, ed Jordan MI, Kearns MJ and Solla SA (Cambridge, MA: MIT Press, 1998).
Weigerinck W, Komoda A, Heskes T. J. Phys. A 27, 4425 (1994).
Prügel-Bennett A. Unpublished notes, (1996).
Rattray M, Saad D. Incorporating curvature information into on-line learning Proc. of the On-line Learning Themed Week, (Isaac Newton Institute, Cambridge, 1997).
Rattray M, Saad D, Amari S. Natural gradient descent for on-line learning (in preparation, 1998).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag London
About this paper
Cite this paper
Rattray, M., Saad, D. (1998). The Dynamics of Matrix Momentum. In: Niklasson, L., Bodén, M., Ziemke, T. (eds) ICANN 98. ICANN 1998. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1599-1_24
Download citation
DOI: https://doi.org/10.1007/978-1-4471-1599-1_24
Published:
Publisher Name: Springer, London
Print ISBN: 978-3-540-76263-8
Online ISBN: 978-1-4471-1599-1
eBook Packages: Springer Book Archive