Advertisement

The Dynamics of Matrix Momentum

  • Magnus Rattray
  • David Saad
Conference paper
  • 72 Downloads
Part of the Perspectives in Neural Computing book series (PERSPECT.NEURAL)

Abstract

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton’s method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Keywords

Gradient Descent Hide Node Generalization Error Asymptotic Performance Order Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. [1]
    Fabian V. Ann. Math. Statist. 39 1327 (1968).MathSciNetCrossRefGoogle Scholar
  2. [2]
    Amari S. Neural Computation 10(2) 251 (1998).CrossRefGoogle Scholar
  3. [3]
    Orr GB, Leen TK. Advances in Neural Information Processing Systems vol 9, ed Mozer MC, Jordan MI and Petsche T (Cambridge, MA: MIT Press, 1997) p 606Google Scholar
  4. [4]
    Orr GB. Ph.D. Dissertation, Oregon Graduate Institute of Science & Technology (1995).Google Scholar
  5. [5]
    Saad D, Solla SA. Phys. Rev. Lett. 74, 4337 (1995); Phys. Rev. E 52 4225 (1995).CrossRefGoogle Scholar
  6. [6]
    Leen TK, Schottky B, Saad D. Advances in Neural Information Processing Systems vol 10, ed Jordan MI, Kearns MJ and Solla SA (Cambridge, MA: MIT Press, 1998).Google Scholar
  7. [7]
    Weigerinck W, Komoda A, Heskes T. J. Phys. A 27, 4425 (1994).MathSciNetCrossRefGoogle Scholar
  8. [8]
    Prügel-Bennett A. Unpublished notes, (1996).Google Scholar
  9. [9]
    Rattray M, Saad D. Incorporating curvature information into on-line learning Proc. of the On-line Learning Themed Week, (Isaac Newton Institute, Cambridge, 1997).Google Scholar
  10. [10]
    Rattray M, Saad D, Amari S. Natural gradient descent for on-line learning (in preparation, 1998).CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 1998

Authors and Affiliations

  • Magnus Rattray
    • 1
  • David Saad
    • 1
  1. 1.Neural Computing Research GroupAston UniversityBirminghamUK

Personalised recommendations