Skip to main content

The Dynamics of Matrix Momentum

  • Conference paper
  • First Online:
  • 278 Accesses

Part of the Perspectives in Neural Computing book series (PERSPECT.NEURAL)

Abstract

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton’s method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Keywords

  • Gradient Descent
  • Hide Node
  • Generalization Error
  • Asymptotic Performance
  • Order Algorithm

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-1-4471-1599-1_24
  • Chapter length: 6 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-1-4471-1599-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Fabian V. Ann. Math. Statist. 39 1327 (1968).

    MathSciNet  CrossRef  Google Scholar 

  2. Amari S. Neural Computation 10(2) 251 (1998).

    CrossRef  Google Scholar 

  3. Orr GB, Leen TK. Advances in Neural Information Processing Systems vol 9, ed Mozer MC, Jordan MI and Petsche T (Cambridge, MA: MIT Press, 1997) p 606

    Google Scholar 

  4. Orr GB. Ph.D. Dissertation, Oregon Graduate Institute of Science & Technology (1995).

    Google Scholar 

  5. Saad D, Solla SA. Phys. Rev. Lett. 74, 4337 (1995); Phys. Rev. E 52 4225 (1995).

    CrossRef  Google Scholar 

  6. Leen TK, Schottky B, Saad D. Advances in Neural Information Processing Systems vol 10, ed Jordan MI, Kearns MJ and Solla SA (Cambridge, MA: MIT Press, 1998).

    Google Scholar 

  7. Weigerinck W, Komoda A, Heskes T. J. Phys. A 27, 4425 (1994).

    MathSciNet  CrossRef  Google Scholar 

  8. Prügel-Bennett A. Unpublished notes, (1996).

    Google Scholar 

  9. Rattray M, Saad D. Incorporating curvature information into on-line learning Proc. of the On-line Learning Themed Week, (Isaac Newton Institute, Cambridge, 1997).

    Google Scholar 

  10. Rattray M, Saad D, Amari S. Natural gradient descent for on-line learning (in preparation, 1998).

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 1998 Springer-Verlag London

About this paper

Cite this paper

Rattray, M., Saad, D. (1998). The Dynamics of Matrix Momentum. In: Niklasson, L., Bodén, M., Ziemke, T. (eds) ICANN 98. ICANN 1998. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1599-1_24

Download citation

  • DOI: https://doi.org/10.1007/978-1-4471-1599-1_24

  • Published:

  • Publisher Name: Springer, London

  • Print ISBN: 978-3-540-76263-8

  • Online ISBN: 978-1-4471-1599-1

  • eBook Packages: Springer Book Archive

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.