Skip to main content

Capacity of the Upstart Algorithm

  • Chapter
  • 2209 Accesses

Part of the Operations Research/Computer Science Interfaces Series book series (ORCS,volume 8)

Abstract

The storage capacity of multilayer networks with overlapping receptive fields is investigated for a constructive algorithm within a one-step replica symmetry breaking (RSB) treatment. We find that the storage capacity increases logarithmically with the number of hidden units K without saturating the Mitchison-Durbin bound. The slope of the logarithmic increase decays exponentionally with the stability with which the patterns have been stored.

Keywords

  • Hide Unit
  • Output Unit
  • Constructive Algorithm
  • Multilayer Network
  • Logarithmic Increase

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-1-4615-6099-9_65
  • Chapter length: 6 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   269.00
Price excludes VAT (USA)
  • ISBN: 978-1-4615-6099-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   349.99
Price excludes VAT (USA)
Hardcover Book
USD   349.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. E. Gardner, The space of interactions in neural network models, J. Phys. A Vol. 21 (1988), pp257–270.

    MathSciNet  CrossRef  Google Scholar 

  2. E. Barkai, D. Hansel and H. Sompolinsky, Broken symmetries in multilayered perceptrons, Phys. Rev. A Vol. 45 (1992), pp4146–4161.

    CrossRef  Google Scholar 

  3. G. J. Mitchison and R. M. Durbin, Bounds on the learning capacity of some multi-layer networks, Biological Cybernetics Vol. 60 (1989), pp345–356.

    MathSciNet  MATH  CrossRef  Google Scholar 

  4. D. Saad, Explicit symmetries and the capacity of multilayer neural networks, J. Phys. A Vol. 27(1994), pp2719–2734.

    Google Scholar 

  5. A. H. L. West, and D. Saad, Threshold induced phase transitions in perceptrons, To appear in J. Phys. A (March 1997).

    Google Scholar 

  6. M. Méz ard and J.-P. Nadal, Learning in feed-forward layered networks: the tiling algorithm, J. Phys. A Vol. 22 (1989), pp2191–2203.

    MathSciNet  CrossRef  Google Scholar 

  7. J.-P. Nadal, Study of a growth algorithm for a feed-forward network, International Journal of Neural Systems, Vol.1 (1989), pp55–59.

    CrossRef  Google Scholar 

  8. M. Frean, The upstart algorithm: a method for constructing and training feed-forward neural networks, Neural Computation Vol. 2 (1990), pp198–209.

    CrossRef  Google Scholar 

  9. For an overview, see e.g., M. Mézard, G. Parisi and M. G. Virasoro, Spin Glass Theory and Beyond, World Scientific, Singapore (1987).

    Google Scholar 

  10. M. Frean, A thermal perceptron learning rule, Neural Computation Vol. 4 (1992), pp946-957, and references therein.

    Google Scholar 

  11. A. H. L. West and D. Saad, Statistical mechanics of constructive algorithms, in preparation (1997).

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 1997 Springer Science+Business Media New York

About this chapter

Cite this chapter

West, A.H.L., Saad, D. (1997). Capacity of the Upstart Algorithm. In: Ellacott, S.W., Mason, J.C., Anderson, I.J. (eds) Mathematics of Neural Networks. Operations Research/Computer Science Interfaces Series, vol 8. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-6099-9_65

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-6099-9_65

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-7794-8

  • Online ISBN: 978-1-4615-6099-9

  • eBook Packages: Springer Book Archive