Skip to main content
SpringerLink
Log in

Associative Data Storage and Retrieval in Neural Networks

  • Chapter
Models of Neural Networks III

Part of the book series: Physics of Neural Networks ((NEURAL NETWORKS))

Synopsis

Associative storage and retrieval of binary random patterns in various neural net models with one-step threshold-detection retrieval and local learning rules are the subject of this chapter. For different heteroas-sociation and autoassociation memory tasks specified by the properties of the pattern sets to be stored and upper bounds on the retrieval errors, we compare the performance of various models of finite as well as asymptotically infinite sizes. In infinite models, we consider the case of asymptotically sparse patterns, where the mean activity in a pattern vanishes, and study two asymptotic fidelity requirements: constant error probabilities and vanishing error probabilities. A signal-to-noise ratio analysis is carried out for one retrieval step where the calculations are comparatively straightforward and easy. As performance measures we propose and evaluate information capacities in bits/synapse which also take into account the important property of fault tolerance. For autoassociation we compare one-step and fixed-point retrieval that is analyzed in the literature by methods of statistical mechanics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Hodgkin, A.L., Huxley, A.F. (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. (Lond.) 117:500–544

    Google Scholar 

  2. McCulloch, W.S., Pitts, W. (1943) A logical calculus of the ideas immanent in neural activity. Bull. Math. Biophys. 5

    Google Scholar 

  3. Steinbuch, K (1936) Die Lernmatrix. Kybernetik 1:36

    Article  Google Scholar 

  4. Willshaw, D.J., Buneman, O.P., Longuet-Higgans, H.C. (1969) Nonholographic associative memory. Nature (London) 222:960–962

    Article  ADS  Google Scholar 

  5. Rosenblatt, F. (1962) Principle of Neurodynamics (Spartan Books, New York)

    Google Scholar 

  6. Little, W.A. (1974) The existence of persistent states in the brain. Math. Biosci. 19:101–120

    Article  MATH  Google Scholar 

  7. Kirkpatrick, S., Sherrington, D. (1978) Infinite-ranged models of spin-glasses. Phys. Rev. B 17:4384–4403

    Article  ADS  Google Scholar 

  8. Amit, D.J., Gutfreund, H., Sompolinsky, H. (1987) Statistical mechanics of neural networks near saturation. Ann. Phys. 173:30–67

    Article  ADS  Google Scholar 

  9. Domany, E., van Hemmen, J.L., Schulten, K. (1991) Models of Neural Networks (Springer-Verlag, Berlin)

    Book  MATH  Google Scholar 

  10. Amit, D. J. (1989) Modelling Brain Function (Cambridge University Press, Cambridge)

    Google Scholar 

  11. Hertz, J., Krogh, A., Palmer, R. G. (1991) Introduction to the Theory of Neural Computation (Addison Wesley, Redwood City, CA)

    Google Scholar 

  12. Uttley, A.M. (1956) Conditional probability machines and conditional reflexes. In: An. Math. Studies 34, Shannon, C.E., McCarthy, J. (Eds.) (Princeton Univ. Press, Princeton, NJ), pp. 237–252

    Google Scholar 

  13. Longuett-Higgins, H.C., Willshaw, D.J., Buneman, O.P. (1970) Theories of associative recall. Q. Rev. Biophys. 3:223–244

    Article  Google Scholar 

  14. Amari, S.I. (1971) Characteristics of randomly connected threshold-element networks and network systems. Proc. IEEE 59:35–47

    Article  MathSciNet  Google Scholar 

  15. Gardner-Medwin, A.R. (1976) The recall of events through the learning of associations between their parts. Proc. R. Soc. Lond. B. 194:375–402

    Article  ADS  Google Scholar 

  16. Kohonen, T. (1977) Associative Memory (Springer-Verlag, Berlin)

    Book  MATH  Google Scholar 

  17. Caianiello, E.R. (1961) Outline of a theory of thought processes and thinking machines. J. Theor. Biol. 1:204–225

    Article  MathSciNet  Google Scholar 

  18. Holden, A.V. (1976) Models of the Stochastic Activity of Neurons (Springer-Verlag, Berlin)

    Book  Google Scholar 

  19. Abeles, M. (1982) Local Cortical Circuits (Springer-Verlag, Berlin)

    Book  Google Scholar 

  20. Buhmann, J., Schulten, K. (1986) Associative recognition and storage in a model network of physiological neurons. Biol. Cybern. 54:319–335

    Article  MATH  Google Scholar 

  21. Anderson, J.A. (1968) A memory storage model utilizing spatial correlation functions. Kybernetik 5:113–119

    Article  Google Scholar 

  22. Anderson, J.A. (1972) A simple neural network generating an interactive memory. Math. Biosci. 14:197–220

    Article  MATH  Google Scholar 

  23. Palm, G. (1980) On associative memory. Biol. Cybern. 36:19–31

    Article  MATH  Google Scholar 

  24. Nadal, J.-P., Toulouse, G. (1990) Information storage in sparsely coded memory nets. Network 1:61–74

    Article  MathSciNet  MATH  Google Scholar 

  25. Hopfield, J. J. (1982) Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Sci. 79:2554–2558

    Article  MathSciNet  ADS  Google Scholar 

  26. van Hemmen, J.L. (1987) Nonlinear networks near saturation. Phys. Rev. A: Math. Gen. 36:1959–1962

    Article  ADS  Google Scholar 

  27. Tsodyks, M.V., Feigelman, M.V. (1988) The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6:101–105

    Article  ADS  Google Scholar 

  28. Amari, S.I. (1989) Statistical neurodynamics of associative memory. Neural Networks 1:63–73

    Article  Google Scholar 

  29. Fontanari, J.F., Köberle, R. (1988) Information processing in synchronous neural networks. J. Phys.France 49:13–23

    Article  Google Scholar 

  30. Palm, G., Sommer, F. T. (1992) Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states. Network 3:1–10

    Article  Google Scholar 

  31. Gibson, W.G., Robinson, J. (1992) Statistical analysis of the dynamics of a sparse associative memory. Neural Networks 5:645–662

    Article  Google Scholar 

  32. Hebb, D.O. (1949) The Organization of Behavior (Wiley, New York)

    Google Scholar 

  33. Herz, A., Sulzer, B., Kühn, R., van Hemmen, J.L. (1988) The Hebb rule: Storing static and dynamic objects in an associative neural network. Europhys. Lett. 7:663–669; (1989) Hebbian learning reconsidered: Representation of static and dynamic objects in associative neural nets. Biol. Cybern. 60:457-467

    Article  ADS  Google Scholar 

  34. Personnaz, L., Dreyfus, G., Toulouse, G. (1986) A biologically constrained learning mechanism in networks of formal neurons. J. Stat. Phys. 43:411–422

    Article  MathSciNet  ADS  MATH  Google Scholar 

  35. Personnaz, L., Guyon, I., Dreyfus, G. (1986) Collective computational properties of neural networks: New learning mechanisms. Phys. Rev. A: Math. Gen. 34:4217–4228

    Article  MathSciNet  ADS  Google Scholar 

  36. Palm, G. (1982) Neural Assemblies (Springer-Verlag, Berlin)

    Book  Google Scholar 

  37. Willshaw, D.J., Dayan, P. (1990) Optimal plasticity from matrix memories: What goes up must come down. Neural Comp. 2:85–93

    Article  Google Scholar 

  38. Barto, A.G., Sutton, R.S., Brouwer, P.S. (1981) Associative search network: A reinforcement learning associative memory. Biol. Cybern. 40:201–211

    Article  MATH  Google Scholar 

  39. Lamperti, J. (1966) Probability (Benjamin, New York)

    MATH  Google Scholar 

  40. Shannon, C, Weaver, W. (1949) The Mathematical Theory of Communication (University of Illinois Press, Urbana, IL)

    MATH  Google Scholar 

  41. Palm, G. (1992) On the information storage capacity of local learning rules. Neural Comp. 4:703–711

    Article  Google Scholar 

  42. Gardner, E. (1987) Maximum storage capacity in neural networks. Europhys. Lett. 4:481–485

    Article  ADS  Google Scholar 

  43. Gardner, E. (1988) The space of interactions in neural network models. J. Phys. A: Math. Gen. 21:257–270

    Article  ADS  Google Scholar 

  44. Schwenker, F., Sommer, F.T., Palm, G. (1993) Iterative retrieval of sparsely coded patterns in associative memory. Neuronet′93 Prague

    Google Scholar 

  45. Sommer, F.T. (1993) Theorie neuronaler Assoziativspeicher; Lokales Lernen und iteratives Retrieval von Information. Ph.D. thesis, Düsseldorf

    Google Scholar 

  46. Palm, G., Schwenker, F., Sommer, F.T. (1993) Associative memory and sparse similarity perserving codes. In: From Statistics to Neural Networks: Theory and Pattern Recognition Applications, Cherkassky, V. (Ed.) (Springer NATO ASI Series F) (Springer-Verlag, New York)

    Google Scholar 

  47. Palm, G. (1990) Local learning rules and sparse coding in neural networks. In: Advanced Neural Computers, Eckmiller, R. (Ed.) (Elsevier, Amsterdam), pp. 145–150

    Google Scholar 

  48. Horner, H. (1989) Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons. Z. Phys. B 75:133–136

    Article  ADS  Google Scholar 

  49. Willshaw, D.J., Dayan, P. (1991) Optimizing synaptic learning rules in linear associative memories. Biol. Cybern. 50:253–265

    Google Scholar 

  50. Peretto, P. (1988) On learning rules and memory storage abilities. J. Phys. France 49:711–726

    Article  MathSciNet  Google Scholar 

  51. Palm, G. (1991) Memory capacities of local rules for synaptic modification. Concepts in Neuroscience 2:97–128

    Google Scholar 

  52. Horner, H., Bormann, D., Frick, M., Kinzelbach, H., Schmidt, A. (1989) Transients and basins of attraction in neural network models. Z. Phys. B 76:381–398

    Article  ADS  Google Scholar 

  53. Buhmann, J., Divko, R., Schulten, K. (1989) Associative memory with high information content. Phys. Rev. A 39:2689–2692

    Article  MathSciNet  ADS  Google Scholar 

  54. Palm, G. (1987) Computing with neural networks. Science 235:1227–1228

    Article  ADS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Abteilung Neuroinformatik, Fakultät für Informatik, Universität Ulm, Oberer Eselsberg, D-89081, Ulm, Germany

    Günther Palm

  2. Institut für Medizinische Psychologic und Verhaltensneurobiologic der Universität Tübingen, Gartenstr. 29, D-72074, Tübingen, Germany

    Friedrich T. Sommer

Authors
  1. Günther Palm
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Friedrich T. Sommer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Electronics, Weizmann Institute of Science, 76100, Rehovot, Israel

    Eytan Domany

  2. Institut für Theoretische Physik, Technische Universität München, D-85747, Garching bei München, Germany

    J. Leo van Hemmen

  3. Department of Physics and Beckman Institute, University of Illinois, Urbana, IL, 61801, USA

    Klaus Schulten

Rights and permissions

Reprints and permissions

Copyright information

© 1996 Springer Science+Business Media New York

About this chapter

Cite this chapter

Palm, G., Sommer, F.T. (1996). Associative Data Storage and Retrieval in Neural Networks. In: Domany, E., van Hemmen, J.L., Schulten, K. (eds) Models of Neural Networks III. Physics of Neural Networks. Springer, New York, NY. https://doi.org/10.1007/978-1-4612-0723-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-1-4612-0723-8_3

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4612-6882-6

  • Online ISBN: 978-1-4612-0723-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation