Advertisement

Biological Cybernetics

, Volume 50, Issue 1, pp 51–62 | Cite as

Collective properties of neural networks: A statistical physics approach

  • P. Peretto
Article

Abstract

Among the various models proposed so far to account for the properties of neural networks, the one devised by Little and the one derived by Hopfield prove to be the most interesting because they allow the use of statistical mechanics techniques. The link between tween the Hopfield model and the statistical mechanics is provided by the existence of an extensive quantity. When the synaptic plasticity behaves according to a Hebbian procedure, the analogy with the classical spin glass models studied by Van Hemmen is complete. In particular exact solutions describing the steady states of noisy systems are found. On the other hand, the Little model introduces a Markovian dynamics. One shows that the evolution equation obeys the microreversibility principle if the synaptic efficiencies are symmetrical. Therefore, assuming that such a symmetry materializes, the Little model has to obey a Gibbs statistics. The corresponding Hamiltonian is derived accordingly. At last, using these results, both models are shown to display associative memory properties. In particular the storage capacity of neural networks working along with the Little dynamics is similar to the capacity of Hopfield neural networks. The conclusion drawn from the study of the Hopfield model can be extended to the Little model, which is certainly a more realistic description of the biological situation.

Keywords

Neural Network Spin Glass Associative Memory Gibbs Statistic Realistic Description 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Amari, S.I.: Neural theory of association and concept formation. Biol. Cybern. 26, 175–185 (1977)Google Scholar
  2. Anninos, P.A., Beek, B., Csermely, T.J., Harth, E.H., Pertile, G.: Dynamics of neural structures. J. Theor. Biol. 26, 121–148 (1970)Google Scholar
  3. Binder, K.: In: Fundamental problems in statistical mechanics. Cohen, E.O.M. (ed.). Amsterdam: North-Holland 1981Google Scholar
  4. Braitenberg, V.: Cell assemblies in the cerebral cortex. In: Theoretical approaches to complex systems. Heim, R., Palm, G. (eds.), p. 171. Berlin, Heidelberg, New York: Springer 1978Google Scholar
  5. Caianiello, E.R., de Luca, A., Ricciardi, L.M.: Reverberations and control of neural network. Kybernetik 4, 10–18 (1967)Google Scholar
  6. Choi, M.Y., Huberman, B.A.: Digital dynamics and the simulation of magnetic systems. Phys. Rev. B28, 2547–2554 (1983)Google Scholar
  7. Cooper, L.N.: A possible organization of animal memory and learning. In: Collective properties of physical systems. Nobel Symp. 24, 252–264 (1973)Google Scholar
  8. Feldman, J.L., Cowan, J.D.: Large scale activity in neural theory with application to motoneuron pool responses. Biol. Cybern. 17, 29–38 (1975)Google Scholar
  9. Fukushima, K.: A model of associative memory in the brain. Kybernetik 12, 58–63 (1973)Google Scholar
  10. Glauber, R.J.: Time-dependent statistics of the Ising model. Phys. Rev. 4, 294–307 (1963)Google Scholar
  11. Hebb, D.O.: The organization of behavior. New York: Wiley 1949Google Scholar
  12. Van Hemmen, J.L.: Classical spin-glass model. Phys. Rev. Lett. 49, 409–412 (1982)Google Scholar
  13. Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79, 2554–2558 (1982)Google Scholar
  14. Ill condensed matter: Les Houches summer school-session 1978. Balian, R., Maynard, R., Toulouse, G. (eds.) Amsterdam: North-Holland 1979Google Scholar
  15. Ingber, L.: Statistical mechanics of neocortical interactions. I. Basic formulation. Physica 5D, 83–107 (1982)Google Scholar
  16. Ingber, L.: Statistical mechanics of neocortical interactions. Dynamics of synaptic modifications. Phys. Rev. A8, 395–416 (1983)Google Scholar
  17. Katz, B., Miledi, R.: A study of synaptic transmission in the absence of nerve impulses. J. Physiol. 192, 407–436 (1967)Google Scholar
  18. Kirkpatrick, S., Sherrington, D.: Infinite-ranged models of spinglasses. Phys. Rev. B17, 4384–4403 (1978)Google Scholar
  19. Little, W.A.: The existence of persistent states in the brain. Math. Biosci. 19, 101–120 (1974)Google Scholar
  20. Little, W.A., Shaw, G.L.: Analytic study of the memory storage capacity of a neural network. Math. Biosci. 39, 281–290 (1978)Google Scholar
  21. Mattis, P.C.: Solvable spin systems with random interactions. Phys. Lett. A56, 421–422 (1976)Google Scholar
  22. Von Neuman, J.: The computer and the brain. New Haven, London: Yale University Press 1958Google Scholar
  23. Parasi, G.: Infinite number of order parameters for spin-glasses. Phys. Rev. Lett. 43, 1754–1756 (1979)Google Scholar
  24. Pastur, L.A., Figotin, A.L.: Theory of disordered spin systems. Teor. Mat. Fiz. 35, 193–210 (1978)Google Scholar
  25. Thompson, R.S., Gibson, W.G.: Neural model with probabilistic firing behavior. I. General considerations. Math. Biosci. 56, 239–253 (1981)Google Scholar
  26. Thompson, R.S., Gibson, W.G.: Neural model with probabilistic firing behavior. II. One- and two-neuron networks. Math. Biosci. 56, 255–285 (1981)Google Scholar
  27. Vannimenus, J., Maillard, J.P., De Seze, L.: Ground-state correlations in the two-dimensional Ising frustation model. J. Phys. C 12, 4523–4532 (1979)Google Scholar
  28. Willwacher, G.: Fahigkeiten eines assoziativen Speichersystems im Vergleich zu Gehirnfunktionen. Biol. Cybern. 24, 181–198 (1976)Google Scholar

Copyright information

© Springer-Verlag 1984

Authors and Affiliations

  • P. Peretto
    • 1
  1. 1.Laboratoire Interactions Hyperfines, 85 XCentre d'Etudes Nucléaires de Grenoble DRFGrenoble CedexFrance

Personalised recommendations