Collective properties of neural networks: A statistical physics approach

Abstract

Among the various models proposed so far to account for the properties of neural networks, the one devised by Little and the one derived by Hopfield prove to be the most interesting because they allow the use of statistical mechanics techniques. The link between tween the Hopfield model and the statistical mechanics is provided by the existence of an extensive quantity. When the synaptic plasticity behaves according to a Hebbian procedure, the analogy with the classical spin glass models studied by Van Hemmen is complete. In particular exact solutions describing the steady states of noisy systems are found. On the other hand, the Little model introduces a Markovian dynamics. One shows that the evolution equation obeys the microreversibility principle if the synaptic efficiencies are symmetrical. Therefore, assuming that such a symmetry materializes, the Little model has to obey a Gibbs statistics. The corresponding Hamiltonian is derived accordingly. At last, using these results, both models are shown to display associative memory properties. In particular the storage capacity of neural networks working along with the Little dynamics is similar to the capacity of Hopfield neural networks. The conclusion drawn from the study of the Hopfield model can be extended to the Little model, which is certainly a more realistic description of the biological situation.

This is a preview of subscription content, access via your institution.

References

  1. Amari, S.I.: Neural theory of association and concept formation. Biol. Cybern. 26, 175–185 (1977)

    Google Scholar 

  2. Anninos, P.A., Beek, B., Csermely, T.J., Harth, E.H., Pertile, G.: Dynamics of neural structures. J. Theor. Biol. 26, 121–148 (1970)

    Google Scholar 

  3. Binder, K.: In: Fundamental problems in statistical mechanics. Cohen, E.O.M. (ed.). Amsterdam: North-Holland 1981

    Google Scholar 

  4. Braitenberg, V.: Cell assemblies in the cerebral cortex. In: Theoretical approaches to complex systems. Heim, R., Palm, G. (eds.), p. 171. Berlin, Heidelberg, New York: Springer 1978

    Google Scholar 

  5. Caianiello, E.R., de Luca, A., Ricciardi, L.M.: Reverberations and control of neural network. Kybernetik 4, 10–18 (1967)

    Google Scholar 

  6. Choi, M.Y., Huberman, B.A.: Digital dynamics and the simulation of magnetic systems. Phys. Rev. B28, 2547–2554 (1983)

    Google Scholar 

  7. Cooper, L.N.: A possible organization of animal memory and learning. In: Collective properties of physical systems. Nobel Symp. 24, 252–264 (1973)

    Google Scholar 

  8. Feldman, J.L., Cowan, J.D.: Large scale activity in neural theory with application to motoneuron pool responses. Biol. Cybern. 17, 29–38 (1975)

    Google Scholar 

  9. Fukushima, K.: A model of associative memory in the brain. Kybernetik 12, 58–63 (1973)

    Google Scholar 

  10. Glauber, R.J.: Time-dependent statistics of the Ising model. Phys. Rev. 4, 294–307 (1963)

    Google Scholar 

  11. Hebb, D.O.: The organization of behavior. New York: Wiley 1949

    Google Scholar 

  12. Van Hemmen, J.L.: Classical spin-glass model. Phys. Rev. Lett. 49, 409–412 (1982)

    Google Scholar 

  13. Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79, 2554–2558 (1982)

    Google Scholar 

  14. Ill condensed matter: Les Houches summer school-session 1978. Balian, R., Maynard, R., Toulouse, G. (eds.) Amsterdam: North-Holland 1979

    Google Scholar 

  15. Ingber, L.: Statistical mechanics of neocortical interactions. I. Basic formulation. Physica 5D, 83–107 (1982)

    Google Scholar 

  16. Ingber, L.: Statistical mechanics of neocortical interactions. Dynamics of synaptic modifications. Phys. Rev. A8, 395–416 (1983)

    Google Scholar 

  17. Katz, B., Miledi, R.: A study of synaptic transmission in the absence of nerve impulses. J. Physiol. 192, 407–436 (1967)

    Google Scholar 

  18. Kirkpatrick, S., Sherrington, D.: Infinite-ranged models of spinglasses. Phys. Rev. B17, 4384–4403 (1978)

    Google Scholar 

  19. Little, W.A.: The existence of persistent states in the brain. Math. Biosci. 19, 101–120 (1974)

    Google Scholar 

  20. Little, W.A., Shaw, G.L.: Analytic study of the memory storage capacity of a neural network. Math. Biosci. 39, 281–290 (1978)

    Google Scholar 

  21. Mattis, P.C.: Solvable spin systems with random interactions. Phys. Lett. A56, 421–422 (1976)

    Google Scholar 

  22. Von Neuman, J.: The computer and the brain. New Haven, London: Yale University Press 1958

    Google Scholar 

  23. Parasi, G.: Infinite number of order parameters for spin-glasses. Phys. Rev. Lett. 43, 1754–1756 (1979)

    Google Scholar 

  24. Pastur, L.A., Figotin, A.L.: Theory of disordered spin systems. Teor. Mat. Fiz. 35, 193–210 (1978)

    Google Scholar 

  25. Thompson, R.S., Gibson, W.G.: Neural model with probabilistic firing behavior. I. General considerations. Math. Biosci. 56, 239–253 (1981)

    Google Scholar 

  26. Thompson, R.S., Gibson, W.G.: Neural model with probabilistic firing behavior. II. One- and two-neuron networks. Math. Biosci. 56, 255–285 (1981)

    Google Scholar 

  27. Vannimenus, J., Maillard, J.P., De Seze, L.: Ground-state correlations in the two-dimensional Ising frustation model. J. Phys. C 12, 4523–4532 (1979)

    Google Scholar 

  28. Willwacher, G.: Fahigkeiten eines assoziativen Speichersystems im Vergleich zu Gehirnfunktionen. Biol. Cybern. 24, 181–198 (1976)

    Google Scholar 

Download references

Author information

Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Peretto, P. Collective properties of neural networks: A statistical physics approach. Biol. Cybern. 50, 51–62 (1984). https://doi.org/10.1007/BF00317939

Download citation

Keywords

  • Neural Network
  • Spin Glass
  • Associative Memory
  • Gibbs Statistic
  • Realistic Description