Hopfield Models as Generalized Random Mean Field Models

  • Anton Bovier
  • Véronique Gayrard
Part of the Progress in Probability book series (PRPR, volume 41)

Abstract

Twenty years ago, Pastur and Figotin [FP1,FP2] first introduced and studied what has become known as the Hopfield model and which turned out, over the years, to be one of the more successful and important models of a disordered system. This is also reflected in the fact that several contributions in this book are devoted to it. The Hopfield model is quite versatile and models various situations. Pastur and Figotin introduced it as a simple model for a spin glass, and, in 1982, Hopfield independently considered it as a model for associative memory.

Keywords

Entropy Covariance lIMe Convolution Stein 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [A]
    D.J. Amit, Modelling brain function, Cambridge University Press, Cambridge, 1989.Google Scholar
  2. [AGS]
    D.J. Amit, H. Gutfreund and H. Sompolinsky, Statistical mechanics of neural networks near saturation, Ann. Phys. 173, 30–67 (1987).CrossRefGoogle Scholar
  3. [AGS2]
    D.J. Amit, H. Gutfreund, and H. Sompolinsky, Spin-glass model of neural network, Phys. Rev. A 32, 1007–1018 (1985).MathSciNetGoogle Scholar
  4. [B]
    A. Bovier, Self-averaging in a class of generalized Hopfield models, J. Phys. A 27, 7069–7077 (1994).MathSciNetGoogle Scholar
  5. [BG1]
    A. Bovier and V. Gayrard, Rigorous bounds on the storage capacity for the dilute Hopfield model, J. Stat. Phys. 69, 597–627 (1992)MathSciNetMATHCrossRefGoogle Scholar
  6. [BG2]
    A. Bovier and V. Gayrard, Rigorous results on the thermodynamics of the dilute Hopfield model, J. Stat. Phys. 72, 643–664 (1993).MathSciNetMATHCrossRefGoogle Scholar
  7. [BG3]
    A. Bovier and V. Gayrard, Rigorous results on the Hopfield model of neural networks, Resenhas do IME-USP 2, 161–172 (1994).MathSciNetGoogle Scholar
  8. [BG4]
    A. Bovier and V. Gayrard, An almost sure large deviation principle for the Hopfield model, Ann. Probab. 24, 1444–1475 (1996).MathSciNetMATHCrossRefGoogle Scholar
  9. [BG5]
    A. Bovier and V. Gayrard, The retrieval phase of the Hopfield model, A rigorous analysis of the overlap distribution, Probab. Theor. Rel. Fields 107, 61–98 (1995).MathSciNetCrossRefGoogle Scholar
  10. [BG6]
    A. Bovier and V. Gayrard, Metatstates in the Hopfield model in the replica symmetric regime, preprint (1997).Google Scholar
  11. [BGP1]
    A. Bovier, V. Gayrard, and P. Picco, Gibbs states of the Hopfield model in the regime of perfect memory, Probab. Theor. Rel. Fields 100, 329–363 (1994).MathSciNetMATHCrossRefGoogle Scholar
  12. [BGP2]
    A. Bovier, V. Gayrard, and P. Picco, Large deviation principles for the Hopfield model and the Kac-Hopfield model, Probab. Theor. Rel. Fields 101, 511–546 (1995).MathSciNetMATHCrossRefGoogle Scholar
  13. [BGP3]
    A. Bovier, V. Gayrard, and P. Picco, Gibbs states of the Hopfield model with extensively many patterns, J. Stat. Phys. 79, 395–414 (1995).MathSciNetMATHCrossRefGoogle Scholar
  14. [BGP4]
    A. Bovier, V. Gayrard, and P. Picco, Distribution of overlap profiles in the one-dimensional Kac-Hopfield model, WIAS-preprint 221, to appear in Commun. Math. Phys. (1997).Google Scholar
  15. [BF]
    A. Bovier and J. Fröhlich, A heuristic theory of the spin glass phase, J. Stat. Phys. 44, 347–391 (1986).CrossRefGoogle Scholar
  16. [BL]
    H.J. Brascamp and E.H. Lieb, On extensions of the Brunn-Minkowski and Pékopa-Leindler theorems, including inequalities for log concave functions, and with an application to the diffusion equation, J. Functional Anal. 22, 366–389 (1976).MathSciNetMATHCrossRefGoogle Scholar
  17. [Co]
    F. Comets, Large deviation estimates for a conditional probability distribution. Applications to random Gibbs measures, Probab. Theor. Rel. Fields 80, 407–432 (1989).MathSciNetMATHCrossRefGoogle Scholar
  18. [CT]
    Y.S. Chow and H. Teicher, Probability Theory, 2nd edition, Springer, Berlin-Heidelberg-New York, 1988.MATHGoogle Scholar
  19. [DHS]
    E. Domany, J.L. van Hemmen, and K. Schulten (eds.), Models of Neural Networks, Springer-Verlag, Berlin, 1991.MATHGoogle Scholar
  20. [DS]
    J.-D. Deuschel and D. Stroock, Large Deviations, Academic Press, Boston, 1989.MATHGoogle Scholar
  21. [DZ]
    A. Dembo and O. Zeitouni, Large Deviation Techniques and Applications, Jones and Bartlett, Boston, 1992.Google Scholar
  22. [El]
    R.S. Ellis, Entropy, Large Deviations, and Statistical Mechanics, Springer-Verlag, Berlin, 1985.MATHGoogle Scholar
  23. [EA]
    S.F. Edwards and P.W. Anderson, Theory of spin glasses, J. Phys. F 5, 965–974 (1975).CrossRefGoogle Scholar
  24. [EE]
    T. Eisele and R.S. Ellis, Multiple phase transitions in the generalized Curie-Weiss model, J. Stat Phys. 52, 161–202 (1988).MathSciNetMATHCrossRefGoogle Scholar
  25. [vE]
    A.C.D. van Enter, Stiffness exponent, number of pure states, and Almeida-Thouless line in spin glasses, J. Stat. Phys. 60, 275–279 (1990).MATHCrossRefGoogle Scholar
  26. [vEvHP]
    A.C.D. van Enter, J.L. van Hemmen and C. Pospiech, Mean-field theory of random-site q-state Potts models, J. Phys. A 21, 791–801 (1988).Google Scholar
  27. [FH]
    D.S. Fisher and D.A. Huse, Pure phases in spin glasses, J. Phys. A 20, L997-L1003 (1987)MathSciNetGoogle Scholar
  28. D.S. Fisher and D.A. Huse, Absence of many states in magnetic spin glasses, J. Phys. A 20, L1005-L1010 (1987).MathSciNetGoogle Scholar
  29. [FP1]
    L.A. Pastur and A.L. Figotin, Exactly soluble model of a spin glass, Sov. J. Low Temp. Phys. 3(6), 378–383 (1977).Google Scholar
  30. [FP2]
    L.A. Pastur and A.L. Figotin, On the theory of disordered spin systems, Theor. Math. Phys. 35, 403–414 (1978).MathSciNetCrossRefGoogle Scholar
  31. [FP2]
    L.A. Pastur and A.L. Figotin, Infinite range limit for a class of disordered spin systems, Theor. Math. Phys. 51, 564–569 (1982).MathSciNetCrossRefGoogle Scholar
  32. [FZ]
    C. Fassnacht and A. Zippelius, Recognition and Categorization in a structured neural network with attractor dynamics, Network 2, 63–84 (1991).MathSciNetMATHCrossRefGoogle Scholar
  33. [Gl]
    V. Gayrard, The thermodynamic limit of the g-state Potts-Hopfield model with infinitely many patterns, J. Stat. Phys. 68, 977–1011 (1992).MathSciNetMATHCrossRefGoogle Scholar
  34. [Ge]
    S. Geman, A limit theorem for the norms of random matrices, Ann. Probab. 8, 252–261 (1980).MathSciNetMATHCrossRefGoogle Scholar
  35. [Geo]
    H.O. Georgii, Gibbs Measures and Phase Transitions, Walter de Gruyter (de Gruyter Studies in Mathematics, Vol. 19), Berlin-New York, 1988.MATHCrossRefGoogle Scholar
  36. [Gi]
    V.L. Girko, Limit theorems for maximal and minimal eigenvalues of random matrices, Theor. Probab. Appl. 35, 680–695 (1989).MathSciNetCrossRefGoogle Scholar
  37. [GM]
    E. Golez and S. Martínez, Neural and Automata Networks, Kluwer Academic Publ., Dodrecht, 1990.Google Scholar
  38. [HKP]
    J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City, 1991.Google Scholar
  39. [Ho]
    J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA 79, 2554–2558 (1982).MathSciNetCrossRefGoogle Scholar
  40. [HS]
    R.L. Stratonovich, On a method of calculating quantum distribution functions, Doklady Akad. Nauk S.S.S.R. 115, 1097 (1957) [translation: Soviet Phys. Doklady 2, 416–419 (1958)Google Scholar
  41. J. Hubbard, Calculation of partition functions, Phys. Rev. Lett. 3, 77–78 (1959).CrossRefGoogle Scholar
  42. [JK]
    J. Jedrzejewski and A. Kombda, On equivalent-neighbour, random-site models of disordered systems, Z. Phys. B 63, 247–257 (1986).CrossRefGoogle Scholar
  43. [vH1]
    J.L. van Hemmen, Equilibrium theory of spin-glasses: mean-field theory and beyond, in Heidelberg Colloquium on Spin Glasses, eds. J.L. van Hemmen and I. Morgenstern, 203–233 (1983), LNP 192 Springer, Berlin-Heidelberg-New York, 1983.CrossRefGoogle Scholar
  44. [vH2]
    J.L. van Hemmen, Spin glass models of a neural network, Phys. Rev. A 34, 3435–3445 (1986).MathSciNetCrossRefGoogle Scholar
  45. [vHGHK]
    J.L. van Hemmen, D. Grensing, A. Huber and R. Kühn, Elementary solution of classical spin-glass models, Z. Phys. B 65, 53–63 (1986).CrossRefGoogle Scholar
  46. [vHvE]
    J.L. van Hemmen and A.C.D. van Enter, Chopper model for pattern recognition, Phys. Rev. A 34, 2509–2512 (1986).CrossRefGoogle Scholar
  47. [vHvEC]
    J.L. van Hemmen, A.C.D. van Enter, and J. Canisius. On a classical spin-glass model, Z. Phys. B 50, 311–336 (1983).CrossRefGoogle Scholar
  48. [K]
    H. Koch, A free energy bound for the Hopfield model, J. Phys. A 26, L353-L355 (1993).Google Scholar
  49. [KP]
    H. Koch and J. Piasko, Some rigorous results on the Hopfield neural network model, J. Stat. Phys. 55, 903–928 (1989).MathSciNetMATHCrossRefGoogle Scholar
  50. [Ku]
    Ch. Külske, private communication.Google Scholar
  51. [LT]
    M. Ledoux and M. Talagrand, Probability in Banach Spaces, Springer, Berlin-Heidelberg-New York, 1991.MATHGoogle Scholar
  52. [Lu]
    D. Loukianova, Two rigorous bounds in the Hopfield model of associative memory, to appear in Probab. Theor. Rel. Fields (1996).Google Scholar
  53. [Lut]
    J.M. Luttinger, Exactly soluble spin-glass model, Phys. Rev. Lett. 37, 778–782 (1976).MathSciNetCrossRefGoogle Scholar
  54. [Mar]
    S. Martinez, Introduction to neural networks, preprint, Temuco, (1992).Google Scholar
  55. [Ma]
    D.C. Mattis, Solvable spin system with random interactions, Phys. Lett. 56A, 421–422 (1976).Google Scholar
  56. [McE]
    R.J. McEliece, E.C. Posner, E.R. Rodemich, and S.S. Venkatesh, The capacity of the Hopfield associative memory, IEEE Trans. Inform. Theory 33, 461–482 (1987).MathSciNetMATHCrossRefGoogle Scholar
  57. [Mi]
    Y. Miyashita, Neuronal correlate of visual associative long term memory in the primate temporal cortex, Nature 335, 817–819 (1988).CrossRefGoogle Scholar
  58. [MPR]
    E. Marinari, G. Parisi, and F. Ritort, On the 3D Ising spin glass, J. Phys. A 27, 2687–2708 (1994).MathSciNetGoogle Scholar
  59. [MPV]
    M. Mézard, G. Parisi, and M.A. Virasoro, Spin-Glass Theory and Beyond, World Scientific, Singapore, 1988.Google Scholar
  60. [MR]
    B. Müller and J. Reinhardt, Neural Networks: An Introduction, Springer-Verlag, Berlin, 1990.MATHGoogle Scholar
  61. [MS]
    V.A. Malyshev and F.M. Spieksma, Dynamics of binary neural networks with a finite number of patterns. Part 1: General picture of the asynchronous zero temperature dynamics, MPEJ 3, 1–36 (1997).MathSciNetGoogle Scholar
  62. [N]
    Ch.M. Newman, Memory capacity in neural network models: Rigorous results, Neural Networks 1, 223–238 (1988).CrossRefGoogle Scholar
  63. [NS]
    Ch.M. Newman and D.L. Stein, Non-mean-field behavior in realistic spin glasses, Phys. Rev. Lett. 76, 515–518 (1996);CrossRefGoogle Scholar
  64. Ch.M. Newman and D.L. Stein, Spatial inhomogeneity and thermodynamic chaos, Phys. Rev. Lett. 76, 4821–4824 (1996);CrossRefGoogle Scholar
  65. Ch.M. Newman and D.L. Stein, Topies in Disordered Systems, to appear in Birkhäuser, Boston, 1997; Thermodynamic Chaos and the Structure of Short Range Spin Glasses, this volume.CrossRefGoogle Scholar
  66. [P]
    D. Petritis, Thermodynamic formalism of neural computing, preprint, Université de Rennes, 1995.Google Scholar
  67. [PS]
    L. Pastur and M. Shcherbina, Absence of self-averaging of the order parameter in the Sherrington-Kirkpatrick model, J. Stat. Phys. 62, 1–19 (1991).MathSciNetCrossRefGoogle Scholar
  68. [PST]
    L. Pastur, M. Shcherbina, and B. Tirozzi, The replica symmetric solution without the replica trick for the Hopfield model, J. Stat. Phys. 74, 1161–1183 (1994).MathSciNetMATHCrossRefGoogle Scholar
  69. [PN]
    P. Peretto and J.J. Niez, Long term memory storage capacity of multiconnected neural networks, Biolog. Cybernetics 39, 53–63 (1986).CrossRefGoogle Scholar
  70. [Ro]
    R. T. Rockafellar, Convex Analysis, Princeton University Press, Princeton, 1970.MATHGoogle Scholar
  71. [RV]
    A.W. Roberts and D.E. Varberg, Convex Functions, Academic Press, New York and London, 1973.MATHGoogle Scholar
  72. [SK]
    D. Sherrington and S. Kirkpatrick, Solvable model of a spin glass, Phys. Rev. Lett. 35, 1792–1796 (1972).CrossRefGoogle Scholar
  73. [Si]
    J. Silverstein, Eigenvalues and eigenvectors of large sample covariance matrices, Contemporary Math. 50, 153–159 (1986).MathSciNetGoogle Scholar
  74. [Sn]
    A.-S. Snitzman, Equations de type Boltzmann spatialement homogènes, Z. Wahrscheinlichkeitstheorie verw. Gebiete 66, 559–592 (1986).Google Scholar
  75. [ST]
    M. Shcherbina and B. Tirozzi, The free energy for a class of Hopfield models, J. Stat Phys. 72, 113–125 (1992).MathSciNetCrossRefGoogle Scholar
  76. [SW]
    M. Schlüter and E. Wagner, Phys. Rev. E49, 1690 (1994).Google Scholar
  77. [Yu]
    V.V. Yurinskii, Exponential inequalities for sums of random vectors, J. Multivariate Anal. 6, 473–499 (1976).MathSciNetMATHCrossRefGoogle Scholar
  78. [Tl]
    M. Talagrand, Concentration of measure and isoperimetric inequalities in product space, Publ. Math. I.H.E.S., 81, 73–205 (1995).MathSciNetMATHGoogle Scholar
  79. [T2]
    M. Talagrand, A new look at independence, Ann. Probab. 24, 1–34 (1996).MathSciNetMATHCrossRefGoogle Scholar
  80. [T3]
    M. Talagrand, Résultats rigoureux pour le modèle de Hopfield, C. R. Acad. Sci. Paris, t. 321, Série I, 109–112 (1995).MathSciNetGoogle Scholar
  81. [T4]
    M. Talagrand, Rigorous results for the Hopfield model with many patterns, preprint 1996, to appear in Probab. Theor. Rel. Fields.Google Scholar
  82. [T5]
    M. Talagrand, The Sherrington-Kirkpatrick model: A challenge for mathematicians, preprint (1996), to appear in Probab. Theor. Rel. Fields.Google Scholar
  83. [TAP]
    D.J. Thouless, P.W. Anderson, and R.G. Palmer, Phil Mag. 35, 593 (1977).CrossRefGoogle Scholar
  84. [YBK]
    Y.Q. Yin, Z.D. Bai, and P.R. Krishnaiah, On the limit of the largest eigenvalue of the large dimensional sample covariance matrix, Probab. Theor. Rel. Fields 78, 509–521 (1988).MathSciNetMATHCrossRefGoogle Scholar
  85. [Yu]
    V.V. Yurinskii, Exponential inequalities for sums of random vectors, J. Multivariate Anal. 6, 473–499 (1976)MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Birkhäuser Boston 1998

Authors and Affiliations

  • Anton Bovier
    • 1
  • Véronique Gayrard
    • 2
  1. 1.Weierstrass-Insitut für Angewandte Analysis und StochastikBerlinGermany
  2. 2.Centre de Physique Théorique-CNRSMarseille Cedex 9France

Personalised recommendations