Skip to main content

On a Model of Associative Memory with Huge Storage Capacity

Abstract

In Krotov et al. (in: Lee (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., Red Hook, 2016) Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the ”limit” as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model.

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Spin-glass models of neural networks. Phys. Rev. A. 32(2), 1007–1018 (1985a). doi:10.1103/PhysRevA.32.1007

    ADS  MathSciNet  Article  Google Scholar 

  2. 2.

    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55, 1530–1533 (1985b). doi:10.1103/PhysRevLett.55.1530

    ADS  Article  Google Scholar 

  3. 3.

    Bovier, A.: Sharp upper bounds on perfect retrieval in the Hopfield model. J. Appl. Probab. 36(3), 941–950 (1999)

    MathSciNet  Article  MATH  Google Scholar 

  4. 4.

    Bovier, A., Niederhauser, B.: The spin-glass phase-transition in the Hopfield model with \(p\)-spin interactions. Adv. Theor. Math. Phys. 5(6), 1001–1046 (2001). doi:10.4310/ATMP.2001.v5.n6.a2

    MathSciNet  Article  MATH  Google Scholar 

  5. 5.

    Dembo, A., Zeitouni, O.: Large deviations techniques and applications. Stochastic Modelling and Applied Probability, vol. 38, p. 396. Springer, Berlin (2010) Corrected reprint of the second (1998) edition. ISBN 978-3-642-03310-0. doi:10.1007/978-3-642-03311-7

  6. 6.

    Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79(8), 2554–2558 (1982)

    ADS  MathSciNet  Article  Google Scholar 

  7. 7.

    Krotov, D., Hopfield, J.J.: Dense associative memory for pattern recognition. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 1172–1180. Curran Associates, Inc., Red Hook (2016)

    Google Scholar 

  8. 8.

    Loukianova, D.: Lower bounds on the restitution error in the Hopfield model. Probab. Theory Relat. Fields 107(2), 161–176 (1997). doi:10.1007/s004400050081

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Löwe, M.: The storage capacity of generalized Hopfield models with semantically correlated patterns. Markov Process. Relat. Fields 5(1), 1–19 (1999)

    MathSciNet  MATH  Google Scholar 

  10. 10.

    Löwe, M.: On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8(4), 1216–1250 (1998). doi:10.1214/aoap/1028903378

    MathSciNet  Article  MATH  Google Scholar 

  11. 11.

    Löwe, M., Vermet, F.: The storage capacity of the Hopfield model and moderate deviations. Stat. Probab. Lett. 75(4), 237–248 (2005). doi:10.1016/j.spl.2005.06.001

    MathSciNet  Article  MATH  Google Scholar 

  12. 12.

    Löwe, M., Vermet, F.: The capacity of \(q\)-state Potts neural networks with parallel retrieval dynamics. Stat. Probab. Lett. 77(14), 1505–1514 (2007). doi:10.1016/j.spl.2007.03.030

    MathSciNet  Article  MATH  Google Scholar 

  13. 13.

    McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33(4), 461–482 (1987). doi:10.1109/TIT.1987.1057328

    MathSciNet  Article  MATH  Google Scholar 

  14. 14.

    Newman, C.M.: Memory capacity in neural network models: rigorous lower bounds. Neural Netw. 1(3), 223–238 (1988)

    Article  Google Scholar 

  15. 15.

    Talagrand, M.: Rigorous results for the Hopfield model with many patterns. Probab. Theory Relat. Fields 110(2), 177–276 (1998). doi:10.1007/s004400050148

    MathSciNet  Article  MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Franck Vermet.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Demircigil, M., Heusel, J., Löwe, M. et al. On a Model of Associative Memory with Huge Storage Capacity. J Stat Phys 168, 288–299 (2017). https://doi.org/10.1007/s10955-017-1806-y

Download citation

Keywords

  • Neural networks
  • Associative memory
  • Hopfield model
  • Exponential inequalities