Skip to main content
SpringerLink
Log in

A Comparative Study of Sparse Associative Memories

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

We study various models of associative memories with sparse information, i.e. a pattern to be stored is a random string of 0s and 1s with about \(\log N\) 1s, only. We compare different synaptic weights, architectures and retrieval mechanisms to shed light on the influence of the various parameters on the storage capacity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Aliabadi, B.K., Berrou, C., Gripon, V., Jiang, X.: Storing sparse messages in networks of neural cliques. IEEE Trans. Neural Netw. Learn. Syst. 25, 980–989 (2014)

    Article  Google Scholar 

  2. Bollé, D., Verbeiren, T.: Thermodynamics of fully connected Blume-Emery-Griffiths neural networks. J. Phys. A: Math. Gen. 36(6), 295–305 (2003)

    Article  ADS  MATH  MathSciNet  Google Scholar 

  3. Boutsikas, M.V., Koutras, M.V.: A bound for the distribution of the sum of discrete associated or negatively associated random variables. Ann. Appl. Probab. 10(4), 1137–1150 (2000)

    MATH  MathSciNet  Google Scholar 

  4. Bovier, A.: Sharp upper bounds on perfect retrieval in the Hopfield model. J. Appl. Probab. 36(3), 941–950 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  5. Burshtein, D.: Nondirect convergence radius and number of iterations of the Hopfield associative memory. IEEE Trans. Inf. Theory 40(3), 838–847 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  6. Esary, J.D., Proschan, F., Walkup, D.W.: Association of random variables, with applications. Ann. Math. Stat. 38, 1466–1474 (1967)

    Article  MATH  MathSciNet  Google Scholar 

  7. Gripon, V., Berrou, C.: Sparse neural networks with large learning diversity. IEEE Trans. Neural Netw. 22(7), 1087–1096 (2011)

    Article  Google Scholar 

  8. Heusel, J., Löwe, M., Vermet, F.: On the capacity of an associative memory model based on neural cliques. Stat. Probab. Lett. 106, 256–261 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  9. Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79(8), 2554–2558 (1982)

    Article  ADS  MathSciNet  Google Scholar 

  10. Amari, S.I.: Characteristics of sparsely encoded associative memory. Neural Netw. 2(6), 451–457 (1989)

    Article  Google Scholar 

  11. Jarollahi, H., Gripon, V., Onizawa, N., Gross, W.J.: Algorithm and architecture for a low-power content-addressable memory based on sparse-clustered networks. IEEE Trans. Very Large Scale Integr. Syst. 27(2), 375–387 (2016)

    Google Scholar 

  12. Jarollahi, H., Onizawa, N., Gripon, V., Gross, W.J.: Algorithm and architecture of fully-parallel associative memories based on sparse clustered networks. J. Signal Process. Syst. 76(3), 235–247 (2014)

  13. Jiang, X., Gripon, V., Berrou, C., Rabbat, M.: Storing sequences in binary tournament-based neural networks. IEEE Trans. Neural Netw. Learn. Syst. 27(5), 913–925 (2016)

    Article  Google Scholar 

  14. Löwe, M.: On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8(4), 1216–1250 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  15. Löwe, M.: On the storage capacity of the Hopfield model with biased patterns. IEEE Trans. Inf. Theory 45(1), 314–318 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  16. Löwe, M., Vermet, F.: The storage capacity of the Blume-Emery-Griffiths neural network. J. Phys. A 38(16), 3483–3503 (2005)

    Article  ADS  MATH  MathSciNet  Google Scholar 

  17. Löwe, M., Vermet, F.: The capacity of \(q\)-state Potts neural networks with parallel retrieval dynamics. Stat. Probab. Lett. 77(14), 1505–1514 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  18. Löwe, M., Vermet, F.: Capacity of an associative memory model on random graph architectures. Bernoulli 21(3), 1884–1910 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  19. McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of the Hopfield associative memory. IEEE Trans. Inf. Theory 33(4), 461–482 (1987)

    Article  MATH  MathSciNet  Google Scholar 

  20. Okada, M.: Notions of associative memory and sparse coding. Four major hypotheses in neuroscience. Neural Netw. 9(8), 1429–1458 (1996)

    Article  MATH  Google Scholar 

  21. Palm, G.: On associative memory. Biol. Cybern. 36(1), 19–31 (1980)

    Article  MATH  Google Scholar 

  22. Palm, G.: Neural associative memories and sparse coding. Neural Netw., 37(0), 165–171 (2013) (Twenty-fifth Anniversay Commemorative Issue)

  23. Schwenker, F., Sommer, F., Palm, G.: Iterative retrieval of sparsely coded associative memory patterns. Neural Netw. 9(3), 445–455 (1996)

    Article  Google Scholar 

  24. Willshaw, D.J., Buneman, O.P., Longuet-Higgins, H.C.: Non-holographic associative memory. Nature 222, 960–962 (1969)

    Article  ADS  Google Scholar 

  25. Yao, Z., Gripon, V., Rabbat, M.: A GPU-based Associative Memory using Sparse Neural Networks. In: Proceedings of the PCNN-14 conference, pp. 688–692 (2014)

Download references

Acknowledgments

We are very grateful to two anonymous referees for a very careful reading of a first version of the manuscript and valuable remarks that helped to improve its readability significantly.

Author information

Authors and Affiliations

  1. Telecom Bretagne UMR CNRS Lab-STICC, Technopole Brest Iroise, 29238, Brest, France

    Vincent Gripon

  2. Fachbereich Mathematik und Informatik, University of Münster, Einsteinstraße 62, 48149, Münster, Germany

    Judith Heusel & Matthias Löwe

  3. Laboratoire de Mathématiques, UMR CNRS 6205, Université de Bretagne Occidentale, 6, Avenue Victor Le Gorgeu, CS 93837, 29238, Brest Cedex 3, France

    Franck Vermet

Authors
  1. Vincent Gripon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Judith Heusel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthias Löwe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Franck Vermet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Judith Heusel.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gripon, V., Heusel, J., Löwe, M. et al. A Comparative Study of Sparse Associative Memories. J Stat Phys 164, 105–129 (2016). https://doi.org/10.1007/s10955-016-1530-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-016-1530-z

Keywords

Mathematics Subject Classification

Search

Navigation