Advertisement

Journal of Statistical Physics

, Volume 164, Issue 1, pp 105–129 | Cite as

A Comparative Study of Sparse Associative Memories

  • Vincent Gripon
  • Judith Heusel
  • Matthias Löwe
  • Franck Vermet
Article

Abstract

We study various models of associative memories with sparse information, i.e. a pattern to be stored is a random string of 0s and 1s with about \(\log N\) 1s, only. We compare different synaptic weights, architectures and retrieval mechanisms to shed light on the influence of the various parameters on the storage capacity.

Keywords

Neural networks Associative memory Sparse patterns Storage capacity Exponential inequalities 

Mathematics Subject Classification

Primary: 82C32 60K35 Secondary: 68T05 92B20 

Notes

Acknowledgments

We are very grateful to two anonymous referees for a very careful reading of a first version of the manuscript and valuable remarks that helped to improve its readability significantly.

References

  1. 1.
    Aliabadi, B.K., Berrou, C., Gripon, V., Jiang, X.: Storing sparse messages in networks of neural cliques. IEEE Trans. Neural Netw. Learn. Syst. 25, 980–989 (2014)CrossRefGoogle Scholar
  2. 2.
    Bollé, D., Verbeiren, T.: Thermodynamics of fully connected Blume-Emery-Griffiths neural networks. J. Phys. A: Math. Gen. 36(6), 295–305 (2003)ADSCrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Boutsikas, M.V., Koutras, M.V.: A bound for the distribution of the sum of discrete associated or negatively associated random variables. Ann. Appl. Probab. 10(4), 1137–1150 (2000)zbMATHMathSciNetGoogle Scholar
  4. 4.
    Bovier, A.: Sharp upper bounds on perfect retrieval in the Hopfield model. J. Appl. Probab. 36(3), 941–950 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Burshtein, D.: Nondirect convergence radius and number of iterations of the Hopfield associative memory. IEEE Trans. Inf. Theory 40(3), 838–847 (1994)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Esary, J.D., Proschan, F., Walkup, D.W.: Association of random variables, with applications. Ann. Math. Stat. 38, 1466–1474 (1967)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Gripon, V., Berrou, C.: Sparse neural networks with large learning diversity. IEEE Trans. Neural Netw. 22(7), 1087–1096 (2011)CrossRefGoogle Scholar
  8. 8.
    Heusel, J., Löwe, M., Vermet, F.: On the capacity of an associative memory model based on neural cliques. Stat. Probab. Lett. 106, 256–261 (2015)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79(8), 2554–2558 (1982)ADSCrossRefMathSciNetGoogle Scholar
  10. 10.
    Amari, S.I.: Characteristics of sparsely encoded associative memory. Neural Netw. 2(6), 451–457 (1989)CrossRefGoogle Scholar
  11. 11.
    Jarollahi, H., Gripon, V., Onizawa, N., Gross, W.J.: Algorithm and architecture for a low-power content-addressable memory based on sparse-clustered networks. IEEE Trans. Very Large Scale Integr. Syst. 27(2), 375–387 (2016)Google Scholar
  12. 12.
    Jarollahi, H., Onizawa, N., Gripon, V., Gross, W.J.: Algorithm and architecture of fully-parallel associative memories based on sparse clustered networks. J. Signal Process. Syst. 76(3), 235–247 (2014)Google Scholar
  13. 13.
    Jiang, X., Gripon, V., Berrou, C., Rabbat, M.: Storing sequences in binary tournament-based neural networks. IEEE Trans. Neural Netw. Learn. Syst. 27(5), 913–925 (2016)CrossRefGoogle Scholar
  14. 14.
    Löwe, M.: On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8(4), 1216–1250 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Löwe, M.: On the storage capacity of the Hopfield model with biased patterns. IEEE Trans. Inf. Theory 45(1), 314–318 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Löwe, M., Vermet, F.: The storage capacity of the Blume-Emery-Griffiths neural network. J. Phys. A 38(16), 3483–3503 (2005)ADSCrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Löwe, M., Vermet, F.: The capacity of \(q\)-state Potts neural networks with parallel retrieval dynamics. Stat. Probab. Lett. 77(14), 1505–1514 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Löwe, M., Vermet, F.: Capacity of an associative memory model on random graph architectures. Bernoulli 21(3), 1884–1910 (2015)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of the Hopfield associative memory. IEEE Trans. Inf. Theory 33(4), 461–482 (1987)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Okada, M.: Notions of associative memory and sparse coding. Four major hypotheses in neuroscience. Neural Netw. 9(8), 1429–1458 (1996)CrossRefzbMATHGoogle Scholar
  21. 21.
    Palm, G.: On associative memory. Biol. Cybern. 36(1), 19–31 (1980)CrossRefzbMATHGoogle Scholar
  22. 22.
    Palm, G.: Neural associative memories and sparse coding. Neural Netw., 37(0), 165–171 (2013) (Twenty-fifth Anniversay Commemorative Issue)Google Scholar
  23. 23.
    Schwenker, F., Sommer, F., Palm, G.: Iterative retrieval of sparsely coded associative memory patterns. Neural Netw. 9(3), 445–455 (1996)CrossRefGoogle Scholar
  24. 24.
    Willshaw, D.J., Buneman, O.P., Longuet-Higgins, H.C.: Non-holographic associative memory. Nature 222, 960–962 (1969)ADSCrossRefGoogle Scholar
  25. 25.
    Yao, Z., Gripon, V., Rabbat, M.: A GPU-based Associative Memory using Sparse Neural Networks. In: Proceedings of the PCNN-14 conference, pp. 688–692 (2014)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Vincent Gripon
    • 1
  • Judith Heusel
    • 2
  • Matthias Löwe
    • 2
  • Franck Vermet
    • 3
  1. 1.Telecom Bretagne UMR CNRS Lab-STICCTechnopole Brest IroiseBrestFrance
  2. 2.Fachbereich Mathematik und InformatikUniversity of MünsterMünsterGermany
  3. 3.Laboratoire de Mathématiques, UMR CNRS 6205Université de Bretagne OccidentaleBrest Cedex 3France

Personalised recommendations