Advertisement

Journal of Statistical Physics

, Volume 172, Issue 5, pp 1247–1269 | Cite as

Non-convex Multi-species Hopfield Models

  • Elena AgliariEmail author
  • Danila Migliozzi
  • Daniele Tantari
Article

Abstract

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern deep neural-network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton–Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the Mattis magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine with Gaussian hidden layer and the Bidirectional Associative Memory model.

Keywords

Neural networks Boltzmann machines Multipartite models 

Notes

Acknowledgements

E.A. acknowledges financial support by Sapienza Università di Roma (Project No. RG11715C7CC31E3D). D.T. is supported by Scuola Normale Superiore and National Group of Mathematical Physics GNFM-INdAM.

References

  1. 1.
    Bengio, Y., LeCun, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)ADSCrossRefGoogle Scholar
  2. 2.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. Google book (2016)Google Scholar
  3. 3.
    Amit, D.J.: Modeling Brain Function: The World of Attractor Neural Networks. Cambridge University Press, Cambridge (1992)Google Scholar
  4. 4.
    Coolen, A.C.C., Kühn, R., Sollich, P.: Theory of Neural Information Processing Systems. Oxford Press, Oxford (2005)zbMATHGoogle Scholar
  5. 5.
    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Spin Glass model of neural networks. Phys. Rev. A 32, 1007–1018 (1985)ADSMathSciNetCrossRefGoogle Scholar
  6. 6.
    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Storing infinite numbers of patterns in a spin glass model of neural networks. Phys. Rev. Lett. 55, 1530–1533 (1985)ADSCrossRefGoogle Scholar
  7. 7.
    Hackley, D.H., Hinton, G.E., Sejnowski, T.J.: A learning alghoritm for Boltzmann machines. Cogn. Sci. 9(1), 147 (1985)CrossRefGoogle Scholar
  8. 8.
    Salakhutdinov, R., Hinton, G.E.: Deep Boltzmann machines. AISTATS 1, 3 (2009)zbMATHGoogle Scholar
  9. 9.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Larocelle, H., Mandel, M., Pascanu, R., Bengio, Y.: Learning algorithms for the classification restricted Boltzmann machine. J. Mach. Learn. 13, 643–669 (2012)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Barra, A., Bernacchia, A., Santucci, E., Contucci, P.: On the equivalence of Hopfield networks and Boltzmann machines. Neural Netw. 34, 1–9 (2012)CrossRefGoogle Scholar
  12. 12.
    Barra, A., Genovese, G., Sollich, P., Tantari, D.: Phase transitions in Restricted Boltzmann Machines with generic priors. Phys. Rev. E 96(4), 042156 (2017)ADSCrossRefGoogle Scholar
  13. 13.
    Barra, A., Genovese, G., Sollich, P., Tantari, D.: Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors. Phys. Rev. E 97(2), 022310 (2018)ADSCrossRefGoogle Scholar
  14. 14.
    Tubiana, J., Monasson, R.: Emergence of compositional representations in restricted Boltzmann machines. Phys. Rev. Lett. 118, 138301 (2017)ADSCrossRefGoogle Scholar
  15. 15.
    Huang, H.: Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses. J. Stat. Mech. 2017(5), 053302 (2017)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Huang, H.: Role of zero synapses in unsupervised feature learning. J. Phys. A 51(8), 08LT01 (2018)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Hebb, O.D.: The Organization of Behaviour: A Neuropsychological Theory. Pshyc. Press, Melbourne (1949)Google Scholar
  18. 18.
    Kosko, B.: Bidirectional associative memories. IEEE Trans. Syst. Man Cybern. 18(1), 49–60 (1988)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kurchan, J., Peliti, L., Saber, M.: A statistical investigation of bidirectional associative memories (BAM). J. Phys. I 4(11), 1627–1639 (1994)Google Scholar
  20. 20.
    Englisch, H., Mastropietro, V., Tirozzi, B.: The BAM storage capacity. J. Phys. I 5(1), 85–96 (1995)Google Scholar
  21. 21.
    Barra, A., Contucci, P., Mingione, E., Tantari, D.: Multi-species mean field spin glasses. Rigorous results. Annales Henri Poincaré 16, 691–708 (2015)ADSMathSciNetCrossRefGoogle Scholar
  22. 22.
    Barra, A., Genovese, G., Guerra, F.: Equilibrium statistical mechanics of bipartite spin systems. J. Phys. A 44, 245002 (2011)ADSMathSciNetCrossRefGoogle Scholar
  23. 23.
    Barra, A., Galluzzi, A., Guerra, F., Pizzoferrato, A., Tantari, D.: Mean field bipartite spin models treated with mechanical techniques. Eur. Phys. J. B 87(3), 74 (2014)ADSMathSciNetCrossRefGoogle Scholar
  24. 24.
    Panchenko, D.: The free energy in a multi-species Sherrington-Kirkpatrick model. Ann. Probab. 43(6), 3494–3513 (2015)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Genovese, G., Tantari, D.: Overlap synchronisation in multipartite random energy models. J. Stat. Phys. 169(6), 1162–1170 (2017)ADSMathSciNetCrossRefGoogle Scholar
  26. 26.
    Contucci, P., Fedele, M., Vernia, C.: Inverse problem robustness for multi-species mean field spin models. J. Phys. A 46, 065001 (2013)ADSMathSciNetCrossRefGoogle Scholar
  27. 27.
    Genovese, G., Tantari, D.: Non-convex multipartite ferromagnets. J. Stat. Phys. 163(3), 492–513 (2016)ADSMathSciNetCrossRefGoogle Scholar
  28. 28.
    Agliari, E., Barra, A., Galluzzi, A., Tantari, D., Tavani, F.: A walk in the statistical mechanical formulation of neural networks—alternative routes to Hebb prescription. NCTA2014 7, 210–217 (2014)Google Scholar
  29. 29.
    McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5, 115–133 (1943)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Gardner, E.J., Wallace, D.J., Stroud, N.: Training with noise and the storage of correlated patterns in a neural network model. J. Phys. A 22(12), 2019 (1989)ADSMathSciNetCrossRefGoogle Scholar
  31. 31.
    Agliari, E., Barra, A., De Antoni, A., Galluzzi, A.: Parallel retrieval of correlated patterns: from Hopfield networks to Boltzmann machines. Neural Netw. 38, 52–63 (2013)CrossRefGoogle Scholar
  32. 32.
    Gutfreund, H.: Neural networks with hierarchically correlated patterns. Phys. Rev. A 37(2), 570 (1988)ADSMathSciNetCrossRefGoogle Scholar
  33. 33.
    Agliari, E., Barra, A., Galluzzi, A., Guerra, F., Moauro, F.: Multitasking associative networks. Phys. Rev. Lett. 109, 268101 (2012)ADSCrossRefGoogle Scholar
  34. 34.
    Sollich, P., Tantari, D., Annibale, A., Barra, A.: Extensive parallel processing on scale free networks. Phys. Rev. Lett. 113, 238106 (2014)ADSCrossRefGoogle Scholar
  35. 35.
    Agliari, E., Annibale, A., Barra, A., Coolen, A.C.C., Tantari, D.: Immune networks: multitasking capabilities near saturation. J. Phys. A 46, 415003 (2013)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Agliari, E., Annibale, A., Barra, A., Coolen, A.C.C., Tantari, D.: Immune networks: multi-tasking capabilities at medium load. J. Phys. A 46, 335101 (2013)ADSMathSciNetCrossRefGoogle Scholar
  37. 37.
    Agliari, E., Annibale, A., Barra, A., Coolen, A.C.C., Tantari, D.: Retrieving infinite numbers of patterns in a spin-glass model of immune networks. Europhys. Let. 117(2), 28003 (2017)ADSCrossRefGoogle Scholar
  38. 38.
    Agliari, E., Barra, A., Galluzzi, A., Isopi, M.: Multitasking attractor networks with neuronal threshold noise. Neural Netw. 49, 19–29 (2014)CrossRefGoogle Scholar
  39. 39.
    Barra, A., Genovese, G., Guerra, F.: The replica symmetric approximation of the analogical neural network. J. Stat. Phys. 140(4), 784–796 (2010)ADSMathSciNetCrossRefGoogle Scholar
  40. 40.
    Barra, A., Genovese, G., Guerra, F., Tantari, D.: How glassy are neural networks? J. Stat. Mech. 2012(07), P07009 (2012)CrossRefGoogle Scholar
  41. 41.
    Barra, A., Guerra, F.: About the ergodic regime in the analogical Hopfield neural networks: moments of the partition function. J. Math. Phys. 49, 125217 (2008)ADSMathSciNetCrossRefGoogle Scholar
  42. 42.
    Barra, A., Genovese, G., Guerra, F., Tantari, D.: About a solvable mean field model of a Gaussian spin glass. J. Phys. A 47(15), 155002 (2014)ADSMathSciNetCrossRefGoogle Scholar
  43. 43.
    Genovese, G., Tantari, D.: Legendre duality of spherical and Gaussian spin glasses. Math. Phys. Anal. Geom. 18, 10 (2015)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Agliari, E., Barra, A., Del Ferraro, G., Guerra, F., Tantari, D.: Anergy in self-directed B lymphocytes: a statistical mechanics perspective. J. Theor. Biol. 375, 21–31 (2015)CrossRefGoogle Scholar
  45. 45.
    Sompolinsky, H.: Neural networks with nonlinear synapses and a static noise. Phys. Rev. A 34, 2571(R) (1986)Google Scholar
  46. 46.
    Wemmenhove, B., Coolen, A.C.C.: Finite connectivity attractor neural networks. J. Phys. A 36, 9617 (2003)ADSMathSciNetCrossRefGoogle Scholar
  47. 47.
    Agliari, E., Barra, A., Galluzzi, A., Guerra, F., Tantari, D., Tavani, F.: Retrieval capabilities of hierarchical networks: from Dyson to Hopfield. Phys. Rev. Lett. 114, 028103 (2015)ADSCrossRefGoogle Scholar
  48. 48.
    Agliari, E., Barra, A., Galluzzi, A., Guerra, F., Tantari, D., Tavani, F.: Hierarchical neural networks perform both serial and parallel processing. Neural Netw. 66, 22–35 (2015)CrossRefGoogle Scholar
  49. 49.
    Agliari, E., Barra, A., Galluzzi, A., Guerra, F., Tantari, D., Tavani, F.: Metastable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network. J. Phys. A 48(1), 015001 (2014)ADSMathSciNetCrossRefGoogle Scholar
  50. 50.
    Agliari, E., Barra, A., Galluzzi, A., Guerra, F., Tantari, D., Tavani, F.: Topological properties of hierarchical networks. Phys. Rev. E 91(6), 062807 (2015)ADSMathSciNetCrossRefGoogle Scholar
  51. 51.
    Folli, V., Leonetti, M., Ruocco, G.: On the maximum storage capacity of the hopfield model. Front. Comput. Neurosci. 10, 144 (2017)CrossRefGoogle Scholar
  52. 52.
    Rocchi, J., Saad, D., Tantari, D.: High storage capacity in the Hopfield model with auto-interactions—stability analysis. J. Phys. A 50(46), 465001 (2017)ADSMathSciNetCrossRefGoogle Scholar
  53. 53.
    Albeverio, S., Tirozzi, B., Zegarlinski, B.: Rigorous results for the free energy in the Hopfield model. Commun. Math. Phys. 150, 337–373 (1992)ADSMathSciNetCrossRefGoogle Scholar
  54. 54.
    Pastur, L., Shcherbina, M., Tirozzi, B.: The replica-symmetric solution without replica trick for the Hopfield model. J. Stat. Phys. 74(5), 1161–1183 (1994)ADSMathSciNetCrossRefGoogle Scholar
  55. 55.
    Bovier, A., Gayrard, V., Picco, P.: Gibbs states of the Hopfield model with extensively many patterns. J. Stat. Phys. 79, 395–414 (1995)ADSMathSciNetCrossRefGoogle Scholar
  56. 56.
    Bovier, A., Gayrard, V.: The retrieval phase of the Hopfield model, a rigorous analysis of the overlap distribution. Probab. Theor. Rel. Fields 107, 61–98 (1995)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Bovier, A., Gayrard, V.: Hopfield models as generalized random mean field models. In: Bovier, A., Picco, P. (eds.) Progress in Probability, vol. 41. Birkauser, Boston (1997)zbMATHGoogle Scholar
  58. 58.
    Agliari, E., Barra, A., Tirozzi, B.: Boltzmann machines:self-averaging properties and thermodynamic limits, submitted (2018)Google Scholar
  59. 59.
    Scacciatelli, E., Tirozzi, B.: Fluctuation of the free energy in the Hopfeld model. J. Stat. Phys. 67, 981–1108 (1992)ADSCrossRefGoogle Scholar
  60. 60.
    Talagrand, M.: Rigorous results for the Hopfield model with many patterns. Probab. Theory Rel. Fields 110(2), 177–275 (1998)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Talagrand, M.: Exponential inequalities and convergence of moments in the replica-symmetric regime of the Hopfield model. Ann. Probab. 28(4), 1393–1469 (2000)MathSciNetCrossRefGoogle Scholar
  62. 62.
    Barra, A.: The mean field Ising model trough interpolating techniques. J. Stat. Phys. 132(5), 787–809 (2008)ADSMathSciNetCrossRefGoogle Scholar
  63. 63.
    Guerra, F.: Sum rules for the free energy in the mean field spin glass model. Fields Inst. Commun. 30, 161 (2001)MathSciNetzbMATHGoogle Scholar
  64. 64.
    Liao, X., Yu, J.: Qualitative analysis of Bi-directional Associative Memory with time delay. Int. J. Circ. Theor. Appl. 26(3), 219–229 (1998)CrossRefGoogle Scholar
  65. 65.
    Cao, J., Xiao, M.: Stability and Hopf Bifurcation in a simplified BAM neural network with two time delays. IEEE Trans. Neural Netw. 18(2), 416–430 (2007)CrossRefGoogle Scholar
  66. 66.
    Cao, J., Wang, L.: Exponential stability and periodic oscillatory solution in BAM networks with delays. IEEE Trans. Neural Netw. 13(2), 457–463 (2007)Google Scholar
  67. 67.
    Cao, J.: Global asymptotic stability of delayed bi-directional associative memory neural networks. Appl. Math. Comput. 142(2–3), 333–339 (2003)MathSciNetzbMATHGoogle Scholar
  68. 68.
    Cao, J., Wan, Y.: Matrix measure strategies for stability and synchronization of inertial BAM neural network with time delays. Neural Netw. 53, 165–172 (2014)CrossRefGoogle Scholar
  69. 69.
    Park, J.H., Park, C.H., Kwon, O.M., Leed, S.M.: A new stability criterion for bidirectional associative memory neural networks of neutral-type. Appl. Math. Comput. 199(2), 716–722 (2008)MathSciNetzbMATHGoogle Scholar
  70. 70.
    Gabrié, M., Tramel, E.W., Krzakala, F.: Training restricted Boltzmann machine via the Thouless-Anderson-Palmer free energy. Adv. Neural Inf. Process. Syst. 1, 640–648 (2015)Google Scholar
  71. 71.
    Mezard, M.: Mean-field message-passing equations in the Hopfield model and its generalizations. Phys. Rev. E 95(2), 022117 (2017)ADSMathSciNetCrossRefGoogle Scholar
  72. 72.
    Barra, A., Di Biasio, A., Guerra, F.: Replica symmetry breaking in mean-field spin glasses through the Hamilton Jacobi technique. J. Stat. Mech. 2010(09), P09006 (2010)MathSciNetCrossRefGoogle Scholar
  73. 73.
    Barra, A., Dal Ferraro, G., Tantari, D.: Mean field spin glasses treated with PDE techniques. Eur. Phys. J. B 86(7), 332 (2013)ADSMathSciNetCrossRefGoogle Scholar
  74. 74.
    Genovese, G., Barra, A.: A mechanical approach to mean field spin models. J. Math. Phys. 50(5), 053303 (2009)ADSMathSciNetCrossRefGoogle Scholar
  75. 75.
    Evans, L.: Partial Differential Equations (Graduate Studies in Mathematics), vol. 19. American Mathematical Society, Providence (1998)Google Scholar
  76. 76.
    Cannarsa, P., Sinestrari, C.: Semiconcave Functions, Hamilton-Jacobi Equations, and Optimal Control. Birkhauser, Boston (2004)zbMATHGoogle Scholar
  77. 77.
    Barbier, J., Dia, M., Macris, N., Krzakala, F., Lesieur, T., Zdeborova, L.: Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula. Advances in Neural Information Processing Systems, 424-432, (2015)Google Scholar
  78. 78.
    Kabashima, Y., Krzakala, F., Mèzard, M., Sakata, A., Zdeborova, L.: Phase transitions and sample complexity in Bayes-optimal matrix factorization. IEEE Trans. Inf. Theory 62(7), 4228–4265 (2016)MathSciNetCrossRefGoogle Scholar
  79. 79.
    Mézard, M., Parisi, G., Virasoro, M.A.: Spin Glass Theory and Beyond. World Scientific, Singapore (1987)zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Dipartimento di MatematicaSapienza Università di RomaRomeItaly
  2. 2.Scuola Normale SuperiorePisaItaly

Personalised recommendations