The Replica Symmetric Approximation of the Analogical Neural Network

Abstract

In this paper we continue our investigation of the analogical neural network, by introducing and studying its replica symmetric approximation in the absence of external fields. Bridging the neural network to a bipartite spin-glass, we introduce and apply a new interpolation scheme to its free energy, that naturally extends the interpolation via cavity fields or stochastic perturbations from the usual spin glass case to these models.

While our methods allow the formulation of a fully broken replica symmetry scheme, in this paper we limit ourselves to the replica symmetric case, in order to give the basic essence of our interpolation method. The order parameters in this case are given by the assumed averages of the overlaps for the original spin variables, and for the new Gaussian variables. As a result, we obtain the free energy of the system as a sum rule, which, at least at the replica symmetric level, can be solved exactly, through a self-consistent mini-max variational principle.

The so gained replica symmetric approximation turns out to be exactly correct in the ergodic region, where it coincides with the annealed expression for the free energy, and in the low density limit of stored patterns. Moreover, in the spin glass limit it gives the correct expression for the replica symmetric approximation in this case. We calculate also the entropy density in the low temperature region, where we find that it becomes negative, as expected for this kind of approximation. Interestingly, in contrast with the case where the stored patterns are digital, no phase transition is found in the low temperature limit, as a function of the density of stored patterns.

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Amit, D.J.: Modeling Brain Function: The World of Attractor Neural Network. Cambridge University Press, Cambridge (1992)

    Google Scholar 

  2. 2.

    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Spin glass model of neural networks. Phys. Rev. A 32, 1007–1018 (1985)

    Article  MathSciNet  ADS  Google Scholar 

  3. 3.

    Amit, D.J., Gutfreund, H., Sompolinsky, H.: Storing infinite numbers of patterns in a spin glass model of neural networks. Phys. Rev. Lett. 55, 1530–1533 (1985)

    Article  ADS  Google Scholar 

  4. 4.

    Barra, A.: The mean field Ising model trough interpolating techniques. J. Stat. Phys. 132, 12–32 (2008)

    Article  MathSciNet  Google Scholar 

  5. 5.

    Barra, A., Guerra, F.: About the ergodic regime in the analogical Hopfield neural networks: moments of the partition function. J. Math. Phys. 50, 125217 (2008)

    Article  MathSciNet  ADS  Google Scholar 

  6. 6.

    Bovier, A., Picco, P.: Mathematical Aspects of Spin Glasses and Neural Networks. Birkhäuser, Basel (1998) and references therein

    Google Scholar 

  7. 7.

    Bovier, A., van Enter, A.C.D., Niederhauser, B.: Stochastic symmetry-breaking in a Gaussian Hopfield-model. J. Stat. Phys. 95, 181–213 (1999)

    MATH  Article  Google Scholar 

  8. 8.

    Coolen, A.C.C., Kuehn, R., Sollich, P.: Theory of Neural Information Processing Systems. Oxford University Press, Oxford (2005)

    Google Scholar 

  9. 9.

    Genovese, G., Barra, A.: A certain class of Curie-Weiss models. arXiv:0906.4673 (2009)

  10. 10.

    Guerra, F.: About the overlap distribution in mean field spin glass models. Int. J. Mod. Phys. B 10, 1675–1684 (1996)

    Article  MathSciNet  ADS  Google Scholar 

  11. 11.

    Guerra, F.: Sum rules for the free energy in the mean field spin glass model. In: Mathematical Physics in Mathematics and Physics: Quantum and Operator Algebraic Aspects. Fields Institute Communications, vol. 30 Amer. Math. Soc., Providence (2001)

    Google Scholar 

  12. 12.

    Guerra, F.: Broken replica symmetry bounds in the mean field spin glass model. Commun. Math. Phys. 233, 1–12 (2003)

    MATH  Article  MathSciNet  ADS  Google Scholar 

  13. 13.

    Guerra, F.: An introduction to mean field spin glass theory: methods and results. In: Bovier, A., et al. (eds.) Mathematical Statistical Physics, pp. 243–271. Elsevier, Oxford (2006)

    Google Scholar 

  14. 14.

    Guerra, F., Toninelli, F.L.: The thermodynamic limit in mean field spin glass models. Commun. Math. Phys. 230, 71–79 (2002)

    MATH  Article  MathSciNet  ADS  Google Scholar 

  15. 15.

    Guerra, F., Toninelli, F.L.: The infinite volume limit in generalized mean field disordered models. Markov Process. Relat. Fields 9, 195–207 (2003)

    MATH  MathSciNet  Google Scholar 

  16. 16.

    Hebb, D.O.: Organization of Behaviour. Wiley, New York (1949)

    Google Scholar 

  17. 17.

    Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. 79, 2554–2558 (1982)

    Article  MathSciNet  ADS  Google Scholar 

  18. 18.

    Pastur, L., Scherbina, M., Tirozzi, B.: The replica symmetric solution of the Hopfield model without replica trick. J. Stat. Phys. 74, 1161–1183 (1994)

    MATH  Article  ADS  Google Scholar 

  19. 19.

    Pastur, L., Scherbina, M., Tirozzi, B.: On the replica symmetric equations for the Hopfield model. J. Math. Phys. 40, 3930–3947 (1999)

    MATH  Article  MathSciNet  ADS  Google Scholar 

  20. 20.

    Talagrand, M.: Rigourous results for the Hopfield model with many patterns. Probab. Theory Relat. Fields 110, 177–276 (1998)

    MATH  Article  MathSciNet  Google Scholar 

  21. 21.

    Talagrand, M.: Exponential inequalities and convergence of moments in the replica-symmetric regime of the Hopfield model. Ann. Probab. 38, 1393–1469 (2000)

    Article  MathSciNet  Google Scholar 

  22. 22.

    Talagrand, M.: Spin Glasses: A Challenge for Mathematicians. Cavity and Mean Field Models. Springer, Berlin (2003)

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Adriano Barra.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Barra, A., Genovese, G. & Guerra, F. The Replica Symmetric Approximation of the Analogical Neural Network. J Stat Phys 140, 784–796 (2010). https://doi.org/10.1007/s10955-010-0020-y

Download citation

Keywords

  • Spin-glasses
  • Neural networks
  • Replica symmetry