Abstract
A single McCulloch–Pitts neuron, that is, the simple perceptron is studied, with focus on the region beyond storage capacity. It is shown that Parisi's hierarchical ansatz for the overlap matrix of the synaptic couplings with so called continuous replica symmetry breaking is a solution, and as we propose it is the exact one, to the equilibrium problem. We describe some of the most salient features of the theory and give results about the low temperature region. In particular, the basics of the Parisi technique and the way to calculate thermodynamical expectation values is explained. We have numerically extremized the replica free energy functional for some parameter settings, and thus obtained the order parameter function, i.e., the probability distribution of overlaps. That enabled us to evaluate the probability density of the local stability parameter. We also performed a simulation and found a local stability density closer to the theoretical curve than previous numerical results were.
Similar content being viewed by others
REFERENCES
J. J. Hopfield, Proc. Natl. Acad. Sci. USA 79:2554 (1982).
W. A. Little, Math. Biosci. 19:101 (1974).
W. S. McCulloch and W. Pitts, Bull. Mathem. Biophys. 5:115 (1943).
E. Gardner, J. Phys. A 21:257 (1988).
J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation (Addison-Wesley, Reading, MA, 1991).
M. Mézard, G. Parisi, and M. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987).
K. H. Fischer and J. A. Hertz, Spin Glasses (Cambridge University Press, Cambridge, UK, 1991).
F. Rosenblatt, Principles of Neurodynamics (Spartan, New York, 1962).
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Nature 323:533 (1986).
T. L. H. Watkin, A. Rau, and M. Biehl, Rev. Mod. Phys. 65:499 (1993).
S. Amari, Neural Computation 10:251 (1998).
E. Gardner and B. Derrida, J. Phys. A 21:271 (1988).
M. L. Minsky and S. A. Papert, Perceptrons (MIT Press, Cambridge, MA, 1969).
M. Biehl and N. Caticha, in The Handbook of Brain Theory and Neural Networks, 2nd edn., M. A. Arbib, ed. (MIT Press, Cambridge, MA, 1999).
D. Malzahn, A. Engel, and I. Kanter, Phys. Rev. E 55:7369 (1997).
B. Schottky, J. Phys. A 28:4515 (1995).
A. H. L. West and D. Saad, J. Phys. A 31:8977 (1998).
O. Winther, B. Lautrup, and J.-B. Zhang, Phys. Rev. E 55:836 (1997).
W. Whyte and D. Sherrington, J. Phys. A 29:3063 (1996).
M. Opper and W. Kinzel, in Models of Neural Networks, Vol. III, E. Domany, J. L. van Hemmen, and K. Schulten, eds. (Springer-Verlag, New York, 1996).
G. Györgyi and N. Tishby, Statistical mechanics of learning a rule, in Proceedings of the STATPHYS-17 Workshop on Neural Networks and Spin Glasses, W. K. Theumann and R. Köberle, eds. (Singapore-Teaneck, NJ-Hongkong, 1990, World Scientific).
P. Majer, A. Engel, and A. Zippelius, J. Phys. A 26:7405 (1993).
R. Erichsen and W. K. Theumann, J. Phys. A 26:L61 (1993).
G. Györgyi and P. Reimann, Phys. Rev. Lett. 79:2746 (1997).
M. Griniasty and H. Gutfreund, J. Phys. A 24:715 (1991).
M. Aizenman, J. L. Lebowitz, and D. Ruelle, Comm. Math. Phys. 112:3 (1987).
G. Györgyi, Phys. Rep., to be published.
M. Bouten, J. Phys. A 27:6021 (1994).
G. Parisi, J. Phys. A 13:L115 (1980).
J. R. L. de Almeida and E. J. S. Lage, J. Phys. C 16:939 (1983).
H.-J. Sommers and W. Dupont, J. Phys. C 17:5785 (1984).
H. Sompolinsky and A. Zippelius, Phys. Rev. Lett. 47:935 (1981).
M. Mézard and M. A. Virasoro, J. Phys. France 46:1293 (1985).
T. Temesvári, C. De Dominicis, and I. Kondor, J. Phys. A 27:7569 (1994).
C. De Dominicis, T. Temesvári, and I. Kondor, J. Phys. IV France 8:Pr6-13 (1998).
Th. M. Nieuwenhuizen, Phys. Rev. Lett. 74:4289 (1995).
A. Engel has pointed out this fact to us.
A. Wendemuth, J. Phys. A 28:5423 (1995).
A. Wendemuth, J. Phys. A 28:5485 (1995).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Györgyi, G., Reimann, P. Beyond Storage Capacity in a Single Model Neuron: Continuous Replica Symmetry Breaking. Journal of Statistical Physics 101, 679–702 (2000). https://doi.org/10.1023/A:1026441500710
Issue Date:
DOI: https://doi.org/10.1023/A:1026441500710