Skip to main content
SpringerLink
Log in

Thermodynamic Formalism of Neural Computing

  • Chapter
Dynamics of Complex Interacting Systems

Part of the book series: Nonlinear Phenomena and Complex Systems ((NOPH,volume 2))

Abstract

Neural networks are systems of interconnected processors mimicking some of the brain functions. After a rapid overview of neural computing, the thermodynamic formalism of the learning procedure is introduced. Besides its use in introducing efficient stochastic learning algorithms, it gives an insight in terms of information theory. Main emphasis is given in the information restitution process; stochastic evolution is used as the starting point for introducing statistical mechanics of associative memory. Instead of formulating problems in their most general setting, it is preferred stating precise results on specific models. In this report are mainly presented those features that are relevant when the neural net becomes very large. A survey of the most recent results is given and the main open problems are pointed out.

Work partially supported by EU network CHRX-CT93-0411.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abbott, L.F., Learning in Neural Network Memories, Network 1, 105–122 (1990).

    Article  MathSciNet  Google Scholar 

  2. Aizenman, M., J.L. Lebowitz, D. Ruelle, Some Rigorous Results on the Sherrington-Kirkpatrick Model, Commun. Math. Phys., 112, 3–20 (1987).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  3. Albeverio, S., B. Tirozzi, B. Zegarlinski, Rigorous Results for the Free Energy in the Hopfield Model, Commun. Math. Phys., 150, 337–373 (1992).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  4. Amaro de Matos, J.M.G., A.E. Patrick, V.A. Zagrebnov, Random Infinite Volume Gibbs States for the Curie-Weiss Random Field Ising Model, J. Stat. Phys., 66, 139–164 (1992).

    Article  ADS  MATH  Google Scholar 

  5. Amit, D.J., Modelling Brain Function, Cambridge University Press, Cambridge (1989).

    Google Scholar 

  6. Amit, D.J., G. Gutfreund, H. Sompolinsky, Statistical Mechanisms of Neural Networks near Saturation, Ann. Phys., 173, 30–67 (1987).

    Article  ADS  Google Scholar 

  7. Anily, S., A. Federgruen, Simulated Annealing Methods with General Acceptance Probabilities, J. Appl. Prob. 24, 657–667 (1968).

    Article  MathSciNet  Google Scholar 

  8. Aubin, J.P., Mathematical Methods for Neural Networks, lecture notes of a COMETT graduate module held in Les Houches, 16–29 March (1992).

    Google Scholar 

  9. Azencott, R., Simulated Annealing, Séminaire Bourbaki No. 697, 1–15 (1988).

    Google Scholar 

  10. Ben Arous, G., A. Guionnet, Large Deviations for Langevin Spin Glass Dynamics, preprint DMI, École Normale Supérieure (1995).

    Google Scholar 

  11. Bernier, O., Stochastic Analysis of the Dynamics of a General Class of Synchronous Neural Networks, J. Phys. A: Math. Gen., 26, 6879–6892 (1993).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  12. Billingsley, P., Convergence of Probability Measures, Wiley, New York (1968).

    MATH  Google Scholar 

  13. Bovier, A., Self-Averaging in a Class of Generalised Hopfield Models, J. Phys. A: Math. Gen. 27, 7069–7077 (1994).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  14. Bovier, A., V. Gayrard, Rigorous Bounds on the Storage Capacity of the Dilute Hopfield Model, J. Stat. Phys. 69, 597–627 (1992).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  15. Bovier, A., V. Gayrard, Rigorous Results on the Thermodynamics of the Dilute Hopfield Model, J. Stat. Phys., 72, 79–112 (1993).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  16. Bovier, A., V. Gayrard, An almost sure Large Deviation Principle for the Hopfield Model, Weierstraß IAAS preprint (1995).

    Google Scholar 

  17. Bovier, A., V. Gayrard, P. Picco, Gibbs States of the Hopfield Model in the Regime of Perfect Memory, Prob. Th. Rel. Fields, 100, 329–363 (1994).

    Article  MathSciNet  MATH  Google Scholar 

  18. Bovier, A., V. Gayrard, P. Picco, Gibbs States of the Hopfield Model with Extensively Many Patterns, J. Stat. Phys., 79, 395–414 (1995).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  19. Bovier, A., V. Gayrard, P. Picco, Large Deviation Principles for the Hopfield and the Kac-Hopfield model, Weierstraß IAAS preprint (1994).

    Google Scholar 

  20. Broeck, C. van den, Statistical Physics of Learning from Examples: a Brief Introduction, Acta Phys. Polon. B, 25, 903–923 (1994).

    MathSciNet  MATH  Google Scholar 

  21. Cassandro, M., A. Galves, E. Olivieri, M. E. Vares, Metastable Behaviour of Stochastic Dynamics: a Pathwise Approach, J. Stat. Phys., 35, 603— (1984).

    Google Scholar 

  22. Changeux, J.-P., L’Homme Neuronal, Fayard, Paris (1983).

    Google Scholar 

  23. Collet, P., J.-P. Eckmann, Iterated Maps on the Interval as Dynamical Systems, Birkhäuser, Basel (1980).

    MATH  Google Scholar 

  24. Cottrell, M., Mathematical Analysis of a Neural Network with Inhibitory Coupling, Stoch. Proc. Appl., 40, 103–126 (1992).

    Article  MathSciNet  MATH  Google Scholar 

  25. Derrida, B., R.B. Griffiths, A Prügel-Bennett, Finite-Size Effects and Bounds for Perceptron Models, J. Phys. A: Math. Gen., 24, 4907–4940 (1991).

    Article  ADS  MATH  Google Scholar 

  26. Dobrushin, R.L., The Description of a Random Field by Means of Conditional Probabilities and Condition of its Regularities, Th. Prob. Appl., 13, 458–486 (1968).

    Google Scholar 

  27. Fassnacht, C., A. Zippelius, A recognition and categorisation in a structured neural network with attractor dynamics, Network, 2, 63–84 (1992).

    Article  MathSciNet  Google Scholar 

  28. Ferrari, P.A., S. Martinez, P. Picco, A Lower Bound for the Memory Capacity in the Potts-Hopfield Model, J. Stat. Phys., 66, 1643–1652 (1992).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  29. Folk, R., A. Kartashov, P. Lisonék, P. Paule, Symmetries in Neural Networks: a Linear Group Action Approach, J. Phys. A: Math. Gen., 26, 3159–3164 (1993).

    Article  ADS  MATH  Google Scholar 

  30. Freidlin, M.I., A.D. Wentzell, Random Perturbations of Dynamical Systems, Springer-Verlag, Berlin (1984).

    Book  MATH  Google Scholar 

  31. Gardner, E., The Space of Interactions in Neural Network Models, J. Phys. A: Math. Gen., 21, 257–270 (1987).

    Article  ADS  Google Scholar 

  32. Gardner, E., B. Derrida, Optimal Storage Properties of Neural Network Models, J. Phys. A: Math. Gen., 21, 271–284 (1988).

    Article  MathSciNet  ADS  Google Scholar 

  33. Gayrard, V., The Thermodynamic Limit of the Potts-Hopfield Model for Infinitely Many Patterns, J. Stat. Phys., 68, 977–1011 (1992).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  34. Georgii, H.-O., Gibbs Measures and Phase Transitions, Walter de Gruyter, Berlin (1988).

    Book  MATH  Google Scholar 

  35. Girko, V.L., Random Matrices, Vishcha Shkola, Izdat. Kiev Univ., Kiev (1975).

    Google Scholar 

  36. Girko, V L, Limit Theorems for Maximal and Minimal Eignevalues of Random Matrices, Th. Prob. Appli., 35, 680–695 (1988).

    Article  MathSciNet  Google Scholar 

  37. Girko, V.L., Theory of Random Determinants, Kluwer, Dodrecht (1990).

    Book  Google Scholar 

  38. Goles, E., S. Martinez, Neural and automata networks, Kluwer Academic Publ., Dodrecht (1990).

    Book  MATH  Google Scholar 

  39. Goles, E., S. Martinez, Statistical Physics, Automata Networks, and Dynamical Systems, Kluwer Academic Publ., Dodrecht (1992).

    Google Scholar 

  40. Griniasty, M., M.V. Tsodyks, D.J. Amit, Conversion of Temporal Correlations between Stimuli to Spatial Correlations between Attractors, preprint Università di Roma 1 (1992).

    Google Scholar 

  41. Haario, H., E. Saksman, Simulated Annealing Process in General State Space, Adv. Appl. Prob., 23, 866–893 (1991).

    Article  MathSciNet  MATH  Google Scholar 

  42. Hebb, D., The Organisation of Behaviour: a Neurolophysiological Theory, Wiley, New York (1949).

    Google Scholar 

  43. Hemmen, J.L. van, D. Grensing, A. Huber, R. Kühn, Nonlinear Neural Networks I: General Theory, J. Stat. Phys., 50, 231–257 (1988).

    Article  ADS  MATH  Google Scholar 

  44. Hemmen, J.L. van, D. Grensing, A. Huber, R. Kühn, Nonlinear neural networks II: information processing, J. Stat. Phys., 50, 259–293 (1988).

    Article  ADS  MATH  Google Scholar 

  45. Hertz, J., A. Krogh, R. Palmer, Introduction to the theory of neural computation, Addison-Wesley, Redwood City CA (1991).

    Google Scholar 

  46. Hopfield, J.J., Neural Networks and Physical Systems with Emergent Collective Computational Abilities, Proc. Natl. Acad. Sci., 79, 2554–2558 (1982).

    Article  MathSciNet  ADS  Google Scholar 

  47. Kallianpur, G., Weak Convergence of Stochastic Neuronal Models, in Stochastic Methods in Biology, Nagoya (1985), M. Kimura, G. Kallianpur, and T. Hida (eds.).

    Google Scholar 

  48. Khinchin, A.I., Mathematical Foundations of Information Theory, Dover, New York (1957).

    MATH  Google Scholar 

  49. Koch, H., A Free Energy Bound for the Hopfield Model, J. Phys. A: Math. Gen., 26, L353 — L355 (1993).

    Article  ADS  MATH  Google Scholar 

  50. Koch, H., J. Piasko, Some Rigorous Results on the Hopfield Neural Network Model, J. Stat. Phys., 55, 903–928 (1989).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  51. Kolmogorov, A.N., Foundations of Probability Theory, Chelsea Publishing, New York (1950).

    Google Scholar 

  52. Komlós, J., R. Paturi, Convergence Results in a Autoassociative Memory Model, Neural Networks, 1, 239–250 (1988).

    Article  Google Scholar 

  53. Kosko, B., Neural Networks and Fuzzy Systems, Prentice-Hall, London (1992).

    MATH  Google Scholar 

  54. Krauth, W., M. Mézard, Storage Capacity of Memory Networks with Binary Couplings, J. Physique, 50, 3057–3066 (1989).

    Article  Google Scholar 

  55. Kuramoto, Y., Chemical Oscillation, Waves, and Turbulence, Springer-Verlag, Berlin (1984).

    Google Scholar 

  56. Lanford III, O.E., D. Ruelle, Observables at Infinity and States with Short Range Correlations in Statistical Mechanics, Commun. Math. Phys. 13, 194215 (1969).

    Google Scholar 

  57. Lang, S., Real and Functional Analysis, Springer-Verlag, Berlin (1993).

    Book  MATH  Google Scholar 

  58. Ledoux, M., M. Talagrand, Probability in Banach Spaces, Springer-Verlag, Berlin (1991).

    Book  MATH  Google Scholar 

  59. Little, W., G. Shaw, Analytic Study of the Memory Storage Capacity of a Neural Network, Math. Biosc., 39, 281–290 (1978).

    Article  MathSciNet  MATH  Google Scholar 

  60. Loukianova, D., Capacité de Mémoire dans le Modèle de Hopfield, C. R. Acad. Sci. Paris, 318, 157–160 (1994).

    MathSciNet  MATH  Google Scholar 

  61. Loukianova, D., Étude Rigoureuse du Modèle de Mémoire Associative, PhD Thesis, Université de Paris VII, presented on 2 December (1994).

    Google Scholar 

  62. MacEliece, R., E. Posner, E. Rodemich, S. Venkatesh, The Capacity of the Hopfield Associative Memory, IEEE Trans. Inf. Theory, 33, 461–482 (1987).

    Article  Google Scholar 

  63. Maes, C., V. Zagrebnov, On the Parallel Dynamics of a Multi-Layered Perceptron, (1991).

    Google Scholar 

  64. Martínez, S., Introduction to Neural Networks: Storage Capacity and Optimisation, proceedings CIMPA school on “Dynamical and disordered systems”, Temuco (1992).

    Google Scholar 

  65. McCulloch, W.S., W. Pitts, A Logical Calculus of the Ideas Immanent in Nervous Activity, Bull. Math. Biophys. 5, 115–133 (1943).

    Article  MathSciNet  MATH  Google Scholar 

  66. Mézard, M., G. Parisi, M.A. Virasoro, Spin-glass Theory and Beyond, World scientific, Singapore (1988).

    Google Scholar 

  67. Minlos, R., Gibbs’ Limit Distribution, Funct. Anal. Appl., 2, 60–73; 3, 40–53 (1967).

    MathSciNet  Google Scholar 

  68. Miyashita, Y., Neuronal Correlate of Visual Associative Long-Term Memory in the Primate Temporal Cortex, Nature, 335, 817–819 (1988).

    Article  ADS  Google Scholar 

  69. Monasson, R., Properties of Neural Network Storing Spatially Correlated Patterns, J. Phys. A: Math. Gen., 25, 3701–3720 (1992).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  70. Müller, B., J. Reinhardt, Neural Networks, Springer-Verlag, Berlin (1990).

    Google Scholar 

  71. Newman, C., Memory Capacity in Neural Networks, Neural Networks, 1, 223238 (1988).

    Google Scholar 

  72. Nilson, N.J., Learning Machines, McGraw-Hill, New York (1965).

    Google Scholar 

  73. Nishimori, H., W. Whyte, D. Sherrington, Finite-Dimensional Neural Networks Storing Structured Patterns, preprint University of Oxford (1994).

    Google Scholar 

  74. Parisi, G., Attractor Neural Networks,preprint (1994) available from cond-mat® babbage.sissa.itunder reference 941–2030.

    Google Scholar 

  75. Pastur, L.A., A.L. Figotin, Exactly Solvable Model of a Spin Glass, Soy. J. Low Temp. Phys., 3, 378–383 (1977).

    Google Scholar 

  76. Pastur, L.A., A.L. Figotin, On the Theory of Disordered Spin Systems, Theor. Math. Phys., 35, 404–414 (1978).

    Article  MathSciNet  Google Scholar 

  77. Pastur, L.A., M. Shcherbina, Absence of Self-Averaging of the Order Parameter in the Sherrington-Kirkpatrick Model, J. Stat. Phys., 62, 1–19 (1991).

    Article  MathSciNet  ADS  Google Scholar 

  78. Pastur, L.A., M. Shcherbina, B. Tirozzi, The Replica Symmetric Solution without Replica Trick for the Hopfield Model, J. Stat. Phys., 74, 1161–1183 (1994).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  79. Patrick, A.E., V.A. Zagrebnov, Parallel Dynamics for an Extremely Diluted Neural Network, J. Phys. A: Math. Gen., 23, L1323 - L1329 (1990).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  80. Patrick, A.E., V.A. Zagrebnov, A Probabilistic Approach to Parallel Dynamics for the Little Hopfield Model, J. Phys. A: Math. Gen., 24, 3413–3426 (1991).

    Article  MathSciNet  ADS  Google Scholar 

  81. Petritis, D., Simulations Numériques Monte Carlo, preprint Université de Rennes I, to be published by Masson, Paris (1995).

    Google Scholar 

  82. Petritis, D., Equilibrium Statistical Mechanics of Frustrated Disordered Systems: a Survey of Mathematical Results,preprint Université de Rennes I (1994), available from cond-matebabbage sissa. it under reference 941–1098.

    Google Scholar 

  83. Picco, P., Artificial Neural Networks, preprint CNRS Marseille (1995).

    Google Scholar 

  84. Radons, G., H.G. Schuster, D. Werner, Fractal Measures and Diffusion as Results of Learning in Neural Networks, Phys. Lett. A, 174, 293–297 (1993).

    Article  MathSciNet  ADS  Google Scholar 

  85. Rockafellar, R.T., Convex Analysis, Princeton Univ. Press, Princeton (1970).

    Google Scholar 

  86. Rosenblatt, F., Principles of Neurodynamics, Spartan, New York (1962).

    MATH  Google Scholar 

  87. Ruelle, D., Thermodynamic formalism, Addison-Wesley, Reading (1978).

    MATH  Google Scholar 

  88. Rumelhart, D.E., G.E. Hinton, R.J. Williams, Learning Representations by Back-Propagating Errors, Nature, 323, 533— (1986).

    Google Scholar 

  89. Sakaguchi, H., Learning Rules for an Oscillator Network, Phys. Lett. A, 174, 289–292 (1993).

    Article  MathSciNet  ADS  Google Scholar 

  90. Scacciatelli, E., B. Tirrozi, Fluctuation of the Free Energy in the Hopfield Model, J. Stat. Phys., 6’T, 981–1008 (1992).

    Google Scholar 

  91. Scharze, H., J. Hertz, Learning from Examples in fully Connected Committee Machines, J. Phys. A: Math. Gen., 26, 4919–4936 (1993).

    Article  ADS  Google Scholar 

  92. Schlüter, M., E. Wagner, Phys. Rev., E49, 1690— (1994).

    Google Scholar 

  93. Shcherbina, M., B. Tirozzi, The Free Energy for for a Class of Hopfield Models, J. Stat. Phys., 72, 113–125 (1993).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  94. Simon, B., The Statistical Mechanics of Lattice Gases, Princeton University press, Princeton (1993).

    MATH  Google Scholar 

  95. Solla, S., Learning and Generalisation in Layered Neural Networks, in Redes neuronales, teoria y aplicaciones, Escuela de Verano 88 en Física estadística y sistemas cooperativos (1988).

    Google Scholar 

  96. Solla, S., A Theory of Supervised Learning, in Neural networks: from biology to high energy physics, Proc. Elba Int. Physics Centre, O. Benhar, C. Bosio, P. del Giudice, E. Tabet eds. ETS Editrice, Pisa (1991).

    Google Scholar 

  97. Talagrand, M., Concentration of Measure and Isoperimetric Inequalities in Product Spaces, preprint Université Paris VI (1995).

    Google Scholar 

  98. Talagrand, M., A New Look at Independence, preprint Université Paris VI (1995).

    Google Scholar 

  99. Talagrand, M., Résultats Rigoureux pour le Modèle de Hopfield, preprint Université de Paris VI (1995).

    Google Scholar 

  100. Tarkowski, W., M. Lewenstein, Storage of Sets of Correlated Data in Neural Network Memories, J. Phys. A: Math. Gen., 26, 2453–2469 (1993).

    Article  MathSciNet  ADS  MATH  Google Scholar 

  101. Tishby, N., E. Levin, S. Solla, Consistent Inference of Probabilities in Layered Networks: Predictions and Generalisation, IEEE Neural Net., 2, 403–410 (1989).

    Google Scholar 

  102. Vermet, F., Convergence de la Variance de l’Énergie Libre du Modèle de Hopfield, C. R. Acad. Sci. Paris, 315, 1001–1004 (1992).

    MathSciNet  MATH  Google Scholar 

  103. Vermet, F., Étude Asymptotique d’un Réseau Neuronal: le Modale de Mémoire Associative de Hopfield, PhD Thesis, Université de Rennes I, presented on 28 January (1994).

    Google Scholar 

  104. Vermet, F., Asymptotic Study of a Neural Network, preprint Université de Rennes I (1994).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut de Recherche Mathématique, Université de Rennes I and CNRS URA 305, Campus de Beaulieu, F - 35042, Rennes Cedex, France

    Dimitri Petritis

Authors
  1. Dimitri Petritis
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Departamento de Ingeniería Matemática, F.C.F.M., Universidad de Chile, Santiago, Chile

    Eric Goles  & Servet Martínez  & 

Rights and permissions

Reprints and permissions

Copyright information

© 1996 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Petritis, D. (1996). Thermodynamic Formalism of Neural Computing. In: Goles, E., Martínez, S. (eds) Dynamics of Complex Interacting Systems. Nonlinear Phenomena and Complex Systems, vol 2. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-1323-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-94-017-1323-8_3

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-90-481-4734-2

  • Online ISBN: 978-94-017-1323-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation