Long-Term Behavior of Neural Networks

  • John W. Clark
Part of the NATO ASI Series book series (NSSB, volume 222)


Two important interdisciplinary goals to which statistical physics can potentially make vital contributions are (a) the formulation of dynamical models which capture the primary features of information processing and adaptive behavior in living nervous systems and (b) the design of computational algorithms or devices which solve cognitive problems according to the principles of natural intelligence. Currently, both goals are being actively pursued in terms of neural networks.l–10 A neural network consists of a collection of neuron-like units, with synapse-like couplings which may be adjusted according to some learning rule, so as to achieve the desired performance of the network. The equation of motion determining the state of the network at each time is nonlinear and dissipative. Generally, the long-term behavior of such networks is considered to determine their usefulness in both the natural and artificial contexts: it is generally identified with the response of the organism to a given initial stimulus, or else as the solution provided by the algorithm or device. The essential idea is exemplified by an idealized form of content-addressable memory (CAM) discussed by Hopfield, based on fully and symmetrically connected nets of binary neurons governed by an asynchronous threshold dynamics.11 Started from any initial state, the system evolves necessarily to some stable fixed-point configuration. The end point of the evolution is identified with the stored memory with which the stimulus represented by the initial state is associated. Such a memory is distributed, in the sense that information about it is widely distributed over many synapses (couplings) in the system; it is error correcting in that a few errors in the input will not disturb accurate recall of the relevant stored memory.


Thermodynamic Limit Taylor Model Chaotic Time Series Firing Probability Incoming Connection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    J. W. Clark, J. Rafelski, and J. V. Winston, Physics Reports 123, 215 (1985).ADSCrossRefGoogle Scholar
  2. 2.
    D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vols. 1 and 2, ( MIT Press, Cambridge, Mass., 1986 ).Google Scholar
  3. 3.
    R. M. J. Cotterill, ed., Computer Simulation in Brain Science, ( Cambridge University Press, Cambridge, 1988 ).Google Scholar
  4. 4.
    J. W. Clark, Physics Reports 158, 9 (1988).CrossRefGoogle Scholar
  5. 5.
    J. D. Cowan and D. H. Sharp, Quarterly Reviews of Biophysics 21 (1988);Google Scholar
  6. J. D. Cowan and D. H. Sharp, Proc. Am. Acad. Arts. Sci. 117, 85 (1988).Google Scholar
  7. 6.
    E. Domany, J. Stat. Phys. 51, 743 (1988).MathSciNetADSMATHCrossRefGoogle Scholar
  8. 7.
    T. J. Sejnowski, C. Koch, and P. S. Churchland, Science 241, 1229 (1988).CrossRefGoogle Scholar
  9. 8.
    H. Sompolinsky, Physics Today 41 (12), 70 (1988).MathSciNetADSCrossRefGoogle Scholar
  10. 9.
    Y. C. Lee, ed., Evolution, Learning, and Cognition, ( World Scientific, Singapore, 1988 ).Google Scholar
  11. 10.
    J. W. Clark, in Nonlinear Phenomena in Complex Systems, A. N. Proto, ed. ( Elsevier, Amsterdam, 1989 ).Google Scholar
  12. 11.
    J. J. Hopfield, Proc. U. S. Nat. Acad. Sci. 79, 2554 (1982).CrossRefGoogle Scholar
  13. 12.
    F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, ( Spartan Books, Washington, D.C., 1962 ).MATHGoogle Scholar
  14. 13.
    W. S. McCulloch and W. Pitts, Bull. Math. Biophys. 5, 115 (1943).MathSciNetMATHCrossRefGoogle Scholar
  15. 14.
    W. A. Little, Math Biosci. 19, 101 (1974).MATHCrossRefGoogle Scholar
  16. 15.
    J. G. Taylor, J. Theoret. Biol. 36, 513 (1972).CrossRefGoogle Scholar
  17. 16.
    B. Katz, Nerve, Muscle, and Synapse, ( McGraw-Hill, New York, 1966 ).Google Scholar
  18. 17.
    J. W. Clark, J. W. Chen, and K. E. Kürten, in Models of Brain Function,R. M. J. Cotterill, ed. (Cambridge, University Press, Cambridge, 1989), in press.Google Scholar
  19. 18.
    R. B. Stein, K. V. Leung, D. Mangeron, and M. N. Ogurtöreli, Kybernetik 15, 1 (1974);MATHCrossRefGoogle Scholar
  20. R. B. Stein, K. V. Leung, M. N. Ogurtöreli, and D. W. Williams, Kybernetik 14, 223 (1974).MATHGoogle Scholar
  21. 19.
    J. D. Cowan, in Lectures on Mathematics in the Life Sciences, Vol. 2, M. Gerstenhaber, ed. ( American Mathematics Society, Providence, R. I., 1970 ), p. 157.Google Scholar
  22. 20.
    J. J. Hopfield, Proc. U. S. Nat. Acad. Sci. 81, 3088 (1984).CrossRefGoogle Scholar
  23. 21.
    D. J. Amit, H. Gutfreund, and H. Sompolinsky, Ann. Phys. N. Y. 173, 30 (1987).ADSCrossRefGoogle Scholar
  24. 22.
    J. W. Clark, K. E. Kürten, and J. Rafelski in Ref. 3, p. 316.Google Scholar
  25. 23.
    K. E. Kürten, J. Phys. France 50, 2313 (1989).CrossRefGoogle Scholar
  26. 24.
    K. E. Kiirten, Phys. Lett. Al29, 157 (1988).Google Scholar
  27. 25.
    K. E. Kürten, J. Phys. A21, L615 (1988).ADSMATHGoogle Scholar
  28. 26.
    K. E. Kürten, in Condensed Matter Theories,Vol. 3, J. S. Arponen, R. F. Bishop, and M. Manninen, eds. (Plenum, New York, 1988), p. 333; and private communication.Google Scholar
  29. 27.
    B. Derrida and Y. Pomeau, Europhys. Lett. 1, 44 (1986);ADSCrossRefGoogle Scholar
  30. B. Derrida, J. Phys. A20, L721 (1987).MathSciNetADSGoogle Scholar
  31. 28.
    D. R. Cox and H. D. Miller, Theory of Stochastic Processes, ( Chapman and Hall, London, 1965 ).MATHGoogle Scholar
  32. 29.
    M. Y. Choi and B. A. Huberman, Phys. Rev. A 28, 1204 (1983).MathSciNetADSCrossRefGoogle Scholar
  33. 30.
    P. Peretto, Biol. Cybernetics 50, 51 (1984).MATHCrossRefGoogle Scholar
  34. 31.
    J. W. Clark, in Ref. 9, p. 129.Google Scholar
  35. 32.
    J. Schnakenberg, Rev. Mod. Phys. 48, 571 (1976).MathSciNetADSCrossRefGoogle Scholar
  36. 33.
    P. C. Bressloff and J. G. Taylor, to be published.Google Scholar
  37. 34.
    G. L. Shaw and R. Vasudevan, Math. Biosci. 21, 207 (1974).MATHCrossRefGoogle Scholar
  38. 35.
    W. A. Little and G. L. Shaw, Behavioral Biology 14, 115 (1975).CrossRefGoogle Scholar
  39. 36.
    R. S. Thompson and W. G. Gibson, Math. Biosci. 56, 239, 255 (1981).MathSciNetMATHCrossRefGoogle Scholar
  40. 37.
    D. Gorse and J. G. Taylor, Physica D34, 90 (1989).MathSciNetMATHGoogle Scholar
  41. 38.
    K. E. Kürten and J. W. Clark, Phys. Lett. 114A, 413 (1986).MathSciNetCrossRefGoogle Scholar
  42. 39.
    K. E. Kürten and J. W. Clark, in Ref. 3, p. 357.Google Scholar
  43. 40.
    P. Grassberger and I. Procaccia, Physica D9, 189 (1983).MathSciNetMATHGoogle Scholar
  44. 41.
    G. M. Shepherd, The Synaptic Organization of the Brain, 2nd Edition, ( Oxford University Press, Oxford, 1979 ).Google Scholar

Copyright information

© Springer Science+Business Media New York 1990

Authors and Affiliations

  • John W. Clark
    • 1
  1. 1.McDonnell Center for the Space Sciences and Department of PhysicsWashington UniversitySt. LouisUSA

Personalised recommendations