Optimising synaptic learning rules in linear associative memories
 P. Dayan,
 D. J. Willshaw
 … show all 2 hide
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Get AccessAbstract
Associative matrix memories with realvalued synapses have been studied in many incarnations. We consider how the signal/noise ratio for associations depends on the form of the learning rule, and we show that a covariance rule is optimal. Two other rules, which have been suggested in the neurobiology literature, are asymptotically optimal in the limit of sparse coding. The results appear to contradict a line of reasoning particularly prevalent in the physics community. It turns out that the apparent conflict is due to the adoption of different underlying models. Ironically, they perform identically at their coincident optima. We give details of the mathematical results, and discuss some other possible derivations and definitions of the signal/noise ratio.
 Bienenstock, E, Cooper, LN, Munro, P (1982) Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J Neurosci 2: pp. 3248
 Bliss, TVP, Lømo, T (1973) Longlasting potentiation of synaptic transmission in the dentate area of the anæsthetized rabbit following stimulation of the perforant path. J Physiol (London) 232: pp. 331356
 Buckingham JT (1991) Computation in the archicortex. PhD Thesis, Department of Artificial Intelligence, University of Edinburgh
 Buhman, J, Divko, R, Schulten, K On sparsely coded associative memories. In: Personnaz, L, Dreyfus, G eds. (1989) Neural networks: from models to applications. nEURO88, Paris
 Duda, RO, Hart, PE (1973) Pattern classification and scene analysis. Wiley, New York
 Gaffan, D (1974) Recognition impaired and association intact in the memory of monkeys after transection of the fornix. J Comp Physiol Psychol 86: pp. 11001109
 Gardner, E (1988) The space of interactions in neural network models. J Phys A: Math Gen 21: pp. 257 CrossRef
 Hancock PJB, Smith LS, Phillips WA (1991) A biologically supported errorcorrecting learning rule. Neural Computation (in press)
 Hebb, DO (1949) The organization of behavior: a neuropsychological theory. Wiley, New York
 Hopfield, JJ (1982) Neural networks and physical systems with emergent computational abilities. Proc Natl Acad Sci 79: pp. 25542558
 Ito, M, Sakurai, M, Tongroach, P (1982) Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells. J Physiol 324: pp. 113134
 Kanerva, P (1988) Sparse distributed memory. MIT Press, Cambridge
 Kohonen, T (1972) Correlation matrix memories. IEEE Trans Comput C21: pp. 353359
 Marr, D (1971) Simple memory: a theory for archicortex. Philos Trans Soc London B 262: pp. 2381
 Palm, G On the asymptotic information storage capacity of neural networks. In: Eckmiller, R, Malsburg, C eds. (1988) Neural computers. Springer, Berlin, Heidelberg New York, pp. 271280
 Palm, G Local synaptic rules with maximal information storage capacity. In: Haken, H eds. (1988) Neural and synergetic computers Springer Series in Synergetics, vol 42. Springer, Berlin, Heidelberg New York, pp. 100110
 PerezVincente, CJ, Amit, DJ (1989) Optimised network for sparsely coded patterns. J Phys A: Math Gen 22: pp. 559569 CrossRef
 Racine, RJ, Milgram, NW, Hafner, S (1983) Longterm potentiation phenomena in the rat limbic forebrain. Brain Res 260: pp. 217231 CrossRef
 Sejnowski, TJ (1977) Storing covariance with nonlinearly interacting neurons. J Math Biol 4: pp. 303321 CrossRef
 Sejnowski, TJ (1977) Statistical constraints on synaptic plasticity. J Theor Biol 69: pp. 385389 CrossRef
 Singer, W Activitydependent selforganization of synaptic connections as a substrate of learning. In: Changeux, JP, Konishi, M eds. (1985) The neural and molecular bases of learning. Wiley, New York, pp. 301335
 Stanton, P, Sejnowski, TJ (1989) Associative longterm depression in the hippocampus: Induction of synaptic plasticity by Hebbian covariance. Nature 339: pp. 21518 CrossRef
 Stent, GS (1973) A physiological mechanism for Hebb's postulate of learning. Proc Natl Acad Sci 70: pp. 9971001
 Tsodyks, MV, Fiegel'man, MV (1988) The enhanced storage capacity in neural networks with low activity level. Europhys Lett 6: pp. 101105
 Willshaw DJ (1971) Models of distributed associative memory. PhD Thesis, University of Edinburgh
 Willshaw, DJ, Dayan, P (1990) Optimal plasticity in matrix memories: what goes up MUST come down. Neural Comput 2: pp. 8593
 Willshaw, DJ, Buneman, OP, LonguetHiggins, HC (1969) Nonholographic associative memory. Nature 222: pp. 960962
 Title
 Optimising synaptic learning rules in linear associative memories
 Journal

Biological Cybernetics
Volume 65, Issue 4 , pp 253265
 Cover Date
 19910801
 DOI
 10.1007/BF00206223
 Print ISSN
 03401200
 Online ISSN
 14320770
 Publisher
 SpringerVerlag
 Additional Links
 Topics
 Industry Sectors
 Authors

 P. Dayan ^{(1)}
 D. J. Willshaw ^{(1)}
 Author Affiliations

 1. Centre for Cognitive Science, University of Edinburgh, 2 Buccleuch Place, EH8 9LW, Edinburgh, Scotland, UK