Abstract
We describe a class of feed forward neural network models for associative content addressable memory (ACAM) which utilize sparse internal representations for stored data. In addition to the input and output layers, our networks incorporate an intermediate processing layer which serves to label each stored memory and to perform error correction and association. We study two classes of internal label representations: the unary representation and various sparse, distributed representations. Finally, we consider storage of sparse data and sparsification of data. These models are found to have advantages in terms of storage capacity, hardware efficiency, and recall reliability when compared to the Hopfield model, and to possess analogies to both biological neural networks and standard digital computer memories.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albus JS (1971) A theory of cerebellar functions. Math Biosci 10:25
Amit DJ, Gutfreund H, Sompolinsky H (1985) Storing infinite numbers of patterns in a spin glass model of neural networks. Phys Rev Lett 55:1530
Barlow HB (1972) Single units and sensation: a neuron doctrine for perceptual psychology. Perception 1:371
Denker J (ed) (1986) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New York
Erdos P, Hanani H (1963) On a limit theorem in combinatorial analysis. Publ Math Debrecen 10:10–13
Graf HP, de Vegvar P (1987) CMOS Implementatuon of a neural network model. In: Loslever P (ed) Advanced research in VLSI, Proceedings of the 1987 Stanford Conference. MIT Press, Cambridge, Mass
Grossberg S (1982) Studies of mind and brain. Reidel, Boston
Hinton GE, Anderson JE (eds) (1981) Parallel models of associative memory. Lawrence Erlbaum, Hillsdale, NJ
Hinton GE, Sejnowski TJ, Ackley DH (1984) Boltzman machines: constraint satisfaction networks that learn. Carnegie-Mellon University, Tech-Rep. CMU-CS-84-219
Hopfield JJ (1982) Neural networks and physical systems with emergent computational abilities. Proc Natl Acad Sci USA 79:2554
Hopfield JJ (1984) Neurons with graded response have computational properties like those of two-state neurons. Proc Natl Acad Sci USA 81:3088
Hopfield JJ, Tank DW (1985) Simple “neural” optimization networks: an A/D converter, signal decision circuit, and a linear programming circuit. Biol Cybern 52:141
Hopfield JJ, Tank DW (1986) IEEE Trans Circuits Syst CAS- 33:533
Kanerva P (1984) Self-propagating search: a unified theory of memory. (Rep. No. CSLI-84-7) Stanford: Center for the Study of Language and Information. Bradford Books/MIT Press, Cambridge, Mass: (to be published)
Kohonen T (1984) Self-organization and associative memory. Springer, Berlin Heidelberg New York
Kohonen T (1987) Content-addressable memories. Springer, Berlin Heidelberg New York
Kuffler SW, Nichols JG, Martin AR (1984) From neuron to brain, 2nd edn. Sinauer Associates, Sunderland, Mass
Lansner A, Ekeberg Ö (1985) Reliability and speed of recall in an associative network. IEEE PAMI- 7:490
LeCun Y (1985) A learning procedure for asymmetric threshold networks. Proc Cogn 85:599
Lippmann RP (1987) An introduction to computing with neural nets. IEEE ASP Mag (April):4–22
Little WA (1974) The existence of persistent states in the brain. Math Biosci 19:101
McEliece RJ, Posner EC, Rodemich ER, Venkatesh SS (1987) The capacity of the Hopfield associative memory. IEEE Trans IT-33
McLelland HL, Rumelhart DE (1986) The PDP Research Group (eds) (1986) Parallel distributed processing: explorations in the microstructure of cognition. MIT Press/Bradford Books, Cambridge, Mass
MacWilliams FJ, Sloane NJA (1978) The theory of error correcting codes. North-Holland, New York
Marr D (1969) J Physiol 202:437
Martin T (1970) Acoustic recognition of a limited vocabulary in continuous speech. Ph. D. Thesis, Department of Electrical Engineering, University of Pennsylvania
Moopen A, Lambe J, Thakoor AP (1986) Personal communication to EBB
Moopen A, Lambe J, Thakoor AP (1987) Electronic implementation of associative memory based on neural network models. IEEE SMC-17:325–331
Palm G (1982) Neural assemblies and alternative approach to artificial intelligence. Springer, Berlin Heidelberg New York
Rumelhart DE, Hinton GE, Williams RJ (1981) In: Rumelhart DE, McLelland JL (eds), Learning internal representations by error propagation, ref. 1
Sage JP, Thompson K, Wither RS (1986) An artificial neural net integrated circuit based on NNOS/CCD principles. In: Deuker J (ed) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New York
Scalettar R, Zee A (1986) A feed-forward memory with decay. Institute for Theoretical Physics preprint, NSF-ITP-86-118
Sherrington CS (1941) Man on his nature. Cambridge University Press, Cambridge
Spencer EG (1986) Programmable bistable switches and resistors for neural networks. In: Deuher J (ed) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New York
Steinbuch K (1963) Automat und Mensch. Springer, Berlin Heidelberg New York
Steinbuch K, Piske UAW (1963) Learning matrices and their applications. IEEE Trans EC-12:846–862
Willshaw DJ (1971) Models of distributive associative memory. Ph D thesis, University of Edinburgh
Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 22:960–962
Wilson RM (1975) An existence theory for pairwise balanced designs. III. Proof of the existence conjecture. J Comb Theor 18A:71–79
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Baum, E.B., Moody, J. & Wilczek, F. Internal representations for associative memory. Biol. Cybern. 59, 217–228 (1988). https://doi.org/10.1007/BF00332910
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/BF00332910