Biological Cybernetics

, Volume 59, Issue 4–5, pp 217–228 | Cite as

Internal representations for associative memory

  • E. B. Baum
  • J. Moody
  • F. Wilczek


We describe a class of feed forward neural network models for associative content addressable memory (ACAM) which utilize sparse internal representations for stored data. In addition to the input and output layers, our networks incorporate an intermediate processing layer which serves to label each stored memory and to perform error correction and association. We study two classes of internal label representations: the unary representation and various sparse, distributed representations. Finally, we consider storage of sparse data and sparsification of data. These models are found to have advantages in terms of storage capacity, hardware efficiency, and recall reliability when compared to the Hopfield model, and to possess analogies to both biological neural networks and standard digital computer memories.


Neural Network Error Correction Output Layer Internal Representation Neural Network Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Albus JS (1971) A theory of cerebellar functions. Math Biosci 10:25CrossRefGoogle Scholar
  2. Amit DJ, Gutfreund H, Sompolinsky H (1985) Storing infinite numbers of patterns in a spin glass model of neural networks. Phys Rev Lett 55:1530CrossRefPubMedGoogle Scholar
  3. Barlow HB (1972) Single units and sensation: a neuron doctrine for perceptual psychology. Perception 1:371CrossRefPubMedGoogle Scholar
  4. Denker J (ed) (1986) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New YorkGoogle Scholar
  5. Erdos P, Hanani H (1963) On a limit theorem in combinatorial analysis. Publ Math Debrecen 10:10–13Google Scholar
  6. Graf HP, de Vegvar P (1987) CMOS Implementatuon of a neural network model. In: Loslever P (ed) Advanced research in VLSI, Proceedings of the 1987 Stanford Conference. MIT Press, Cambridge, MassGoogle Scholar
  7. Grossberg S (1982) Studies of mind and brain. Reidel, BostonCrossRefGoogle Scholar
  8. Hinton GE, Anderson JE (eds) (1981) Parallel models of associative memory. Lawrence Erlbaum, Hillsdale, NJGoogle Scholar
  9. Hinton GE, Sejnowski TJ, Ackley DH (1984) Boltzman machines: constraint satisfaction networks that learn. Carnegie-Mellon University, Tech-Rep. CMU-CS-84-219Google Scholar
  10. Hopfield JJ (1982) Neural networks and physical systems with emergent computational abilities. Proc Natl Acad Sci USA 79:2554CrossRefPubMedPubMedCentralGoogle Scholar
  11. Hopfield JJ (1984) Neurons with graded response have computational properties like those of two-state neurons. Proc Natl Acad Sci USA 81:3088CrossRefPubMedPubMedCentralGoogle Scholar
  12. Hopfield JJ, Tank DW (1985) Simple “neural” optimization networks: an A/D converter, signal decision circuit, and a linear programming circuit. Biol Cybern 52:141PubMedGoogle Scholar
  13. Hopfield JJ, Tank DW (1986) IEEE Trans Circuits Syst CAS- 33:533CrossRefGoogle Scholar
  14. Kanerva P (1984) Self-propagating search: a unified theory of memory. (Rep. No. CSLI-84-7) Stanford: Center for the Study of Language and Information. Bradford Books/MIT Press, Cambridge, Mass: (to be published)Google Scholar
  15. Kohonen T (1984) Self-organization and associative memory. Springer, Berlin Heidelberg New YorkGoogle Scholar
  16. Kohonen T (1987) Content-addressable memories. Springer, Berlin Heidelberg New YorkCrossRefGoogle Scholar
  17. Kuffler SW, Nichols JG, Martin AR (1984) From neuron to brain, 2nd edn. Sinauer Associates, Sunderland, MassGoogle Scholar
  18. Lansner A, Ekeberg Ö (1985) Reliability and speed of recall in an associative network. IEEE PAMI- 7:490CrossRefGoogle Scholar
  19. LeCun Y (1985) A learning procedure for asymmetric threshold networks. Proc Cogn 85:599Google Scholar
  20. Lippmann RP (1987) An introduction to computing with neural nets. IEEE ASP Mag (April):4–22Google Scholar
  21. Little WA (1974) The existence of persistent states in the brain. Math Biosci 19:101CrossRefGoogle Scholar
  22. McEliece RJ, Posner EC, Rodemich ER, Venkatesh SS (1987) The capacity of the Hopfield associative memory. IEEE Trans IT-33Google Scholar
  23. McLelland HL, Rumelhart DE (1986) The PDP Research Group (eds) (1986) Parallel distributed processing: explorations in the microstructure of cognition. MIT Press/Bradford Books, Cambridge, MassGoogle Scholar
  24. MacWilliams FJ, Sloane NJA (1978) The theory of error correcting codes. North-Holland, New YorkGoogle Scholar
  25. Marr D (1969) J Physiol 202:437CrossRefPubMedPubMedCentralGoogle Scholar
  26. Martin T (1970) Acoustic recognition of a limited vocabulary in continuous speech. Ph. D. Thesis, Department of Electrical Engineering, University of PennsylvaniaGoogle Scholar
  27. Moopen A, Lambe J, Thakoor AP (1986) Personal communication to EBBGoogle Scholar
  28. Moopen A, Lambe J, Thakoor AP (1987) Electronic implementation of associative memory based on neural network models. IEEE SMC-17:325–331Google Scholar
  29. Palm G (1982) Neural assemblies and alternative approach to artificial intelligence. Springer, Berlin Heidelberg New YorkGoogle Scholar
  30. Rumelhart DE, Hinton GE, Williams RJ (1981) In: Rumelhart DE, McLelland JL (eds), Learning internal representations by error propagation, ref. 1Google Scholar
  31. Sage JP, Thompson K, Wither RS (1986) An artificial neural net integrated circuit based on NNOS/CCD principles. In: Deuker J (ed) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New YorkGoogle Scholar
  32. Scalettar R, Zee A (1986) A feed-forward memory with decay. Institute for Theoretical Physics preprint, NSF-ITP-86-118Google Scholar
  33. Sherrington CS (1941) Man on his nature. Cambridge University Press, CambridgeGoogle Scholar
  34. Spencer EG (1986) Programmable bistable switches and resistors for neural networks. In: Deuher J (ed) Neural networks for computing. AIP Conference Proceedings 151. Snowbird, Utah New YorkGoogle Scholar
  35. Steinbuch K (1963) Automat und Mensch. Springer, Berlin Heidelberg New YorkCrossRefGoogle Scholar
  36. Steinbuch K, Piske UAW (1963) Learning matrices and their applications. IEEE Trans EC-12:846–862Google Scholar
  37. Willshaw DJ (1971) Models of distributive associative memory. Ph D thesis, University of EdinburghGoogle Scholar
  38. Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 22:960–962CrossRefGoogle Scholar
  39. Wilson RM (1975) An existence theory for pairwise balanced designs. III. Proof of the existence conjecture. J Comb Theor 18A:71–79CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 1988

Authors and Affiliations

  • E. B. Baum
    • 1
    • 2
  • J. Moody
    • 1
  • F. Wilczek
    • 1
  1. 1.Institute for Theoretical PhysicsUniversity of CaliforniaSanta BarbaraUSA
  2. 2.198-330, Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadenaUSA

Personalised recommendations