Skip to main content
SpringerLink
Log in

Part 3: Brain science, information science and associative memory model

  • Tutorial Series on Brain-Inspired Computing
  • Published:
New Generation Computing Aims and scope Submit manuscript

Abstract

I review recent progress on the associative memory model, which is a kind of neural network model. First, I introduce this model and a mathematical theory called statistical neurodynamics describing its properties. Next, I discuss an associative memory model with hierarchically correlated memory patterns. Initially, in this model, the state approaches a mixed state that is a superposition of memory patterns. After that, it diverges from the mixed state, and finally converges to a memory pattern. I show that this retrieval dynamics can qualitatively replicate the temporal dynamics of face-responsive neurons in the inferior temporal cortex, which is considered to be the final stage of visual perception in the brain. Finally, I show an unexpected link between associative memory and mobile phones (CDMA). The mathematical structure of the CDMA multi-user detection problem resembles that of the associative memory model. It enables us to apply a theoretical framework of the associative memory model to CDMA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hopfield, J.J., “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,”Proceeding National Academy of Sciences, 79, pp. 2554–2558, 1982.

    Article  MathSciNet  Google Scholar 

  2. Anderson, J. A., “A Simple Neural Network Generating Interactive Memory,”Mathematical Biosciences, 14, pp. 197–220, 1972.

    Article  MATH  Google Scholar 

  3. Nakano, K., “Association-A Model of Associative Memory,”IEEE Transaction on Systems, Man and Cybernetics, 2, pp. 381–388, 1972.

    Article  Google Scholar 

  4. Kohonen, T., “Correlation Matrix Memories,”IEEE Transactions on Computers, 21, pp. 353–359, 1972.

    Article  MATH  Google Scholar 

  5. Amari, S., “Learning Patterns and Pattern Sequence by Self-organizing Nets of Threshold Elements,”IEEE Transactions on Computers, 21, pp. 1197–1206, 1972.

    Article  MATH  MathSciNet  Google Scholar 

  6. Amit, D.J., Gutfreund, H. and Sompolinsky, H., “Storing Infinite Numbers of Patterns in a Spin-glass Model of Neural Networks,”Physical Review Letters, 55, pp. 1530–1533, 1985.

    Article  Google Scholar 

  7. Sherrington, D. and Kirkpatrick, S., “Solvable Model of a Spin-glass,”Physical Review Letters, 35, pp. 1792–1796, 1975.

    Article  Google Scholar 

  8. Gardner, E., Derrida, B. and Mottishaw, P., “Zero Temperature Parallel Dynamics for Infinite Range Spin Glasses and Neural Networks,”Journal de Physique, 48, pp. 731–755, 1987.

    Article  Google Scholar 

  9. Amari, S. and Maginu, K., “Statistical Neurodynamics of Associative Memory,”Neural Networks, 1, pp. 63–73, Elsevier, 1988.

    Article  Google Scholar 

  10. Okada, M., “A Hierarchy of Macrodynamical Equations for Associative Memory,”Neural Networks, 8, pp. 833–838, Elsevier, 1995.

    Article  MathSciNet  Google Scholar 

  11. Okada, M., “Notions of Associative Memory and Sparse Coding,”Neural Networks, 9, pp. 1429–1458, Elsevier, 1996.

    Article  MATH  MathSciNet  Google Scholar 

  12. Miyashita, Y. and Chang, H.S., “Neuronal Correlate of Pictorial Short-term Memory in the Primate Temporal Cortex,”Nature, 331, pp. 68–70, 1988.

    Article  Google Scholar 

  13. Miyashita, Y., “Neuronal Correlate of Visual Associative Long-term Memory in the Primate Temporal Cortex,”Nature, 335, pp. 817–820, 1988.

    Article  Google Scholar 

  14. Griniasty, M., Tsodyks, M.V. and Amit, D.J., “Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors,”Neural Computation, 5, pp. 1–17, 1993.

    Article  Google Scholar 

  15. Amit, D.J., Brunel, N. and Tsodyks, M.V., “Correlations of Cortical Hebbian Reverberations: Theory versus Experiment,”Journal of Neuroscience 14, pp. 6435–6445, 1994.

    Google Scholar 

  16. Amari, S., “Neural Theory of Association and Concept-formation,”Biological Cybernetics, 26, pp. 175–185, 1977.

    Article  MATH  MathSciNet  Google Scholar 

  17. Okada, M., Matsumoto, N., Toya, K., Sugase-Miyamoto, Y., and Yamane, S., “Hierarchical Encoding of Faces and Its Neuronal Mechanism in the Inferior-temporal Cortex,”2004 International Symposium on Nonlinear Theory and Its Applications (NOLTA2004), pp. 19–22, 2004.

  18. Sugase, Y., Yamane, S., Ueno, S. and Kawano, K., “Global and Fine Information Coded by Single Neurons in the Temporal Visual Cortex,”Nature, 400, pp. 869–873, 1999.

    Article  Google Scholar 

  19. Matsumoto, N., Okada, M., Sugase-Miyamoto, Y., Yamane, S. and Kawano, K., “Population Dynamics of Face-responsive Neurons in the Inferior Temporal Cortex,”Cerebral Cortex, 15, pp. 1103–1112, Oxford Univ. Press, 2005.

    Article  Google Scholar 

  20. Matsumoto, N., Okada, M., Sugase-Miyamoto, Y. and Yamane, S., “Neuronal Mechanisms Encoding Global-to-fine Information in Inferior-temporal Cortex,”Journal of Computational Neuroscience, 18, pp. 85–103, 2005.

    Article  MathSciNet  Google Scholar 

  21. Rosenblatt, F., “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,”Psychological Review, 65, pp. 386–408, 1958.

    Article  MathSciNet  Google Scholar 

  22. Gardner, E., “The Space of Interactions in Neural Network Model,”Journal of Physics A: Mathematical and General, 21, pp. 257–270, 1988.

    Article  MathSciNet  Google Scholar 

  23. Barkai, E., Hansel, D. and Sompolinsky, H., “Broken Symmetries in Multilayered Perceptrons,”Physical Review A, 45, pp. 4146–4161, 1992.

    Article  Google Scholar 

  24. Seung, H. S., Sompolinsky, H. and Tishby, N., “Statistical Mechanics of Learning from Examples,”Physical Review A, 45, pp. 6056–6091, 1992.

    Article  MathSciNet  Google Scholar 

  25. Timothy, L.H., Watkin, A.R. and Biehl, M., “The Statistical Mechanics of Learning a Rule,”Review of Modern Physics, 65, pp. 499–556, 1993.

    Article  Google Scholar 

  26. Sourlas, N., “Spin-glass Models as Error-correcting Codes,”Nature, 339, p. 693, 1989.

    Article  Google Scholar 

  27. Kabashima, Y. and Saad, D., “Statistical Mechanics of Error Correcting Codes,”Europhysics Letters, 45, pp. 97–103, 1999.

    Article  Google Scholar 

  28. Hosaka, T., Kabashima, Y. and Nishimori, H., “Statistical Mechanics of Lossy Data Compression Using a Non-monotonic Perceptron,”Physical Review E, 66, 036125 2002.

    Article  MathSciNet  Google Scholar 

  29. Murayama, T. and Okada, M., “One-step RSB Scheme for the Rate Distortion Functions,”Journal of Physics A: Mathematical and General, 36, pp. 11123–11130, 2003.

    Article  MATH  MathSciNet  Google Scholar 

  30. Ozeki, T. and Okada, M., “Relaxation Dynamics of Decoding Processes of Sourlas Code,”Progress of Theoretical Physics Supplement, 157, pp. 250–253, 2005.

    Article  Google Scholar 

  31. Cover, T.M., “Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition,”IEEE Transactions on Electronic Computers, 14, pp. 326–334, 1965.

    Article  MATH  Google Scholar 

  32. Verdú, S.,Multiuser Detection, Cambridge University Press, Cambridge, 1998.

    MATH  Google Scholar 

  33. Tanaka, T., “A Statistical-mechanics Approach to Large-system Analysis of CDMA Multiuser Detectors,”IEEE Transactions on Information Theory, 48, pp. 2888–2910, 2002.

    Article  MATH  Google Scholar 

  34. Tanaka, T. and Okada, M., “Approximate Belief Propagation, Density Evolution, and Statistical Neurodynamics for CDMA Multiuser Detection,”IEEE Transactions on Information Theory, 51, pp. 700–706, 2005.

    Article  MathSciNet  Google Scholar 

  35. Mimura, K. and Okada, M., “Generating Functional Analysis of CDMA Detection Dynamics,”Journal of Physics A: Mathematical and General, 38, pp. 9917–9929, 2005.

    Article  MATH  MathSciNet  Google Scholar 

  36. Ishibashi, K., Hamaguchi, K. and Okada, M., in preparation.

Download references

Author information

Authors and Affiliations

  1. The University of Tokyo, 277-8561, Kashiwa Chiba, Japan

    Masato Okada

  2. RIKEN Brain Science Institute, 351-0198, Wako Saitama, Japan

    Masato Okada

Authors
  1. Masato Okada
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Masato Okada, Ph.D.: He is a Professor of Department of Complexity Science and Engineering, Graduate School of Frontier Sciences the University of Tokyo. He received B.S. from Osaka City University, Japan in 1985 and M.S. and Ph.D from Osaka University in 1987 and 1997, respectively.

About this article

Cite this article

Okada, M. Part 3: Brain science, information science and associative memory model. New Gener Comput 24, 185–201 (2006). https://doi.org/10.1007/BF03037297

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF03037297

Keywords

Search

Navigation