Ockham's Razor at Work: Modeling of the ``Homunculus''

Brain and Mind volume 3, pages187–220(2002)Cite this article

Abstract

There is a broad consensus about the fundamental role of thehippocampal system (hippocampus and its adjacent areas) in theencoding and retrieval of episodic memories. This paper presents afunctional model of this system. Although memory is not asingle-unit cognitive function, we took the view that the wholesystem of the smooth, interrelated memory processes may have acommon basis. That is why we follow the Ockham's razor principleand minimize the size or complexity of our model assumption set.The fundamental assumption is the requirement of solving the socalled ``homunculus fallacy'', which addresses the issue ofinterpreting the input. Generative autoassociators seem to offer aresolution of the paradox. Learning to represent and to recallinformation, in these generative networks, imply maximization ofinformation transfer, sparse representation and noveltyrecognition. A connectionist architecture, which integrates theseaspects as model constraints, is derived. Numerical studiesdemonstrate the novelty recognition and noise filtering propertiesof the architecture. Finally, we conclude that the derivedconnectionist architecture can be related to the neurobiologicalsubstrate.

This is a preview of subscription content, access via your institution.

References

  1. Amari, S., Cichocki, A. and Yang, H., 1996: A new learning algorithm for blind signal separation, in Advances in Neural Information Processing Systems, Morgan Kaufmann, San Mateo, CA, pp. 757–763.

    Google Scholar 

  2. Attneave, F., 1954: Some informational aspects of visual perception, Psychological Review 61, 183–193.

    Google Scholar 

  3. Baddeley, R., 1996: An efficient code in v1? Nature 381, 560–561.

    Google Scholar 

  4. Baddeley, R., Abbott, L., Booth, M., Sengpiel, F., Freeman, T., Wakeman, E. and Rolls, E., 1997: Responses of neurons in primary and inferior temporal visual cortices to natural scenes, Proc. Roy. Soc. London B264, 1775–1783.

    Google Scholar 

  5. Barlow, H., 1961: Sensory Communication, MIT Press, Cambridge, MA,w.a. rosenblith edition, pp. 217–234.

    Google Scholar 

  6. Barlow, H., 1987: Learning receptive fields, volume IV of Proceedings of the IEEE 1st Annual Conf on Neural Networks, IEEE Press, U.S.A., pp. 115–121.

    Google Scholar 

  7. Barlow, H., 1999: Cerebral cortex, in C. Koch and J. Davis (eds), The MIT Encyclopedia of the Cognitive Sciences, MIT Press, Cambridge, MA, pp. 111–113.

    Google Scholar 

  8. Bell, A. and Sejnowski, T., 1995: An information–maximization approach to blind separation and blind deconvolution, Neural Computation 7, 1129–1159.

    Google Scholar 

  9. Brand, M., 1999: Voice puppetry, in Proceedings of Siggraph 99, ACM Press, New York, pp. 21–28.

    Google Scholar 

  10. Brand, M., Oliver, N. and Pentland, A., 1997: Coupled hidden Markov models for complex action recognition, in Proceedings of IEEE CVPR97, IEEE Press, pp. 994–999.

  11. Burgess, N. and O'Keefe, J., 1996: Neuronal computation underlying the firing of place cells and their role in navigation, Hippocampus 7, 1–15.

    Google Scholar 

  12. Buzsáki, G., 1984: Feed–forward inhibition in the hippocampal formation, Prog. Neurobiol. 22, 131–153.

    Google Scholar 

  13. Cardoso, J. and Laheld, B., 1996: Equivalent adaptive source separation, IEEE Trans. on Signal Proc. 44, 3017–3030.

    Google Scholar 

  14. Charles, D. and Fyfe, C., 1998: Modelling multiple cause structure using rectification constraints, Network: Computations in Neural Systems 9, 167–182.

    Google Scholar 

  15. Chrobak, J., Lörincz, A. and Buzsáki, G., 2000: Physiological patterns in the hippocampo–entorhinal cortex system, Hippocampus 10, 457–465.

    Google Scholar 

  16. Clark, R. and Squire, L., 1998: Classical conditioning and brain systems: The role of awareness, Science 280, 77–81.

    Google Scholar 

  17. Cohen, M. and Grossberg, S., 1983: Absolute stability of global pattern formation and parallel memory storage by competitive neural networks, IEEE Transactions on Systems, Man, and Cybernetics SMC–13, 815–826.

    Google Scholar 

  18. Cohen, N. and Eichenbaum, H., 1993: Memory, Amnesia and the Hippocampal System, MIT Press, Cambridge, MA.

    Google Scholar 

  19. Cohen, N. and Squire, L., 1980: Preserved learning and retrention of pattern–analyzing skill in amnesia: Dissociation of knowing how and knowing that, Science 210, 207–210.

    Google Scholar 

  20. Comon, P., 1994: Independent component analysis – A new concept? Signal Processing 36, 287–314.

    Google Scholar 

  21. Cover, T., 1974: Universal gambling schemes and the complexity measures of Kolmogorov and Chaitin, Technical Report 12, Department of Statistics, Stanford University, Stanford, CA.

    Google Scholar 

  22. Cover, T. and Thomas, J., 1991: Elements of Information Theory, John Wiley and Sons, New York.

    Google Scholar 

  23. Csató, L., Kovács, G., Harnad, S., Pevtzow, R. and Lörincz, A., 2000: Category learning, categorization difficulty, and categorical perception: Computational and behavioral evidence, preprint.

  24. Dayan, P. and Zemel, R., 1995: Competition and multiple cause models, Neural Computation 7, 565–579.

    Google Scholar 

  25. Dennett, D., 1991: Consciousness Explained, Little Brown, Boston, MA.

    Google Scholar 

  26. Diamond, I., 1979: The subdivision of the neocortex: A proposal to revise the traditional view of sensory, motor, and association areas, in J. Sprague and A. Epstein (eds), Progress in Psychobiology and Physiological Psychology, volume 8, Academic Press, New York, pp. 1–43.

    Google Scholar 

  27. Dong, D. and Atick, J., 1995: Temporal decorrelation: A theory of lagged and nonlagged responses in the lateral geniculate nucleus, Network 6, 159–178.

    Google Scholar 

  28. Eichenbaum, H., 2000: A cortical–hippocampal system for declarative memory, Nature Reviews, Neuroscience 1, 41–50.

    Google Scholar 

  29. Eichenbaum, H., Otto, T. and Cohen, N., 1994: Two functional roles of the hippocampal memory system, Behavioral and Brain Sciences 17, 449–518.

    Google Scholar 

  30. Field, D., 1987: Relations between the statistics of natural images and the response properties of cortical cells, Journal of the Optical Society of America A4, 2379–2394.

    Google Scholar 

  31. Földiák, P., 1990: Forming sparse representation by local anti–hebbian learning, Biological Cybernetics 64, 165–170.

    Google Scholar 

  32. Földiák, P. and Young, M., 1995: Sparse coding in the primate cortex, in M. Arbib (ed.), The Handbook of Brain Theory and Neural Networks, MIT Press, Cambridge, MA, pp. 895–898.

    Google Scholar 

  33. Ghahramani, Z. and Hinton, G., 1996: Parameter estimation for linear dynamical systems, Technical Report CRG–TR–96–2, University of Toronto, Toronto, http://www.gatsby.ucl.ac.uk/zoubin/papers.html. Grossberg, S., 1988: Competitive learning: From interactive activation to adaptive resonance, in S. Grossberg (ed.), Neural Networks and Natural Intelligence, MIT Press, Cambridge, MA.

    Google Scholar 

  34. Grossberg, S. and Carpenter, G., 1993: Normal and amnesic learning, recognition, and memory by a neural model of cortico–hippocampal interactions, Trends in Neurosciences 16, 131–137.

    Google Scholar 

  35. Harnad, S., 1987: Psychophysical and Cognitive Aspects of Categorical Perception: A Critical Overview, chapter 1, Cambridge University Press, New York.

    Google Scholar 

  36. Hateren, J. and Ruderman, D., 1998: Independent component analysis of natural image sequences yields spatio–temporal filters similar to simple cells in primary visual cortex. Proc. R. Soc. London B 265, 2315–2320.

    Google Scholar 

  37. Haykin, S., 1999: Neural Networks: A Comprehensive Foundation, Prentice Hall, New Jersey.

    Google Scholar 

  38. Henson, R., Rugg, M., Shallice, T., Josephs, O. and Dolan, R., 1999: Recollection and familiarity in recognition memory: An event–related functional magnetic resonance imaging study, Journal of Neuroscience 19, 3962–3972.

    Google Scholar 

  39. Henze, D., WE, W.C. and Barrionuevo, G., 1996: Dendritic morphology and its effects on the amplitude and rise–time of synaptic signals in hippocampal ca3 pyramidal cells, J. Comp. Neurology 369, 331–344.

    Google Scholar 

  40. Hinton, G. and Ghahramani, Z., 1997: Generative models for discovering sparse distributed representations, Philosophical Transactions of the Royal Society B 352, 1177–1190.

    Google Scholar 

  41. Hinton, G. and Sejnowski, T., 1983: Optimal perceptual inference, in Proc. of the IEEE Computer Society Conf. on Vision and Pattern Recognition, IEEE Computer Society, New York, pp. 448–453.

    Google Scholar 

  42. Hinton, G. and Zemel, R., 1994: Autoencoders, minimum description length and Helmholtz free energy, in J. Cowan, G. Tesauro and J. Alspector (eds), Advances in Neural Processing Systems, volume 6, Morgan Kaufmann, San Mateo, CA, pp. 3–10.

    Google Scholar 

  43. Hochreiter, S. and Schmidhuber, J., 1999: Lococode performs nonlinear ica without knowing the number of sources, in Proceedings of the ICA'99, Aussois, France, pp. 149–154.

    Google Scholar 

  44. Horn, B., 1977: Understanding image intensities, Artificial Intelligence 8, 201–231.

    Google Scholar 

  45. Hyvärinen, A., 1999: Survey on independent component analysis, Neural Computing Surveys 2, 94–128.

    Google Scholar 

  46. Hyvärinen, A., Hoyer, P. and Oja, E., 1999: Sparse code shrinkage: Denoising by nonlinear maximum likelihood estimation, in Advances in Neural Information Processing Systems 11 (NIPS*98), MIT Press, pp. 1739–1768.

  47. Hyvärinen, A. and Oja, E., 1997: A fast fixed–point algorithm for independent component analysis, Neural Computation 9, 1483–1492.

    Google Scholar 

  48. Jaffe, D.B. and Carnevale, N.T., 1999: Passive normalization of synaptic integration influenced by dendritic architecture, J. Neurophysiol 82, 3268–3285.

    Google Scholar 

  49. Jutten, C. and Herault, J., 1991: Blind separation of sources, Part I: An adaptive algorithm based onneuromimetic architecture, Signal Processing 24, 1–10.

    Google Scholar 

  50. Karhunen, J., Oja, E., Wang, L., Vigario, R. and Joutsensalo, J., 1997: A class of neural networks for independent component analysis, IEEE Trans. on Neural Networks 8, 487–504.

    Google Scholar 

  51. Karhunen, J., Wang, L. and Joutsensalo, J., 1995: Neural estimation of basis vectors in independent component analysis, in Proceedings of the 1995 IEEE International Conference on Neural Networks, Perth, Australia, pp. 995–1000

    Google Scholar 

  52. Koch, C. and Poggio, T., 1999: Predicting the visual world: Silence is golden, Nature Neuroscience 2, 9–10.

    Google Scholar 

  53. Laheld, B. and Cardoso, J., 1994: Adaptive source separation with uniform performance, in Signal Processing VII: Theories and applications. Proceedings of EUSIPCO–94, Edinburgh, UK (September), volume 2, 183–186.

    Google Scholar 

  54. Lee, D. and Seung, H., 1999: Learning the parts of objects by non–negative matrix factorization, Nature 401, 788–791.

    Google Scholar 

  55. Lee, D. and Seung, H., 2001: Algorithms for non–negative matrix factorization, in Advances in Neural Processing Systems, volume 13, Morgan Kaufmann, San Mateo, CA, pp. 556–562.

    Google Scholar 

  56. Levy, W., 1996: A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal–like tasks, Hippocampus 6, 579–590.

    Google Scholar 

  57. Li, Z., 1995: A theory of visual motion coding in the primary visual cortex, Neural Computation 7,705–730.

    Google Scholar 

  58. Lisman, J., 1999: Relating hippocampal circuitry to function: Recall of memory sequences by reciprocal dentate–ca3 interactions, Neuron 22, 233–242.

    Google Scholar 

  59. Lisman, J. and Idiart, M., 1995: A mechanism for storing 7ą2 short–term memories in oscillatory subcycles, Science 267, 1512–1514.

    Google Scholar 

  60. Livingston, K. andrews, J. and Harnad, S., 1998: Categorical perception effects induced by category learning, Journal of Experimental Psychology: Learning, Memory, and Cognition 24, 732–753.

    Google Scholar 

  61. Lörincz, A., 1997: Towards a unified model of cortical computation II: From control architecture to a model of consiousness, Neural Network World 7, 137–152.

    Google Scholar 

  62. Lörincz, A., 1998: Forming independent components via temporal locking of reconstruction architectures: A functional model of the hippocampus, Biological Cybernetics 79, 263–275.

    Google Scholar 

  63. Lörincz, A. and Buzsáki, G., 1999: Computational model of the entorhinal–hippocampal region derived from a single principle, in Proceedings of IJCNN (July 9–16), IEEE Catalog Number: 99CH36339C, ISBN: 0–7803–5532–6, Washington.

  64. Lörincz, A. and Buzsáki, G., 2000: Two–phase computational model training long–term memories in the entorhinal–hippocampal region, in H. Scharfman, M. Witter and R. Schwarz (eds), The Parahippocampal Region: Implications for Neurological and Psychiatric Dieseases, volume 911 of Annals of the New York Academy of Sciences, New York Academy of Sciences, New York, pp. 83–111.

    Google Scholar 

  65. Lörincz, A., Szatmáry, B. and Kabán, A., 2001a: Sign–changing filters similar to cells in primary visual cortex emerge by independent component analysis of temporally convolved natural image sequences, Neurocomputing 38–40, 1437–1442.

    Google Scholar 

  66. Lörincz, A., Szatmáry, B., Szirtes, G. and Takács, B., 2001b: Recognition of novelty made easy: Constraints of channel capacity on generative networks, in R. French (ed.), Connectionist Models of Learning, Development and Evolution, Springer–Verlag, London, pp. 73–82.

    Google Scholar 

  67. Mallat, S., 1998: A Wavelet Tour of Signal Processing, Academic Press, San Diego, CA.

    Google Scholar 

  68. McCarthy, R. and Warrington, E., 1990: Cognitive Neuropsychology, Academic Press, San Diego.

    Google Scholar 

  69. McClelland, J., 1981: Retrieving general and specific information from stored knowledge of specifics, in Proceedings of the Third Annual Meeting of the Cognitive Science Society, pp. 170–172.

  70. McClelland, J. and Rumelhart, D., 1981: An interactive activation model of context effects in letter perception: Part 1 an account of basic findings, Psychological Review 88, 375–407.

    Google Scholar 

  71. McClelland, J. and Rumelhart, D., 1982: An interactive activation model of context effects in letter perception: Part 2 the contextual enhancement effect and some tests and extensions of the model, Psychological Review 89, 60–94.

    Google Scholar 

  72. Merhav, N. and Feder, M., 1998: Universal prediction, IEEE Trans. Inform. Theory. IT–44, 2124–2147.

  73. Mishkin, M. and Murray, E., 1994: Stimulus recognition, Current Opinion in Neurobiology 4, 200–206.

    Google Scholar 

  74. Mozer, M., 1991: Discovering discrete distributed representations with iterative competitive learning, in R. Lippmann, J. Moody and D. Touretzky (eds), Advances in Neural Processing Systems, volume 3, Morgan Kaufmann, San Mateo, CA, pp. 627–634.

    Google Scholar 

  75. O'Keefe, J. and Nadel, L., 1978: The Hippocampus as a Cognitive Map, Clarendon Press, Oxford.

    Google Scholar 

  76. Olshausen, B., 1996: Learning linear, sparse factorial codes, A.I. Memo 1580, MIT AI Lab. C.B.C.L.

  77. Olshausen, B. and Field, D., 1996: Emergence of simple–cell receptive field properties by learning a sparse code for natural images, Nature 381, 607–609.

    Google Scholar 

  78. Olshausen, B. and Field, D., 1997: Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research 37, 3311–3325.

    Google Scholar 

  79. Pajunen, P., 1998: Blind source separation using algorithmic information theory, in C. Fyfe (ed.), Proceedings of Independence and Artificial Neural Networks, ISCS Academic Press, pp. 26–31.

  80. Palm, G., 1992: On the information storage capacity of local learning rules, Neural Computation 4, 703–711.

    Google Scholar 

  81. Papoulis, A., 1984: Probability, Random Variables and Stochastic Processes, 2nd edition, McGraww–Hill, New York.

    Google Scholar 

  82. Parra, L., Deco, G. and Miesbach, S., 1995: Statistical independence and novelty detection with information preserving nonlinear maps, Neural Computation 8(2), 260–269.

    Google Scholar 

  83. Rao, R. and Ballard, D., 1997: Dynamic model of visual recognition predicts neural response properties in the visual cortex, Neural Computation 9, 721–763.

    Google Scholar 

  84. Rao, R. and Ballard, D., 1999: Predictive coding in the visual cortex: A functional interpretation of some extra–classical receptive–field effects, Nature Neuroscience 2, 79–87.

    Google Scholar 

  85. Rezek, I., Sykacek, P. and Roberts, S., 2000: Coupled hiddenMarkov models for biosignal interaction modelling, Technical report PARG–00–5, Oxford University, Oxford, UK to appear in 2000 IEE Special Issue Proceedings on Advances in Medical Signal and Information Processing.

    Google Scholar 

  86. Riedel, G., Micheau, J., Lam, A., Roloff, E., Martin, S., Bridge, H., Hoz, L., Poeschel, B., McVulloch, J. and Morris, R., 1999: Reversible neural inactivation reveals hippocampal participation in several memory processes, Nature Neuroscience 2, 898–905.

    Google Scholar 

  87. Rissanen, J., 1978: Modeling by shortest data description, Automatica 14, 465–471.

    Google Scholar 

  88. Rissanen, J., 1984: Universal coding, information, prediction and estimation, IEEE Trans. Inform. Theory IT–30, 629–636.

    Google Scholar 

  89. Roberts, S., 2000: Novelty Detection Using Extreme Value Statistics, draft, Oxford University, Oxford, UK, http://www.robots.ox.ac.ukparg.

    Google Scholar 

  90. Roweis, A. and Ghahramani, J., 1999: A unifying review of linear gaussian models, Neural Computation 11, 305–345.

    Google Scholar 

  91. Sáry, G., Vogels, R. and Orban, G., 1994: Orientation discrimination of motion–defined gratings, Vision Res. 34, 1331–1334.

    Google Scholar 

  92. Schachter, D., 1987: Implicit memory: History and current status, Journal of Experimental Psychology: Learning, Memory, and Cognition 13, 501–518.

    Google Scholar 

  93. Schlitz, C., Bodart, J., Dubois, S., Dejardin, S., Michel, C., Roucoux, A., Crommelinck, M. and Orban, G., 1999: Neuronal mechanisms of perceptual learning: Changes in human brain activity with training in orientation discrimination, NeuroImage 9, 46–62.

    Google Scholar 

  94. Schölkopf, B., Platt, J., Shawe–Taylor, J., Smola, A. and Williamson, R., 1999: Estimating the support of a high–dimensional distribution, Technical Report 99–87, Microsoft Research.

  95. Scoville, W. and Milner, B., 1957: Loss of recent memory after bilateral hippocampal lesions, Journal of Neurol. Neurosurg. Psychiatry 20, 11–21.

    Google Scholar 

  96. Searle, J., 1992: The Rediscovery of Mind, Bradford Books, MIT Press, Cambridge, MA.

    Google Scholar 

  97. Shallice, T., 1988: From Neuropsychiology to Mental Structure, Cambridge Univ. Press, New York.

    Google Scholar 

  98. Shannon, C., 1948: A mathematical theory of communication, Bell Sys. Tech. Journal 27, 379–423 and 623–656.

    Google Scholar 

  99. Solomonoff, R., 1964: A formal theory of inductive inference, part i, Information and Control 7, 1–22.

    Google Scholar 

  100. Squire, L., 1992a: Declarative and nondeclarative memory: Multiple brain systems supporting learning and memory, J. Cog. Neurosci. 4, 232–243.

    Google Scholar 

  101. Squire, L., 1992b: Memory and the hippocampus: A synthesis of findings with rats, monkeys, and humans, Psychol. Rev. 99, 195–231.

    Google Scholar 

  102. Squire, L. and Kandel, E., 1999: Memory: From Mind to Molecules, Scientific American Press, New York.

    Google Scholar 

  103. Szatmáry, B. and Lörincz, A., 2001: Independent component analysis of temporal sequences subject to constraints by lgn inputs yields all the three major cell types of the primary visual cortex, J. of Comp. Neurosci. 11, 241–248.

    Google Scholar 

  104. Thorpe, S., Fize, D. and Marlot, C., 1996: Speed of processing in the human visual system, Nature 381, 520–522.

    Google Scholar 

  105. Tornay, S., 1938: Ockham: Studies and selections.

  106. Treves, A., Panzeri, S., Rolls, E., Booth, M. and Wakeman, E., 1999: Firing rate distributions and effi–ciency of information transmission of inferior temporal cortex neurons to natural visual stimuli, Neural Computation 11, 601–631.

    Google Scholar 

  107. Tulving, E., 1983: Elements of Episodic Memory, Clarendon Press, Oxford.

  108. Vovk, V. and Gammerman, A., 1999: Complexity approximation principle, Computer Journal 42, 318–322.

    Google Scholar 

  109. Wallace, C. and Boulton, D., 1968: An information theoretic measure for classification, Computer Journal 11, 185–194.

    Google Scholar 

  110. Wan, H., Aggleton, J. and Brown, M., 1999: Different contributions of the hippocampus and perirhinal cortex to recognition memory, J. Neurosci. 19, 1142–1148.

    Google Scholar 

  111. Wimbauer, S., Wenish, O., Miller, K. and van Hemmen, J., 1997: Development of spatio–temporal receptive fields of simple cells: I. Model formulation, Biological Cybernetics 77, 456–461.

    Google Scholar 

  112. Zemel, R. and Hinton, G., 1994: Developing population codes by minimizing description length, in J. Cowan, G. Tesauro and J. Alspector (eds), Advances in Neural Processing Systems, volume 6, Morgan Kaufmann, San Mateo, CA, pp. 11–18.

    Google Scholar 

Download references

Author information

Affiliations

  1. Department of Information Systems, ELTE, Budapest, Hungary

    András Lörincz

  2. Department of Information Systems, ELTE, Budapest, Hungary

    Barnabás Póczos, Gábor Szirtes & Bálint Takács

Authors
  1. András Lörincz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barnabás Póczos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gábor Szirtes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bálint Takács
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to András Lörincz.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Lörincz, A., Póczos, B., Szirtes, G. et al. Ockham's Razor at Work: Modeling of the ``Homunculus''. Brain and Mind 3, 187–220 (2002). https://doi.org/10.1023/A:1019996320835

Download citation

  • functional modeling
  • generative networks
  • homunculus fallacy
  • MMI
  • recognition