Journal of Computational Neuroscience

, Volume 22, Issue 2, pp 135–146 | Cite as

A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields

  • Martin Rehn
  • Friedrich T. SommerEmail author


Computational models of primary visual cortex have demonstrated that principles of efficient coding and neuronal sparseness can explain the emergence of neurones with localised oriented receptive fields. Yet, existing models have failed to predict the diverse shapes of receptive fields that occur in nature. The existing models used a particular “soft” form of sparseness that limits average neuronal activity. Here we study models of efficient coding in a broader context by comparing soft and “hard” forms of neuronal sparseness.

As a result of our analyses, we propose a novel network model for visual cortex. The model forms efficient visual representations in which the number of active neurones, rather than mean neuronal activity, is limited. This form of hard sparseness also economises cortical resources like synaptic memory and metabolic energy. Furthermore, our model accurately predicts the distribution of receptive field shapes found in the primary visual cortex of cat and monkey.


Biological vision Sparse coding Receptive field learning 



We thank B. A. Olshausen for providing ideas, D. Ringach for the experimental data, T. Bell, J. Culpepper, G. Hinton, J. Hirsch, C. v. d. Malsburg, B. Mel, L. Perrinet, D. Warland, L. Zhaoping and J. Zhu for discussions. Further, we thank the reviewers for helpful suggestions. Financial support was provided by the Strauss-Hawkins trust and KTH.


  1. Atick JJ (1992) Could information theory provide an ecological theory of sensory processing? Network: Comput. Neural Syst. 3: 213–251.Google Scholar
  2. Attneave F (1954) Some informational aspects of visual perception. Psychol. Rev. 61: 183–193.Google Scholar
  3. Baddeley R (1996) An efficient code in V1? Nature 381: 560–561.Google Scholar
  4. Barlow HB (1983) Understanding natural vision. Springer-Verlag, Berlin.Google Scholar
  5. Baum EB, Moody J, Wilczek F (1988) Internal representations for associative memory. Biol. Cybern. 59: 217–228.Google Scholar
  6. Bell AJ, Sejnowski TJ (1997) The independent components of natural images are edge filters. Vis. Res. 37: 3327–3338.Google Scholar
  7. Bethge M, Rotermund D, Pawelzik K (2003) Second order phase transition in neural rate coding: binary encoding is optimal for rapid signal transmission. Phys. Rev. Lett. 90(8): 088104–0881044.Google Scholar
  8. Buhmann J, Schulten K (1988) Storing sequences of biased patterns in neural networks with stochastic dynamics. In R Eckmiller, C von der Malsburg, eds. Neural Computers, Berlin. (Neuss 1987), Springer-Verlag, pp. 231–242.Google Scholar
  9. Chen SS, Donoho DL, Saunders MA (1998) Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1):33–61.Google Scholar
  10. Dayan P, Abbott L (2003) Theoretical Neuroscience. MIT Press, Cambridge MA.Google Scholar
  11. Donoho DL, Elad M (2002) Optimally sparse representation in general (nonorthogonal) dictionaries via \(l^{1}\) minimization. PNAS 100(5): 2197–2002.Google Scholar
  12. Field DJ (1987) Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4: 2379–2394.Google Scholar
  13. Földiak P (1995) Sparse coding in the primate cortex. In MA Arbib ed. The Handbook of Brain Theory and Neural Networks, MIT Press, Cambridge, MA, pp. 165–170.Google Scholar
  14. Gardner-Medwin A (1976) The recall of events through the learning of associations between their parts. Proc. R. Soc. London. B 194: 375–402.Google Scholar
  15. Gibson JJ (1966) The Perception of the Visual World. Houghton Mifflin, Boston, MA.Google Scholar
  16. Hinton GE, Sallans B, Ghahramani ZJ (1997) A hierarchical community of experts. In M Jordan, ed. Learning in Graphical Models, Kluwer Academic Publishers, pp. 479–494.Google Scholar
  17. Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Sci. (USA) 79: 2554–2558.Google Scholar
  18. Hurri J, Hyvärinen A (2003) Simple-cell-like receptive fields maximize temporal coherence in natural video. Neural. Comput. 15: 663–691.Google Scholar
  19. Hyvärinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw. 13: 411–430.Google Scholar
  20. Jones JP, Palmer LA (1987) The two-dimensional spatial structure of simple receptive fields in the striate cortex. J. Neurophysiol. 58: 1187–1211.Google Scholar
  21. Laughlin SB, Sejnowski TJ (2003) Communication in neuronal networks. Science 301: 1870–1874.Google Scholar
  22. Lennie P (2003) The cost of cortical computation. Curr. Biol. 13(6): 493–497.Google Scholar
  23. Li Z, Attick J (1994) Towards a theory of the striate cortex. Neural. Comput. 6: 127–146.Google Scholar
  24. Mallat S, Zhang Z, (1993) Matching pursuit in time-frequency dictionary. IEEE Trans. Signal Process. 41: 3397–3415.Google Scholar
  25. Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381: 607–608.Google Scholar
  26. Olshausen BA, Field DJ (1997) Sparse coding with an overcomplete basis set: a strategy employed in V1. Vis. Res. 37: 3311–3325.Google Scholar
  27. Palm G (1980) On associative memory. Biol. Cybern. 36: 19–31.Google Scholar
  28. Palm G, Sommer FT (1992) Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states. Network: Comput. Neural Syst. 3: 1–10.Google Scholar
  29. Palm G, Sommer FT (1995) Associative data storage and retrieval in neural networks. In E Domany, JL van Hemmen, KS Hemmen, eds. Models of Neural Networks III. Springer, New York, pp. 79 –118.Google Scholar
  30. Pati Y, Reziifar R, Krishnaprasad P (1993) Orthogonal matching pursuits: recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers, pp. 40–44.Google Scholar
  31. Perrinet L, Samuelides M, Thorpe S (2004) Sparse spike coding in an asynchronous feed-forward multi-layer neural network using matching pursuit. Neurocomputing 57: 125–134.Google Scholar
  32. Peters A, Paine BR (1994) A numerical analysis of the genicocortical input to striate cortex in the monkey. Cereb. Cortex 4: 215–229.Google Scholar
  33. Rebollo-Neira L, Lowe D (2002) Optimised orthogonal matching pursuit approach. IEEE Signal Process. Lett. 9: 137–140.Google Scholar
  34. Rehn M, Sommer FT (2006) Storing and restoring visual input with collaborative rank coding and associative memory. Neurocomputing 69(10–12): 1219–1223.Google Scholar
  35. Ringach DL (2002) Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. J. Neurophys. 88(1): 455–463.Google Scholar
  36. Ruderman D (1994) The statistics of natural images. Network: Comput. Neural Syst. 5: 517–548.Google Scholar
  37. Sallee P (2002) Statistical methods for image and signal processing. PhD Thesis, UC Davis.Google Scholar
  38. Sallee P, Olshausen BA (2002) Learning sparse multiscale image representations. In TK Leen, TG Dietterich, V Tresp, eds. Advances in Neural Information Processing Systems, vol. 13. Morgan Kaufmann Publishers, Inc., pp. 887–893.Google Scholar
  39. Sommer FT, Wennekers T (2005) Synfire chains with conductance-based neurons: internal timing and coordination with timed input. Neurocomputing 65–66: 449–454.Google Scholar
  40. Stroud JM (1956) The fine structure of psychological time. In H Quastler, ed. Information Theory in Psychology. Free Press, pp. 174–205.Google Scholar
  41. Teh Y, Welling M, Osindero S, Hinton GE (2003) Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res. 4: 1235–1260.Google Scholar
  42. Treves A (1991) Dilution and sparse coding in threshold-linear nets. J. Phys. A: Math. Gen. 24: 327–335.Google Scholar
  43. Tsodyks MV, Feigelman MV (1988) The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6: 101–105.Google Scholar
  44. Van Essen DC, Anderson CH (1995) Information processing strategies and pathways in the primate retina and visual cortex. In SF Zornetzer, JL Davis, C Lau, eds. Introduction to Neural and Electronic Networks, 2nd ed. Orlando, Academic Press, pp. 45–76.Google Scholar
  45. VanRullen R, Koch C (2003) Is perception discrete or continuous? Trends Cogn. Sci. 7: 207–213.Google Scholar
  46. VanRullen R, Koch C (2005) Attention-driven discrete sampling of motion perception. Proc. Nat. Acad. Sci., USA 102: 5291–5296.Google Scholar
  47. Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 222: 960–962.Google Scholar
  48. Willwacher G (1982) Storage of a temporal sequence in a network. Biol. Cybern. 43: 115–126.Google Scholar
  49. Zetsche C (1990) Sparse coding: the link between low level vision and associative memory. In R Eckmiller, G Hartmann, G Hauske, eds., Parallel Processing in Neural Systems and Computers. Elsevier Science Publishers B.V. (North Holland).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2006

Authors and Affiliations

  1. 1.Computational Biology and NeurocomputingSchool of Computer Science and Communication, Royal Institute of Technology (KTH)StockholmSweden
  2. 2.Redwood Center for Theoretical NeuroscienceUniversity of California, BerkeleyBerkeleyUSA

Personalised recommendations