Skip to main content

Forming sparse representations by local anti-Hebbian learning

Abstract

How does the brain form a useful representation of its environment? It is shown here that a layer of simple Hebbian units connected by modifiable anti-Hebbian feed-back connections can learn to code a set of patterns in such a way that statistical dependency between the elements of the representation is reduced, while information is preserved. The resulting code is sparse, which is favourable if it is to be used as input to a subsequent supervised associative layer. The operation of the network is demonstrated on two simple problems.

This is a preview of subscription content, access via your institution.

References

  • Barlow HB (1961) Possible principles underlying the transformations of sensory messages. In: Rosenblith WA (ed) Sensory communication, MIT Press, Cambridge (Mass) London, pp 217–234

    Google Scholar 

  • Barlow HB (1972) Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1:371–394

    CAS  PubMed  Google Scholar 

  • Barlow HB (1985) Cerebral cortex as model builder. In: Rose D, Dobson VG (eds) Models of the visual cortex. Wiley, Chichester, pp 37–46

    Google Scholar 

  • Barlow HB (1989) Unsupervised learning. Neural Comput 1:295–311

    Google Scholar 

  • Barlow HB, Földiák P (1989) Adaptation and decorrelation in the cortex. In: Durbin RM, Miall C, Mitchison GJ (eds) The computing neuron, chap 4, Addison-Wesley, Wokingham, pp 54–72

    Google Scholar 

  • Barlow HB, Kaushal TP, Mitchison GJ (1989) Finding minimum entropy codes. Neural Comput 1:412–423

    Google Scholar 

  • Földiák P (1989) Adaptive network for optimal linear feature extraction. Proceedings of the IEEE/INNS International Joint Conference on Neural Networks, Washington D. C. June 18–22, 1989, vol. 1. IEEE Press, New York, pp 401–405

    Google Scholar 

  • Grossberg S (1976) Adaptive pattern classification and universal recoding. I. Parallel development and coding of neural feature detectors. Biol Cybern 23:121–134

    Article  CAS  PubMed  Google Scholar 

  • Harris CS (1980) Insight or out of sight?: Two examples of perceptual plasticity in the human adult. In: Harris CS (ed) Visual coding and adaptability. Erlbaum, Hillsdale, NJ

    Google Scholar 

  • Hebb DO (1949) The organization of behaviour. Wiley, New York

    Google Scholar 

  • Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA 79:2554–2558

    CAS  PubMed  Google Scholar 

  • Kohonen T (1984) Self-organization and associative memory. Springer, Berlin Heidelberg New York

    Google Scholar 

  • Malsburg Ch von der (1973) Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14:85–100

    Article  PubMed  Google Scholar 

  • Palm G (1980) On associative memory. Biol Cybern 36:19–31

    Article  CAS  PubMed  Google Scholar 

  • Rubner J, Schulten K (1990) Development of feature detectors by self-organization. Biol Cybern 62:13–199

    Article  Google Scholar 

  • Rumelhart DE, Zipser D (1985) Feature discovery by competitive learning. Cogn Sci 9:75–112

    Article  Google Scholar 

  • Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Rumelhart DE, McClelland J (eds) Parallel distributed processing, vol 1. MIT Press, Cambridge Mass London, pp 318–362

    Google Scholar 

  • Watanabe S (1960) Information-theoretical aspects of inductive and eductive inference. IBM J Res Dev 4:208–231

    Google Scholar 

  • Watanabe S (1985) Pattern recognition: human and mechanical. Wiley, New York

    Google Scholar 

  • Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 222:960–962

    CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Földiák, P. Forming sparse representations by local anti-Hebbian learning. Biol. Cybern. 64, 165–170 (1990). https://doi.org/10.1007/BF02331346

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02331346

Keywords

  • Sparse Representation
  • Statistical Dependency
  • Simple Problem
  • Associative Layer