Variational Information Maximization for Neural Coding

  • Felix Agakov
  • David Barber
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3316)


Mutual Information (MI) is a long studied measure of coding efficiency, and many attempts to apply it to population coding have been made. However, this is a computationally intractable task, and most previous studies redefine the criterion in forms of approximations. Recently we described properties of a simple lower bound on MI [2]. Here we describe the bound optimization procedure for learning of population codes in a simple point neural model. We compare our approach with other techniques maximizing approximations of MI, focusing on a comparison with the Fisher Information criterion.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agakov, F.V., Barber, D.: Variational Information Maximization and Fisher Information. Technical report, UoE (2004)Google Scholar
  2. 2.
    Barber, D., Agakov, F.V.: The IM Algorithm: A Variational Approach to Information Maximization. In: NIPS (2003)Google Scholar
  3. 3.
    Barlow, H.: Unsupervised Learning. Neural Computation 1, 295–311 (1989)CrossRefGoogle Scholar
  4. 4.
    Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7(6), 1129–1159 (1995)CrossRefGoogle Scholar
  5. 5.
    Brunel, N., Nadal, J.-P.: Mutual Information, Fisher Information and Population Coding. Neural Computation 10, 1731–1757 (1998)CrossRefGoogle Scholar
  6. 6.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, Chichester (1991)MATHCrossRefGoogle Scholar
  7. 7.
    Linsker, R.: An Application of the Principle of Maximum Information to Linear Systems. In: NIPS (1989)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Felix Agakov
    • 1
  • David Barber
    • 2
  1. 1.University of EdinburghEdinburghUK
  2. 2.IDIAPRue du Simplon 4MartignySwitzerland

Personalised recommendations