Summary of Information Theoretic Quantities
Information theory is a practical and theoretic framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. As a framework, it has a number of useful properties: it provides a general measure sensitive to any relationship, not only linear effects; its quantities have meaningful units which, in many cases, allow a direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single experimental trials rather than in averages over multiple trials. A variety of information theoretic quantities are in common use in neuroscience – including the Shannon entropy, Kullback–Leibler divergence, and mutual information. In this entry, we introduce and define these quantities. Further details on how these quantities can be estimated in...
KeywordsMutual Information Conditional Entropy Discrete Random Variable Leibler Divergence Differential Entropy
Research supported by the SI-CODE (FET-Open, FP7-284533) project and by the ABC and NETT (People Programme Marie Curie Actions PITN-GA-2011-290011 and PITN-GA-2011-289146) projects of the European Union’s Seventh Framework Programme FP7 2007–2013.
- Bell AJ (2003) The co-information lattice. In: 4th International Symposium on independent component analysis blind signal separation ICA2003, Nara, pp 921–926Google Scholar
- Fuglede B, Topsoe F (2004) Jensen-Shannon divergence and Hilbert space embedding. In: Proceedings of the international symposium on information theory, 2004 (ISIT 2004), p 31, Chicago, USAGoogle Scholar
- Green DM, Swets JA (1966) Signal detection theory and psychophysics. Wiley, New York. http://andrei.gorea.free.fr/Teaching_fichiers/SDT%20and%20Psytchophysics.pdf. Accessed 17 Jan 2014
- Jakulin A, Bratko I (2003) Quantifying and visualizing attribute interactions. arXiv:cs/0308002. http://arxiv.org/abs/cs/0308002. Accessed 13 Feb 2012
- Shannon CE, Weaver W (1949) The mathematical theory of communication. University of Illinois Press, Urbana, 19:1Google Scholar